Data engineering portfolio project

Company intelligence pipeline for release-readiness markets.

Compass Ultra Website Intelligence turns a company name or website into a data run: discovery, crawling, Snowflake loading, dbt modeling, and Streamlit analysis.

run_status ready_for_review
Python ingestion Snowflake warehouse dbt Core + dbt Snowflake Streamlit app Tavily discovery Fivetran API metadata

Architecture

From public website signals to modeled analytics tables.

01

Company Input

Accepts a company name or source website from the Streamlit interface.

02

Tavily Discovery

Finds related public pages and candidate sources for analysis.

03

Python Ingestion

Crawls pages, parses text, validates configuration, and prepares load records.

04

Snowflake Raw Layer

Stores website records in Snowflake for analytics and repeatable modeling.

05

dbt Models

Builds staging models, facts, marts, and tests with dbt-snowflake.

06

Streamlit Analysis

Displays domain tables, ranked website signals, and sourced summaries.

Signal Engine

Scores websites for Compass Ultra fit.

Release language

Feature flags, deploys, rollbacks, canaries, release gates, and CAB handoffs.

Control language

Audit, compliance, SOC 2, approval workflows, change management, and review evidence.

Workflow language

Slack, Jira, GitHub, CI/CD, runbooks, ownership, cleanup, and stale flag debt.

ANALYTICS.MART_WEBSITE_QUERY_INDEX ANALYTICS.MART_PROSPECT_ACCOUNTS ANALYTICS.FCT_WEBSITE_SIGNALS

Repository Evidence

Files that show the hands-on implementation.

Stack

What each layer is responsible for.

Layer Tool Purpose
App Streamlit Live company runner and query interface.
Ingestion Python Discovery, crawling, parsing, validation, and Snowflake loading.
Warehouse Snowflake Raw public pages and analytics outputs.
Modeling dbt Core + dbt Snowflake Staging models, marts, and tests.
Discovery Tavily Company lookup and related-site search.
Optional ops Fivetran, Stripe, Vercel, Compass backend Operational metadata sources when configured.

Runtime

GitHub Pages is the portfolio page. Streamlit runs the app.

This static page is safe to share publicly. The live workflow needs a Python runtime plus secrets for Snowflake, Tavily, and optional AI providers, so it belongs on Streamlit Community Cloud or another secure runtime.

Local setup

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
Copy-Item .env.example .env

Run analysis app

python scripts\validate_environment.py
python scripts\crawl_websites_to_snowflake.py --bootstrap-only
streamlit run app\streamlit_app.py