Company Input
Accepts a company name or source website from the Streamlit interface.
Data engineering portfolio project
Compass Ultra Website Intelligence turns a company name or website into a data run: discovery, crawling, Snowflake loading, dbt modeling, and Streamlit analysis.
Architecture
Accepts a company name or source website from the Streamlit interface.
Finds related public pages and candidate sources for analysis.
Crawls pages, parses text, validates configuration, and prepares load records.
Stores website records in Snowflake for analytics and repeatable modeling.
Builds staging models, facts, marts, and tests with dbt-snowflake.
Displays domain tables, ranked website signals, and sourced summaries.
Signal Engine
Feature flags, deploys, rollbacks, canaries, release gates, and CAB handoffs.
Audit, compliance, SOC 2, approval workflows, change management, and review evidence.
Slack, Jira, GitHub, CI/CD, runbooks, ownership, cleanup, and stale flag debt.
ANALYTICS.MART_WEBSITE_QUERY_INDEX
ANALYTICS.MART_PROSPECT_ACCOUNTS
ANALYTICS.FCT_WEBSITE_SIGNALS
Repository Evidence
Loads crawled page records into the raw warehouse layer and supports bootstrap setup.
dbt SQLModels release-readiness signals into an analytics fact table.
dbt martTurns raw page content and signal scoring into account-level analysis.
Fivetran-readyUses the Fivetran API pattern for connection and destination metadata when configured.
StreamlitRuns company lookup, discovery, crawl controls, logs, and refreshed result views.
dbt profileDefines dbt connection patterns for password, private key, and browser authentication.
Stack
Runtime
This static page is safe to share publicly. The live workflow needs a Python runtime plus secrets for Snowflake, Tavily, and optional AI providers, so it belongs on Streamlit Community Cloud or another secure runtime.
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
Copy-Item .env.example .env
python scripts\validate_environment.py
python scripts\crawl_websites_to_snowflake.py --bootstrap-only
streamlit run app\streamlit_app.py