An energy distributor needed clean, daily pricing data to train and run a machine-learning pricing model.
Build
Oracle extracts land in S3 as Parquet, are crawled into the Data Catalog and exposed to Redshift. Bronze, silver and gold layers refresh on a fixed schedule, with public weather data joined to each delivery.
Key choices
Medallion layers on materialized views, Step Functions orchestration from crawl to model run, and partitioned daily extracts for the data science team.
02 · 2024
Legacy transformation tool moved to a modern data stack
A software vendor ran its data transformations in an in-house legacy tool that was hard to maintain and to change.
Build
Client databases and files are loaded into Snowflake with Airbyte, Fivetran and staged S3 loads. dbt Cloud rebuilds the logic in three tested SQL layers and produces the pivot model.
Key choices
The pivot tables are unloaded back to S3 and into the existing application database, so the product kept working while the engine underneath was replaced.
03 · 2024
Automated ad-revenue reporting from emailed exports
Ad tech · Google CloudCloud Functions · Airbyte · BigQuery
An ad-tech team received its revenue data only as daily emailed reports, copied by hand into spreadsheets.
Build
A scheduled Cloud Function reads the inbox and saves each report to Cloud Storage. Airbyte loads it into BigQuery, and a scheduled query merges it incrementally into the serving tables.
Key choices
Append-then-merge with a staging truncate keeps reruns safe, and a separate backfill path reloads history without touching the daily flow.
04 · 2021–2024
Data platform for a subscription e-commerce brand
E-commerce · 12 sources → 10 data martsFivetran · Airbyte · BigQuery · dbt · Power BI
A fast-growing subscription brand needed one trusted view of sales, retention, delivery and marketing across a dozen tools.
Build
Managed and open-source connectors plus event webhooks land everything in BigQuery. dbt turns it into four tested layers, with snapshots for customer and subscription history.
Key choices
KPIs defined with the head of data, one shared Power BI model for finance and operations, and reverse ETL to feed segments back to marketing.
05 · 2022–2023
Modern data stack for a services company
Local services · platform migrationBigQuery · dbt · Tableau
A growing services company was moving to a new operational platform and needed one reliable view across operations, recruiting and CRM data.
Build
API sources land in BigQuery through custom Python connectors and a managed EL tool. dbt models, versioned in Git, turn staging data into tested, documented marts.
Outcome
Dashboards rebuilt for operations teams in a short time, and modelled data synced back into the CRM and the ops workspace through reverse ETL.
06 · 2021–2022
Price mapping and competitor-gap reporting
Energy distribution · on-premise BISQL Server · SSIS · Power BI
An energy distributor wanted to see its fuel prices against competitors, station by station, and react faster to price gaps.
Build
SSIS packages scheduled by SQL Agent pull sales and delivery data from the AS400 ERP and the public daily fuel-price feed into SQL Server staging, then into incremental reporting views.
Outcome
Power BI price maps and competitor-gap reports went into production and replaced manual spreadsheet comparisons.
07 · 2021
Event-processing pipeline for a customer-journey analytics product
A customer-journey analytics product had to turn large client event logs into journeys and cycles that its web app could query quickly.
Build
An insertion API stores uploaded logs in S3. Spark jobs rebuild events, journeys, cycles and graph edges and bulk-load them into Vertica, while MongoDB keeps the app metadata.
Key choices
A columnar analytical store for aggregation queries, and a clear split between batch processing and the API that serves the app.