David Balli

Portfolio

7 solution architectures · client names withheld
  1. 01Medallion data lake feeding a pricing model2024–2025
  2. 02Legacy transformation tool moved to a modern data stack2024
  3. 03Automated ad-revenue reporting from emailed exports2024
  4. 04Data platform for a subscription e-commerce brand2021–2024
  5. 05Modern data stack for a services company2022–2023
  6. 06Price mapping and competitor-gap reporting2021–2022
  7. 07Event-processing pipeline for a customer-journey analytics product2021
01 · 2024–2025

Medallion data lake feeding a pricing model

Energy distribution · AWSS3 · Glue · Redshift · SageMaker
01SOURCES02LAND03CATALOG & ORCHESTRATE04BRONZE05SILVER & GOLD06SERVEProduction ERPOracle · facts, incrementalProduction ERPOracle · dimensions, fullPublic weather dataOpen data filesS3 landing zoneParquet filesGlue jobDownload · unzip · convertGlue crawlersScheduled 07:00 UTCData CatalogExternal schemaStep FunctionsEventBridge-triggeredRedshift bronzeMaterialized views · 08:00SilverDedup · renaming · 08:15GoldBusiness viewsNearest stationWeather per deliveryGlue exportDaily, per subsidiaryS3 serving zonePartitioned extractsSageMakerPricing model
  • Context

    An energy distributor needed clean, daily pricing data to train and run a machine-learning pricing model.

  • Build

    Oracle extracts land in S3 as Parquet, are crawled into the Data Catalog and exposed to Redshift. Bronze, silver and gold layers refresh on a fixed schedule, with public weather data joined to each delivery.

  • Key choices

    Medallion layers on materialized views, Step Functions orchestration from crawl to model run, and partitioned daily extracts for the data science team.

02 · 2024

Legacy transformation tool moved to a modern data stack

Software vendor · migrationAirbyte · Fivetran · Snowflake · dbt Cloud
01SOURCES02INGEST03STORAGE04TRANSFORM05PIVOT MODEL06EXPORTClient databasesSQL Server · on-premiseLocal storageExported filesShared driveReference sheetsAirbyteDatabase connectorsFivetranFile & SaaS connectorsS3 stageSnowflake load commandSnowflakeLanding schemasdbt Cloud3 SQL layersData quality testsdbt testsGit repositoryGitHub · CISnowflakePivot tablesS3 unloadSnowflake unload commandLegacy app databaseSQL Server · destinationDownstream jobTriggered after load
  • Context

    A software vendor ran its data transformations in an in-house legacy tool that was hard to maintain and to change.

  • Build

    Client databases and files are loaded into Snowflake with Airbyte, Fivetran and staged S3 loads. dbt Cloud rebuilds the logic in three tested SQL layers and produces the pivot model.

  • Key choices

    The pivot tables are unloaded back to S3 and into the existing application database, so the product kept working while the engine underneath was replaced.

03 · 2024

Automated ad-revenue reporting from emailed exports

Ad tech · Google CloudCloud Functions · Airbyte · BigQuery
01SOURCES02CAPTURE03LAND04STAGING05SERVING06REPORTSAd platform reportsEmailed CSV exportsBackfill filesManual, headers fixedCloud Scheduler11:25 · 17:25 · 23:25Cloud FunctionReads inbox, saves CSVCloud StorageTwo CSV feedsAirbyteCSV jobs at :30BigQueryRaw staging tablesScheduled queryIncremental merge · 11:40Truncate stagingAfter each mergeRevenue reportsAgency & reseller views
  • Context

    An ad-tech team received its revenue data only as daily emailed reports, copied by hand into spreadsheets.

  • Build

    A scheduled Cloud Function reads the inbox and saves each report to Cloud Storage. Airbyte loads it into BigQuery, and a scheduled query merges it incrementally into the serving tables.

  • Key choices

    Append-then-merge with a staging truncate keeps reruns safe, and a separate backfill path reloads history without touching the daily flow.

04 · 2021–2024

Data platform for a subscription e-commerce brand

E-commerce · 12 sources → 10 data martsFivetran · Airbyte · BigQuery · dbt · Power BI
01SOURCES02INGEST03STORAGE04TRANSFORM05SERVING06ANALYTICS & ACTIVATIONE-commerce platformOrders · productsSubscriptionsRecurring billingERPStock · costsMarketing & CRMEmail · reviews · NPSReference dataShared spreadsheetsFivetranManaged connectorsAirbyteOpen-source connectorsWebhooksPub/Sub · Cloud FunctionsBigQueryLanding datasetsdbt models4 SQL layersSnapshotsCustomers · subscriptionsData quality testsdbt testsGit repositoryGitHub · CIBigQuery10 data martsPower BIShared semantic modelMetabaseSelf-service queriesReverse ETLBack to marketing tools
  • Context

    A fast-growing subscription brand needed one trusted view of sales, retention, delivery and marketing across a dozen tools.

  • Build

    Managed and open-source connectors plus event webhooks land everything in BigQuery. dbt turns it into four tested layers, with snapshots for customer and subscription history.

  • Key choices

    KPIs defined with the head of data, one shared Power BI model for finance and operations, and reverse ETL to feed segments back to marketing.

05 · 2022–2023

Modern data stack for a services company

Local services · platform migrationBigQuery · dbt · Tableau
01SOURCES02EXTRACT & LOAD03STAGING04TRANSFORM05SERVING06ACTIVATE & REPORTOperations platformSaaS · REST APIRecruiting / HRSaaS · REST APICRMSaaS · REST APIInternal filesOn-premise · versionedCustom connectorsPython · API codeEL toolManaged connectorsBigQueryRaw & staging datasetsdbt modelsSQL · tests · docsGit repositoryGitHub · CIBigQueryMarts · serving layerReverse ETLSyncs modelled dataCRMdestinationOps workspaceAirtable · destinationTableauOps dashboards
  • Context

    A growing services company was moving to a new operational platform and needed one reliable view across operations, recruiting and CRM data.

  • Build

    API sources land in BigQuery through custom Python connectors and a managed EL tool. dbt models, versioned in Git, turn staging data into tested, documented marts.

  • Outcome

    Dashboards rebuilt for operations teams in a short time, and modelled data synced back into the CRM and the ops workspace through reverse ETL.

06 · 2021–2022

Price mapping and competitor-gap reporting

Energy distribution · on-premise BISQL Server · SSIS · Power BI
01SOURCES02INTEGRATE03STAGING04PROCESS05SERVING06REPORTSERP on AS400Sales · deliveries · clientsDepot price gapsAS400 · incrementalPublic fuel pricesOpen data · daily XMLCRM & SharePointJoined in Power BISQL Agent jobsScheduled dailySSIS packagesDownload · unzip · importSQL ServerStaging schemasIncremental loadsLast 12 monthsReference tablesProducts · stations · marginsSQL ServerDimensions · reporting viewsPrice mapPrices by segmentCompetitor gapsDaily & by periodMargin analysisPer depot & clientExcel extractsFor sales teams
  • Context

    An energy distributor wanted to see its fuel prices against competitors, station by station, and react faster to price gaps.

  • Build

    SSIS packages scheduled by SQL Agent pull sales and delivery data from the AS400 ERP and the public daily fuel-price feed into SQL Server staging, then into incremental reporting views.

  • Outcome

    Power BI price maps and competitor-gap reports went into production and replaced manual spreadsheet comparisons.

07 · 2021

Event-processing pipeline for a customer-journey analytics product

SaaS · big dataSpark · Vertica · MongoDB
01CLIENTS02INGEST03PROCESS04STORE05APPLICATIONClient event logsCSV / ZIP uploadsInsertion APIReceives event filesS3Raw event logsSpark jobsEvents · journeys · cycles · edgesReference dataUnits · event typesVerticaColumnar store · COPY INTOMongoDBApp metadataBack-office APIPython · JSONWeb appHTTPS · nginxApp usersWeb browser
  • Context

    A customer-journey analytics product had to turn large client event logs into journeys and cycles that its web app could query quickly.

  • Build

    An insertion API stores uploaded logs in S3. Spark jobs rebuild events, journeys, cycles and graph edges and bulk-load them into Vertica, while MongoDB keeps the app metadata.

  • Key choices

    A columnar analytical store for aggregation queries, and a clear split between batch processing and the API that serves the app.