A Fresh Data Platform on Google Cloud

The Question

A retailer’s data platform foundation squeaked and creaked. Analytical data went missing, but what made it disappear in the first place? Fullstaq was asked to review and rebuild the data ingestion solution.

Our Challenge

The data ingest solution, built by the retailer itself, had bugs, showed some errors, and offered insufficient insight into issues. So while we could just fix that, we quickly extended our scope to the entire data platform: CI/CD, version control - all the good stuff.

Services

Shopping for a Data Platform

With more and more people relying on online grocery shopping, supermarkets process enormous amounts of data to ensure timely, correct delivery. After all: no data, no delivery. Unfortunately, this retailer’s self-build data ingestion solution was wonky: the supermarket lost important insights because of data arriving later than expected. And if that isn’t frustrating enough, try spending days figuring out what’s wrong and hours debugging.

Fullstaq’s engineers reviewed the set-up and decided with the in-house data scientist that they wouldn’t stop there. To future-proof the supermarket’s e-commerce data solution, we designed and developed a new data platform on Google Cloud with serverless data warehouse BigQuery. In addition, we incorporated CI/CD (using GitHub Actions), version control, monitoring, re-usable components, and building blocks for data ingestion.

Lifting the Shop’s Data Processing Capabilities

The new platform has many advantages; take data collection, for example. Before, the supermarket could only batch process data. Now, they do that smoothly via Apache Airflow, and they can also stream process data through Pub/Sub and streaming components running on Kubernetes. The supermarket can analyze its data in real-time and act on it: prompt e-mails for abandoned shopping carts, swift recommendations, you name it.

We also designed and developed a standardized way to bring scheduled queries and analytical views to the cloud, which makes for some happy data scientists! And some jealous colleagues – quite a few company divisions are eager to adopt the new set-up, and Fullstaq is keen to help.