Data Platform on Google Cloud for Enhanced Analysis and Informed Decision-Making

Crystalloids partnered with another company to tackle a significant challenge: the lack of a centralized data warehouse, which hindered proper analysis for the clients' data scientists. 

Throughout our rewarding four-year partnership with the retailer, our dedicated team embarked on a mission to engineer a sophisticated data warehouse capable of integrating data from 25 different sources. We also created specialized cloud functions to facilitate data sharing across various departments, enabling them to directly access information from the warehouse. Moreover, Crystalloids adeptly integrated client's existing solutions with BigQuery to harness the power of data for thorough analysis.

Working seamlessly with systems outside of Google Cloud, such as Salesforce, our team efficiently extracted all the necessary data using Google Cloud Composer. In instances where connectors were unavailable, we expertly scripted the integration between applications and BigQuery.

As our work progressed, the retailer's ever-expanding tool integration requests posed new challenges. Each time we completed an integration, they introduced another application to their portfolio, necessitating our expertise in seamless incorporation. Throughout these integrations, we prioritized data quality by conducting regular maintenance and quality tests.

The Goal

Create a unified customer view to facilitate (predictive) analysis and a flexible, scalable and portable platform supporting added-value services for restaurateurs. 

The Process

To expedite results while ensuring efficiency, Crystalloids adopted a two-stage approach: data collection and data implementation for consumption. During the data collection phase, we meticulously set up an architecture and processes to gather information from various sources, including CloudSQL instances, third-party applications with APIs (e.g., Salesforce), flat files (CSV, JSON, etc.), and data published on Pub/Sub topics. The extracted data was then ingested into BigQuery using solutions like Cloud Composer (Airflow), Dataflow, and Python Scripts. This streamlined data became readily accessible to data scientists using Vertex AI, BI specialists leveraging Tableau, and business analysts.