Data Platform to Improve Data Access and Analysis

The Challenge

Rituals needed help to get the most out of their data. The company's data was scattered in different systems and formats, making it difficult for users to access, find and analyze the data they needed. Employees heavily relied on data specialists due to improper structure, naming conventions and documentation. The pressing need was to establish an organized and structured data platform.

The Solution

Crystalloids helped Rituals to implement a data platform similar to a “library” so that people can easily find the information they need.

Initially, Crystalloids and Rituals formed smaller groups dedicated to specific aspects of the library's concept: GDPR compliance, data retention, integrations, security, data structure, and naming conventions. After several months, these groups merged into a cohesive data platform team.

The challenge we faced was how to structure the data effectively. Firstly, we considered using domains as dividers, but this approach only sometimes applied, as specific data spanned multiple domains. Another idea involved using data sources at the project level as dividers, but the vast number of data sources made this impractical. The primary objective was to create a user-friendly system for data retrieval.

Finally, to organize the data lake we built the data platform on Google Cloud Platform using technologies such as Data Fusion, DataFlow, Dataform, and BigQuery, organized into three layers:

• System layer: This layer contains the raw data from the different data sources. Rituals has a variety of data sources, including point-of-sale systems, e-commerce platforms, customer relationship management (CRM) systems and more. The system layer is responsible for loading the raw data from these sources into BigQuery with minimal transformation.
• Domain layer: This layer contains curated data tailored to the needs of specific business domains. For example, the domain layer might have data about Rituals' customers, products, or supply chain. The domain layer is created using Dataform, a service for data analysts to develop, test, version control, and create complex SQL workflows for data transformation in BigQuery. 
• Team layer: This layer contains data specific to individual teams or projects. Teams use this layer to save their work for analysis and validation and to query data from the system and domain layers for discovery and exploration. The team layer is using BigQuery datasets and projects.

The data platform is designed to be flexible and scalable,  accommodating Rituals' growing data needs. The data platform is also designed to be user-friendly so that users from different business domains can easily access and use the data they need.

Data Lake with Advanced Technologies

To build the Data Lake we utilized a suite of Google Cloud Platform technologies:

• BigQuery: Served as the core Data Warehouse, hosting both structured and curated data. BigQuery was pivotal in enabling data sharing across multiple teams and feeding into various systems and pipelines.
• BigQuery Studio: Used extensively for data analysis, discovery, and preparation. It was instrumental in facilitating data inputs for AI and dashboarding purposes.
• Dataform: This tool was essential in creating SQL workflows, managing dependencies across different data stages and layers, ensuring a streamlined data flow.
• Policy Tags: Implemented for robust security and access management, particularly at the column level within datasets.
• Composer: Within the Data Platform, Composer orchestrated data flows, establishing dependencies between data uploads, Dataform flows, and other data operations.
• Dataflow: Employed for both ELT and ETL jobs, Dataflow was critical in translating and curating data into structured, defined formats.
• DataFusion: Like Dataflow, DataFusion was used for simpler EL jobs, enhancing our data processing capabilities.
• Cloud Functions: These were utilized for triggering data pipelines based on specific events.
• Storage: Google Cloud Storage served as the primary landing zone for input files like CSVs and also functioned as backup and archival storage.
• DLP (Data Loss Prevention): Our team used DLP to scan, discover, and classify sensitive data, applying appropriate policy tags for enhanced data security.
• Looker Studio: Used for creating monitoring dashboards to identify issues related to data incompleteness or inconsistencies.
• Terraform: This tool was critical for resource creation and management within GCP, including projects, datasets, service accounts, and storage buckets.

Looking ahead, we plan to integrate Dataplex with DataCatalog to enhance the documentation and management of data in BigQuery. This integration will further streamline data accessibility and governance within Rituals, fostering a more robust and efficient data ecosystem.