One of the biggest challenges companies are facing is setting up and maintaining processes that reliably collect and process data so you can extract value from it any time you want.
For collecting data we use Google Cloud components such as Cloud Pub/Sub (global message queue), Datastream, Google Cloud Dataflow (Apache Beam), Data Transfer Service (directly land data into BigQuery from SFDC and 100+ business apps), Ads Connector (Google Tentacles) and more.
Once data is collected, it is displayed in a unified database or platform also called Data Hub, Datawarehouse, or Datalake, Data Lakehouse, Data Vault, Data Mesh depending on the architecture. We apply monitoring on the data and the processes to ensure quality that is essential when automating many data sources. You can read more about this on our Maintenance page.
Once data is collected, it will be processed, unified, and transformed. One of the biggest challenges companies are facing is setting up and maintaining processes that reliably collect and transform data so you can extract value from it any time you want. We are very experienced and have extensive knowledge of the way to go.
We are experienced in ETL toolings such as Google Cloud Dataflow and the graphical version named Cloud DataFusion. Cloud Dataflow is an autoscaling ETL tool that unifies batch and streaming analytics. We also apply transformations in a data store such as Big Query, which is called ELT.
Customer identification by establishing an Identity Graph
A unique identifier can be used to link all of an individual customer's information from multiple sources so that it is accessible to all relevant teams and systems. We will build your company's unique Identity Graph to link and, as a result, keep track of all your customer interactions to be able to communicate one to one and to apply reliable measurement. Using Google Cloud Firestore or a Graph database, we build simple and sometimes complex unifications.
Google Cloud Dataprep is one of Google’s data services for cleaning and preparing structured and unstructured data for reporting, machine learning, and analysis. Dataprep interprets the data transformation through a proprietary inference algorithm. You can read the story on how we use Cloud Dataprep for our own needs in this article.
We help you pick the right database for your application. The choice depends heavily on your use case — transactional processing, analytical processing, in-memory database, and so on — but it also depends on other factors. There are different database options available within Google Cloud across relational (SQL) and non-relational (NoSQL) databases. Note that we don’t consider BigQuery as a database but as an Enterprise Data Warehouse / Analytical Datastore.
Broadly, if your data structure is not going to change much, select a relational database. Google Cloud uses Cloud SQL for any general-purpose SQL database and Cloud Spanner for large-scale globally scalable, strongly consistent use cases. In general, if your data structure may change later and if scale and availability are a bigger requirement then a non-relational database is a preferable choice. Google Cloud offers Firestore, Memorystore, and Cloud Bigtable to support a variety of use cases across the document, key-value, and wide column database spectrum. If you want to read more bout
