BigQuery Sync

A Drupal module for synchronizing entity data to Google BigQuery for reporting
and analytics purposes.

It ships with the synchronization framework only — no entity-specific plugins.
You add plugins (in your own module) describing which entities map to which
BigQuery tables, and the module handles change tracking, batching, schema
creation, upserts and deletes.

Features

BigQuery Sync is a framework for pushing Drupal content into a BigQuery dataset
and keeping it there, incrementally, without writing any of the plumbing
yourself. You declare what should end up in BigQuery; the module handles how it
gets there and stays current.

• Plugin system: define a plugin per entity type (optionally scoped to a bundle) to map it to a BigQuery table — one entity becomes one row.
• Incremental sync: entity changes are tracked via entity hooks and a content hash, so only new, changed and deleted entities are pushed on each run. There is no full-table dump on every cron.
• Derived "table sync" plugins: build junction or derived tables that don't map 1:1 to a single entity (for example a user × group × role membership table), reconciled per "dirty key" as the underlying entities change.
• Automatic table and schema management: tables are created from each plugin's declared schema. New columns are added and REQUIRED columns can be relaxed to NULLABLE automatically; destructive changes are never applied silently.
• Upserts and deletes: rows are merged on a configurable primary key, and deleted entities are removed from BigQuery rather than left as stale rows.
• Batch processing: large datasets are processed in configurable batches to keep memory and API usage predictable.
• Drush commands: a full command-line interface for sync, preview, status, targeted resyncs and dataset teardown — suitable for cron and CI.
• Status UI: an admin report showing per-plugin sync state, pending change counts and the entities awaiting sync.

Use this module when you want Drupal content queryable alongside the rest of
your data warehouse — powering Looker Studio dashboards, joining editorial
content against product or CRM data, feeding scheduled analytics queries, or
giving an analytics team access to content metrics without handing them
database credentials or building a bespoke reporting API. It is aimed at sites
where the reporting surface is wide enough that Views and a CSV export stop
being a reasonable answer.

Post-Installation

Refer to the documentation and follow the detailed steps to get started.

Additional Requirements

• The Composer-installed google/cloud-bigquery library (^1.28): composer require google/cloud-bigquery
• A Google Cloud project with the BigQuery API enabled, and a service account key with the BigQuery Data Editor and BigQuery Job User roles
• Outbound HTTPS access from your web/CLI environment to the Google Cloud APIs
• Drush, for the synchronization commands

The module depends only on Drupal core's User module beyond the above.

Similar projects

• BigQuery Integration - A more versatile approach to BigQuery integration, allowing much more than simply syncing entities. If you're looking for more complex solutions, looking to pull data from BigQuery, or simply to do more than pushing data, refer to BigQuery Integration instead.