Setup Guide

Follow the steps below to get fully set up. Note the requirements below before getting started:

• The Composer-installed google/cloud-bigquery library (^1.28): composer require google/cloud-bigquery
• A Google Cloud project with the BigQuery API enabled, and a service account key with the BigQuery Data Editor and BigQuery Job User roles
• Outbound HTTPS access from your web/CLI environment to the Google Cloud APIs
• Drush, for the synchronization commands

1. Set up Google Cloud.

• Create a Google Cloud project.
• Create a service account with the BigQuery Data Editor and BigQuery Job User roles.
• Download the service account key JSON file.
• Create a BigQuery dataset for your Drupal data (the module will also create it on first connect if it does not exist).

2. Configure authentication.

The module uses Application Default Credentials via the standard GOOGLE_APPLICATION_CREDENTIALS environment variable. Point it at the absolute path of your service account JSON key file:

GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

Never commit the key to your repository. Store it per environment, outside the web root, and keep it out of version control. For local testing, use a key that does not have access to sensitive data.

3. Configure the module.

Navigate to Configuration → Web services → BigQuery Sync (/admin/config/services/bigquery) and enter your Google Cloud Project ID and BigQuery Dataset ID, then click Test Connection. The form reports whether GOOGLE_APPLICATION_CREDENTIALS was found before it will let you test. Batch size (default 100) and log level are also set here.

All four settings — project_id, dataset_id, batch_size and log_level — can be overridden per environment in settings.php. Overridden values are shown in the form as read-only with a warning, so the UI always reflects what is actually in effect.

4. Write your sync plugins.

Entity sync plugins map one entity type (optionally one bundle) to one BigQuery table. Create them in your own module under src/Plugin/bigquery_sync/entity/:

namespace Drupal\my_module\Plugin\bigquery_sync\entity;

use Drupal\Core\Entity\EntityInterface;
use Drupal\Core\StringTranslation\TranslatableMarkup;
use Drupal\bigquery_sync\Attribute\EntitySync;
use Drupal\bigquery_sync\Plugin\EntitySyncBase;

#[EntitySync(
  id: 'article_sync',
  label: new TranslatableMarkup('Article Sync'),
  entity_type: 'node',
  table_name: 'articles',
  primary_key: 'entity_id',
  priority: 50,
  bundle: 'article',
)]
class ArticleSync extends EntitySyncBase {

  /**
   * {@inheritDoc}
   */
  public function getTableSchema(): array {
    return [
      'fields' => [
        ['name' => 'entity_id', 'type' => 'INTEGER', 'mode' => 'REQUIRED'],
        ['name' => 'title', 'type' => 'STRING', 'mode' => 'REQUIRED'],
        ['name' => 'created', 'type' => 'DATETIME', 'mode' => 'NULLABLE'],
        ['name' => 'synced_at', 'type' => 'DATETIME', 'mode' => 'REQUIRED'],
      ],
    ];
  }

  /**
   * {@inheritDoc}
   */
  public function transformEntity(EntityInterface $entity): array {
    return [
      'entity_id' => (int) $entity->id(),
      'title' => $entity->label(),
      'created' => $this->formatDatetime($entity->get('created')->value),
    ];
  }
}

Notes:

• Always include a synced_at DATETIME column in the schema. The sync manager stamps it on every row automatically — your transformEntity() should not.
• primary_key is the column used for the upsert MERGE (defaults to entity_id).
• EntitySyncBase provides helpers from BigQueryFieldHelpersTrait: getLabelInLanguage(), referencedLabels(), getBundleLabel(), getListFieldLabel(s)(), formatDate() and formatDatetime().

Table sync plugins build derived tables that do not map 1:1 to a single entity — for example a junction table of (user, group, role) rows assembled from group memberships. They reconcile per "dirty key": an opaque string identifying a unit of the table. Create them under src/Plugin/bigquery_sync/table/.

The reconcile contract for a dirty key:

• buildRowsForKey($key) — the rows that SHOULD exist for the key (empty = none).
• getKeyConditions($key) — the column => value map identifying the key's rows, used to delete rows that should no longer exist.
• getAllKeys() — every key currently present in Drupal (for initial backfill).
• getDirtyKeysForEntity($entity) — given a changed entity, return the dirty keys it affects. The module's entity hooks call this on every insert, update and delete for each table sync plugin, so your derived table stays incrementally in sync. Return an empty array for entities the plugin doesn't care about.

#[TableSync(
  id: 'team_sync',
  label: new TranslatableMarkup('Team Sync'),
  table_name: 'team',
  primary_key: ['user_id', 'group_id', 'role_id'],
  priority: 50,
)]
final class TeamSync extends TableSyncBase {

  public function getDirtyKeysForEntity(EntityInterface $entity): array {
    // Inspect $entity; return e.g. ["{$uid}:{$gid}"] for changes you care about.
    return [];
  }

  // ... getTableSchema(), buildRowsForKey(), getKeyConditions(), getAllKeys()
}

bqsync:sync-tables auto-seeds all keys on a table's first run, so the initial dataset is built without a separate backfill step.

5. Run a sync.

Sync all configured entities:

drush bqsync:sync

Sync specific plugins:

drush bqsync:sync --plugins=user_sync,node_sync

Other commands (all also available under the bigquery-sync: prefix):

• bqsync:status — show sync status for all plugins.
• bqsync:test — test the BigQuery connection.
• bqsync:preview — preview what would change without syncing.
• bqsync:clear-tracking — clear tracking data (forces re-sync).
• bqsync:delete-table [table] — delete one table from the dataset.
• bqsync:wipe-dataset — delete all tables in the dataset.
• bqsync:sync-reset [plugin] — wipe a plugin's table and tracking.
• bqsync:resync [plugin] — wipe and fully resync a plugin.
• bqsync:sync-tables — reconcile table sync plugins.
• bqsync:backfill-tables — mark all table sync keys dirty.
• bqsync:resync-table [plugin] — wipe and rebuild a table sync plugin.

6. Schedule it.

Run the sync commands from your scheduler. Syncs are incremental, so a nightly run during off-peak hours is usually enough; increase the frequency if your reporting needs fresher data. Example cron entry:

0 2 * * *  drush bqsync:sync && drush bqsync:sync-tables

7. Monitor.

Visit /admin/config/services/bigquery/status for a per-plugin status report, including pending change counts and drill-down into the entities awaiting sync.

Schema types. Tables are created automatically from each plugin's getTableSchema(). Use BigQuery's standard types: INTEGER, FLOAT, BOOLEAN, STRING, BYTES, DATE, DATETIME, TIMESTAMP. Repeated (array) columns use 'mode' => 'REPEATED'. Schema changes are applied automatically where safe — new columns are added and REQUIRED columns can be relaxed to NULLABLE. Column removal and incompatible type changes are never performed automatically; use bqsync:resync [plugin] to rebuild a table after that kind of change.

Troubleshooting. If a sync fails, verify the service account key file path and permissions, confirm the BigQuery API is enabled in the Google Cloud Console, and check that outbound HTTPS is allowed from your server. Run drush bqsync:test to validate connection and configuration.