Observability & Data

39 Dataform definitions power the observability layer — all views, no materializations. Queries always read fresh data from nightly mirrors of Stripe snapshots. There is no stale cache to invalidate.

Metadata extraction follows a three-layer pattern: raw Stripe tables → _metadata_json (extracted JSON string) → _metadata (typed fields). Key sources are Stripe raw tables (invoices, subscriptions, items, prices) and the OPE store. Element classification tags each line item as one of: expected_in_stripe, bad_debt_item, deleted_from_ope, contract, delayed_upgrade, or missing_price. These classifications feed the drift detection and reconciliation views downstream.

Two separate drift systems operate independently — different data sources, different candidate selection, different remediation paths. Stripe drift compares OPE subscriptions against Stripe subscriptions. WWW drift compares zone plan levels in the www database against subs-api records. Both are batch-only: daily BigQuery snapshots. Drift that occurs between snapshots is invisible until the next day. New drift is identified by comparing today's snapshot against yesterday's — accounts with persistent drift are excluded from automatic remediation by design.

applystripe is the primary engine: a gather-plan-apply pipeline that reads account state from 7+ sources into a unified view, computes what the correct state should be, and applies the minimal diff. driftcontrol runs the daily cron with safety valves. auto_resync_www handles WWW drift. Enterprise account types (JDC, SFCC, IBM) are excluded — their state is set via contract paths and automatic remediation would break intended state.

An automated pipeline classifies BILL and CUSTESC Jira tickets weekly using the V6 taxonomy — 13 atoms organized into 6 molecules. The digest covers 27 weeks of data (~1,586 tickets) with a 96.3% classification rate. This is the primary signal connecting engineering decisions to customer pain.

The dunning brain's gather phase reads all six stores — subs_dunning, subs_account, subs_customer, billing_prod, cf_prod, and Stripe metadata — making it the most complete single view of account state available. For live investigation, triggering a gather on the account surfaces the full state diff.

NinjaPanel provides the support-facing view for customer accounts. BigQuery drift tables (cloudflare_account_inconsistencies_summary, cloudflare_inconsistent_bad_debt_accounts) surface accounts with cross-store disagreements. Stripe dashboard provides the authoritative payment history. Every healing attempt is logged to cfbill_migrations_audit with params, result, and failed boolean — check this table first when investigating why remediation didn't fire.