[{"content":"An auditor picks a lot your site released fourteen months ago and asks a simple question: show me the batch record data exactly as it was when QA released this lot, and every change made to it since.\nSomeone on the team says that\u0026rsquo;s fine, Delta keeps history, we\u0026rsquo;ll use time travel. Then they run DESCRIBE HISTORY, and the table only goes back 30 days, because that\u0026rsquo;s the default delta.logRetentionDuration. The data files behind older versions are gone too. VACUUM keeps seven days of files by default, and on Unity Catalog managed tables, predictive optimization runs it for you. Nobody chose to delete the evidence. The platform cleaned up on schedule, exactly as designed.\nThat scenario is a composite, but the pattern isn\u0026rsquo;t rare: missing audit trails and unexplained data changes are recurring themes in FDA data-integrity findings. As regulated companies move more of their data onto lakehouses, auditors will follow it there.\nThe lesson is the thesis of this post: Delta history is a convenience feature, not a Part 11 audit trail. Databricks is GxP-capable. Being GxP-ready is a design decision you own.\nStart with the system of record Before any configuration, decide what the lakehouse is. In most GxP landscapes, the MES, LIMS or QMS is the system of record, with its own validated audit trail and electronic signatures. The pipeline\u0026rsquo;s job is to preserve that trail faithfully, carrying the source user, timestamp and reason for change through as columns, not to rebuild it from Delta logs. Almost every choice below follows from that.\nALCOA+ mapped to the lakehouse Principle What an auditor asks What I build on Databricks Attributable Who did it? Source user carried as a column; Unity Catalog identities and system.access.audit for platform actions Legible Can it be read in ten years? Open Delta and Parquet formats, column comments, schemas kept in Git Contemporaneous Was it recorded when it happened? Source and ingest timestamps on every row Original Is the first capture preserved? Append-only bronze, exactly as received Accurate Is it correct? Expectations that fail the update, CHECK and NOT NULL constraints Complete Is anything missing or doubled? Deduplication on event ID, reconciliation between systems Consistent Same answer every time? ACID writes, SCD Type 2 history instead of overwrites Enduring Will it last the retention period? Retention set on purpose, archives, an evidence layer you own Available Can you produce it on request? Queryable history, rehearsed retrieval, lineage IDs on every row 1. Retain on purpose Land every source message in bronze exactly as received, with the message ID, ingest timestamp and source metadata. Then make the table enforce that:\nALTER TABLE gxp.bronze.mes_events SET TBLPROPERTIES ( \u0026#39;delta.appendOnly\u0026#39; = \u0026#39;true\u0026#39;, -- Set both from your record-retention requirement, not from this example \u0026#39;delta.logRetentionDuration\u0026#39; = \u0026#39;interval 3650 days\u0026#39;, \u0026#39;delta.deletedFileRetentionDuration\u0026#39; = \u0026#39;interval 3650 days\u0026#39; ); delta.appendOnly blocks UPDATE and DELETE. VACUUM, including the runs predictive optimization triggers, honors the retention properties. Keeping GxP tables in their own catalog lets you treat maintenance there differently, including switching predictive optimization off with ALTER CATALOG gxp DISABLE PREDICTIVE OPTIMIZATION.\nThe trap\nTime travel needs both the transaction log and the data files. Retaining one without the other still loses the history.\nChange data feed is useful here, but treat it as a pipe, not an archive: its files are cleaned up by the same VACUUM. Read the feed and persist it into an audit table you control.\n2. Build history you can query Silver tables that receive corrections should keep every version instead of overwriting in place. Lakeflow\u0026rsquo;s AUTO CDC (formerly APPLY CHANGES) handles SCD Type 2 cleanly:\nCREATE OR REFRESH STREAMING TABLE batch_records_history; CREATE FLOW batch_records_scd2 AS AUTO CDC INTO batch_records_history FROM STREAM(gxp.bronze.batch_record_changes) KEYS (record_id) SEQUENCE BY source_changed_at STORED AS SCD TYPE 2; Now \u0026ldquo;show me this record as of release\u0026rdquo; is a filter on validity dates, with no dependency on time travel:\nSELECT * FROM gxp.silver.batch_records_history WHERE lot_id = \u0026#39;L-2025-0412\u0026#39; AND __START_AT \u0026lt;= TIMESTAMP \u0026#39;2025-08-14 16:05:00\u0026#39; AND (__END_AT IS NULL OR __END_AT \u0026gt; TIMESTAMP \u0026#39;2025-08-14 16:05:00\u0026#39;); Don\u0026rsquo;t set appendOnly on these tables. The history has to end-date rows.\n3. Make retries boring Message brokers deliver at least once. Picture a broker reconnect that redelivers a batch of material-consumption messages at the same moment a job retries after a timeout, and the ERP records the consumption twice. Reconciliation catches it before the lot is dispositioned, but only if reconciliation exists. In GxP, idempotency isn\u0026rsquo;t tidiness. It decides whether a system of record can be trusted.\nDeduplicate at the door on the source event ID, and make replayed micro-batches harmless with txnAppId and txnVersion:\nevents = ( spark.readStream.table(\u0026#34;gxp.bronze.mes_events\u0026#34;) .withWatermark(\u0026#34;ingested_at\u0026#34;, \u0026#34;2 hours\u0026#34;) .dropDuplicatesWithinWatermark([\u0026#34;event_id\u0026#34;]) ) def write_batch(df, batch_id): (df.write.format(\u0026#34;delta\u0026#34;) .option(\u0026#34;txnAppId\u0026#34;, \u0026#34;mes-consumption-silver\u0026#34;) .option(\u0026#34;txnVersion\u0026#34;, batch_id) # a replayed batch is skipped, not rewritten .mode(\u0026#34;append\u0026#34;) .saveAsTable(\u0026#34;gxp.silver.consumption\u0026#34;)) (events.writeStream .foreachBatch(write_batch) .option(\u0026#34;checkpointLocation\u0026#34;, \u0026#34;/Volumes/gxp/silver/checkpoints/consumption\u0026#34;) .start()) The call to the ERP is a side effect outside Delta, so it\u0026rsquo;s at-least-once whatever you do. Keep a posting ledger: every outbound posting gets an idempotency key and a status, and the job checks the ledger before calling and records the outcome after. Where the target allows it, write against a natural key so a repeat updates instead of duplicating. Then reconcile counts and quantities per lot on a schedule.\n4. Stop schema drift at bronze An upstream release renames a field or changes a unit of measure. mergeSchema absorbs it, the pipeline stays green, and downstream values are wrong. The auditor\u0026rsquo;s question, \u0026ldquo;was this change assessed before it reached production data?\u0026rdquo;, now has an uncomfortable answer.\nKeep automatic schema evolution out of silver and gold, and let Auto Loader rescue unexpected fields instead of evolving:\nraw = ( spark.readStream.format(\u0026#34;cloudFiles\u0026#34;) .option(\u0026#34;cloudFiles.format\u0026#34;, \u0026#34;json\u0026#34;) .option(\u0026#34;cloudFiles.schemaLocation\u0026#34;, \u0026#34;/Volumes/gxp/bronze/schemas/mes\u0026#34;) .option(\u0026#34;cloudFiles.schemaEvolutionMode\u0026#34;, \u0026#34;rescue\u0026#34;) # or \u0026#34;failOnNewColumns\u0026#34; .load(\u0026#34;/Volumes/gxp/landing/mes/\u0026#34;) ) Alert whenever _rescued_data is non-null. For critical fields, make expectations stop the pipeline rather than warn:\nCREATE OR REFRESH STREAMING TABLE consumption ( CONSTRAINT known_uom EXPECT (uom IN (\u0026#39;kg\u0026#39;, \u0026#39;g\u0026#39;, \u0026#39;L\u0026#39;, \u0026#39;mL\u0026#39;)) ON VIOLATION FAIL UPDATE, CONSTRAINT positive_qty EXPECT (quantity \u0026gt; 0) ON VIOLATION FAIL UPDATE ) AS SELECT * FROM STREAM(gxp.bronze.mes_events_parsed); Keep the expected schema in Git, so a change becomes a pull request with a validation run attached.\n5. Lock it down, then prove it Have a group own GxP tables and grant MODIFY only to the pipeline\u0026rsquo;s service principal, so nobody can change retention or run VACUUM from a notebook. Keep DDL, table properties included, in Git, and deploy with Declarative Automation Bundles through CI with an approval gate. No interactive edits in production. Check SHOW TBLPROPERTIES against expected values on a schedule, and watch system.access.audit for ALTER TABLE on GxP schemas. Once a quarter, rehearse the auditor\u0026rsquo;s request: pick an old lot and reproduce its record as of release. That rehearsal is your validation evidence. Where the platform stops Databricks gives you strong data-integrity mechanics, but it\u0026rsquo;s a data platform, not a GxP record system. Six gaps are yours to close:\nIntent. Delta records which identity ran an operation, usually a service principal, not a person\u0026rsquo;s reason for a change or a Part 11 signature. Keep corrections and signatures in the validated source systems, or behind an application that writes user, reason and signature to an append-only audit table. Never a raw UPDATE from a notebook. A platform that keeps moving. Runtimes and serverless compute change underneath you, and nobody can sell you a \u0026ldquo;validated\u0026rdquo; platform. Take a risk-based Computer Software Assurance approach: pin GxP jobs to an LTS runtime on classic compute, cover anything else with scheduled regression tests, and treat Databricks as an assessed supplier whose release notes someone reviews. Immutability below Delta. appendOnly only constrains access through Databricks. Make sure only the Unity Catalog managed identity can reach GxP storage, add resource locks and storage logging, and export archives to immutable (WORM) storage. Don\u0026rsquo;t put WORM on live Delta tables; Delta needs to clean up its own log. Evidence that expires. System tables, job run history, pipeline event logs and expectation results each have their own retention, none tied to yours. If an auditor might ask for it in three years, copy it into a table you own. Lineage at the boundary. Unity Catalog lineage doesn\u0026rsquo;t see the broker or the ERP, the two ends an auditor cares about most. Carry the source message ID and pipeline run ID on every row, so tracing a transaction is a join. Periodic review and recovery. Generate quarterly access and audit-trail exception reports and sign them in your QMS. Schedule DEEP CLONE copies and test restores; time travel and UNDROP cover short windows, not recovery. Checklist The system of record is named, and its audit trail is carried through as data Bronze is append-only, with retention set from your requirement Every corrected table keeps SCD Type 2 history Deduplication on event ID, a posting ledger and per-lot reconciliation No schema auto-evolution past bronze; critical expectations fail the update Retention, grants and DDL under change control, with drift alerts An evidence layer you own, and a quarterly retrieval rehearsal The goal isn\u0026rsquo;t \u0026ldquo;we configured it correctly.\u0026rdquo; It\u0026rsquo;s \u0026ldquo;we can prove how it was configured on any date you ask about.\u0026rdquo; That\u0026rsquo;s the difference between a platform that\u0026rsquo;s GxP-capable and a pipeline that\u0026rsquo;s GxP-ready.\n","permalink":"https://ethanparmer.dev/posts/gxp-ready-pipelines-on-databricks/","summary":"\u003cp\u003eAn auditor picks a lot your site released fourteen months ago and asks a simple question: \u003cem\u003eshow me the batch record data exactly as it was when QA released this lot, and every change made to it since.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eSomeone on the team says that\u0026rsquo;s fine, Delta keeps history, we\u0026rsquo;ll use time travel. Then they run \u003ccode\u003eDESCRIBE HISTORY\u003c/code\u003e, and the table only goes back 30 days, because that\u0026rsquo;s the default \u003ccode\u003edelta.logRetentionDuration\u003c/code\u003e. The data files behind older versions are gone too. \u003ccode\u003eVACUUM\u003c/code\u003e keeps seven days of files by default, and on Unity Catalog managed tables, predictive optimization runs it for you. Nobody chose to delete the evidence. The platform cleaned up on schedule, exactly as designed.\u003c/p\u003e","title":"GxP-ready pipelines on Databricks: an ALCOA+ map"},{"content":"I\u0026rsquo;m Ethan Parmer, a Lead Data Engineer who builds on Databricks for industries where data has to be right the first time: GxP-regulated healthcare and life sciences, and finance.\nWhat I do Most of my work lives in the lakehouse: medallion architectures, Delta Lake, Unity Catalog and pipelines that are validated, traceable and ready for an audit. In regulated work \u0026ldquo;it ran\u0026rdquo; isn\u0026rsquo;t enough. Every record needs a lineage, every change needs a reason, and every number has to tie out. I like that pressure. It rewards people who think carefully and build things properly.\nWhat drives me I hunger for a challenge and I\u0026rsquo;m always learning. If there\u0026rsquo;s a messy migration, a compliance puzzle or a pipeline nobody wants to touch, that\u0026rsquo;s usually where you\u0026rsquo;ll find me. This site is where I write up what I learn along the way.\nSay hello Find me on GitHub or LinkedIn, or follow the writing via RSS.\n","permalink":"https://ethanparmer.dev/about/","summary":"\u003cp\u003eI\u0026rsquo;m Ethan Parmer, a Lead Data Engineer who builds on \u003cstrong\u003eDatabricks\u003c/strong\u003e for industries where data has to be right the first time: GxP-regulated healthcare and life sciences, and finance.\u003c/p\u003e\n\u003ch2 id=\"what-i-do\"\u003eWhat I do\u003c/h2\u003e\n\u003cp\u003eMost of my work lives in the lakehouse: medallion architectures, Delta Lake, Unity Catalog and pipelines that are validated, traceable and ready for an audit. In regulated work \u0026ldquo;it ran\u0026rdquo; isn\u0026rsquo;t enough. Every record needs a lineage, every change needs a reason, and every number has to tie out. I like that pressure. It rewards people who think carefully and build things properly.\u003c/p\u003e","title":"About"},{"content":"Things I\u0026rsquo;ve built. Each one links to its code or a write-up.\nethanparmer.dev This site. Built with Hugo and the PaperMod theme, deployed to GitHub Pages by GitHub Actions on every push.\nSource on GitHub\n","permalink":"https://ethanparmer.dev/projects/","summary":"\u003cp\u003eThings I\u0026rsquo;ve built. Each one links to its code or a write-up.\u003c/p\u003e\n\u003ch2 id=\"ethanparmerdev\"\u003eethanparmer.dev\u003c/h2\u003e\n\u003cp\u003eThis site. Built with \u003ca href=\"https://gohugo.io/\"\u003eHugo\u003c/a\u003e and the PaperMod theme, deployed to GitHub Pages by GitHub Actions on every push.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/LordParmer/ethanparmer.github.io\"\u003eSource on GitHub\u003c/a\u003e\u003c/p\u003e","title":"Projects"}]