git-drs

AnVIL + Terra meets git-drs

Version-controlled dataset layouts—without copying controlled-access payloads into Git.

Reference proof of concept · July 2026

Git moves references. AnVIL moves bytes.

User A Git User B AnVIL
Add references with ADC-authorized metadata validation Commit canonical DRS URI, size, and checksum Clone and authenticate with an independent Google identity Hydrate from a fresh authorized URL and verify bytes

0 payload bytes

committed to Git

2 identities

authenticate independently

1 reference

stays canonical and portable

Security boundary: Git visibility reveals paths and DRS identifiers; it never grants authorization to the underlying data.

One reference model, four perspectives

Data reference author

Publish a reviewable, versioned dataset layout—one object or a manifest—without downloading payloads.

Data consumer

Clone pointers, authenticate independently, then hydrate everything or only paths matching a pattern.

Unauthorized reader

Inspect repository history but receive a clear denial when attempting controlled-data hydration.

Repository maintainer

Review deterministic pointers, diagnose provider failures, and run clean-clone acceptance tests.

Out of scope: uploads, record mutation, provider copying, remote GC, and workspace-table sync.

Add once. Publish with ordinary Git.

```bash gcloud auth application-default login git drs add-ref --remote anvil \ drs://authority/object-1 data/sample.cram # Validate a batch before writing git drs add-ref --remote anvil \ --manifest references.tsv --dry-run git add .git-drs/ .gitattributes data/ git commit -m "Add AnVIL data references" git push ```
### Published - Canonical DRS URI pointers - Paths and Git history - `.gitattributes` - Allowlisted, non-secret remote configuration ### Never published - Payload bytes or cache content - Tokens, headers, or ADC files - Signed download URLs

A clean clone is enough

```bash gcloud auth application-default login git clone cd # Hydrate every authorized reference git drs pull # Or only one dataset slice git drs pull -I "data/*.cram" ``` > No author cache, local Git config, token, or signed URL is transferred. </div>
| Stage | Action | |---|---| | **Git** | Clone pointer and safe public config | | **User B** | Supply ADC; choose all or include pattern | | **Resolver** | Fetch current metadata and fresh access | | **Cache** | Download, verify, atomically promote | | **Worktree** | Hydrate only after validation |
</div> --- ## AnVIL manifest to pointer-only GitHub repo
### 1. Configure and add references ```bash git init git drs remote add terra anvil \ --drs-endpoint https://data.terra.bio \ --auth google-adc --mode read-only # Download the TSV from AnVIL Data Explorer first. manifest=/tmp/anvil-manifest-38dc7537.tsv scripts/anvil-add-ref-commands.sh "$manifest" \ > /tmp/add-anvil-refs.sh cat /tmp/add-anvil-refs.sh bash /tmp/add-anvil-refs.sh ```
### 2. Commit, hydrate, and publish ```bash git add .git-drs/ .gitattributes '*.tsv' git commit -m "Add references to AnVIL data" # Materialize only TSVs locally. git drs pull -I "*.tsv" git status git branch -M main git remote add origin \ https://github.com/bwalsh/\ ANVIL_1000G_PRIMED_data_model.git git push -u origin main ```
> **Result:** the worktree contains hydrated TSV data; GitHub contains only DRS pointers and safe public configuration. --- ## Portable pointer + safe configuration
### Tracked pointer ```text version https://calypr.github.io/spec/v1 oid drs://authority/object-1 size 987654321 sha256 8d969eef… ``` The DRS URI stays canonical. A checksum describes content; it does not identify the AnVIL record.
### Tracked `.git-drs/config.yaml` ```yaml version: 1 default_remote: anvil remotes: anvil: type: terra endpoint: https://data.terra.bio auth: google-adc mode: read-only ``` Only allowlisted public settings load. Secret-like or unknown sensitive fields fail closed.
--- ## Provider-neutral resolution breaks Syfon coupling ```text Commands Resolver interface Identities add-ref · pull · ping -> GetObject · GetAccess -> remote · cache · content | +----------+----------+ | | AnVILResolver SyfonResolver Google ADC · routing Gen3 · local behavior ``` ### Design rules - Commands depend on one resolver contract. - The configured remote selects the resolver. - Authentication and provider routing stay behind the interface. - There is no user-facing `--remote-type` switch. --- ## Three identities—not one overloaded OID
REMOTE

Canonical DRS URI

Normalized provider record identity. Committed to Git and used for object and access resolution.

LOCAL

Cache OID

Filesystem-safe SHA256 of a versioned prefix plus normalized DRS URI. Never replaces remote identity.

CONTENT

Content SHA256

Durable integrity metadata, when supplied. Used to verify bytes—not to locate the record.

```text cache_oid = sha256("git-drs-anvil-ref:v1\n" + normalized_drs_uri) ``` > **Download invariant:** fresh access → temporary file → verify size/checksum → atomic cache promotion → hydrate worktree. --- ## Good component coverage; one decisive gap
IMPLEMENTED

Focused coverage

  • Resolver contract and errors
  • Terra remote and safe config
  • Canonical pointer/cache key
  • Terra ping and push refusal
PARTIAL

Workflow coverage

  • Manifest validation and dry run
  • Pull size/SHA256 checks
  • Selective hydration
  • Cache reuse behavior
MISSING

POC acceptance

  • Independent User A/User B state
  • Real or contract-faithful AnVIL
  • Expired URL and retry journey
  • Authorization leak audit
| Arrange | Act | Assert | |---|---|---| | Separate homes, config, ADC, caches | Commit → clone → authenticate → pull | Verified bytes; isolated credentials | --- ## Most original blockers are now closed
### Implemented - ADC-backed `AnVILResolver` handles metadata and access. - Terra `add-ref` and pull use provider-neutral resolution. - Remote config selects behavior; `--remote-type` is deprecated. - Terra pointers retain DRS URI, size, and optional SHA256. - Cache keys are separate from content checksums.
### Still incomplete - Compact AnVIL IDs still require the trusted Terra resolver. - Service-info does not discover auth, provider, or capabilities. - Remote setup retains provider-specific command shapes. - Downloads lack bounded concurrency, retry, cancellation, and expired-URL re-resolution. - Production and independent two-user acceptance remain unverified.
> **Bottom line:** an implemented vertical slice still needs production-contract validation, resilience, and end-to-end proof. --- ## Turn the vertical slice into a proven POC
P0 · VERIFY

Prove production fit

  • Confirm endpoint and OAuth scopes
  • Certify object/access contracts
  • Test slash and compact DRS IDs
  • Run clean two-user clone/pull
  • Audit logs and history
P1 · HARDEN

Make it resilient

  • Trusted authority routing
  • Persist auth discovery
  • Expired-URL re-resolution
  • Bounded retry and concurrency
  • Remote diagnostics and CI
P2 · GENERALIZE

Keep DRS composable

  • Unify `remote add`
  • Resolver/auth interfaces
  • CGC and Synapse fixtures
  • Keep publishing provider-specific
  • Defer mutation and copying
--- ## Decisions needed to start the POC 1. Which authoritative AnVIL resolver contract, endpoint, and OAuth scopes will we certify? 2. What stable test objects cover checksummed, non-checksummed, large, and denied cases? 3. Is the pointer extension compatible with every parser that must read it? 4. What is the supported override policy for tracked versus local remote configuration? 5. Who owns the independent-user acceptance environment and compatibility matrix? ### Questions? The maintained design source is [`docs/anvil-terra-poc.md`](/git-drs/anvil-terra-poc.html).