ADR-0209accepted

S3-First Storage Migration

Context

ADR-0208 originally established MinIO as the local S3-compatible API backed by NAS storage. ADR-0212 supersedes that runtime choice with AIStor as the maintained local S3 runtime. The leverage analysis in Vault/Resources/minio-leverage-analysis.md identified 12 joelclaw subsystems still using ad-hoc filesystem, NFS, and SCP patterns.

Without a canonical object storage contract, storage behavior diverges across pipelines, retention policies stay manual, and AWS portability remains high-friction.

Decision

Migrate joelclaw storage to an S3-first pattern in four phases. Introduce a shared @joelclaw/object-store adapter package so every subsystem uses one object storage contract that behaves identically on local AIStor and AWS S3.

Shared Adapter: @joelclaw/object-store

Create packages/object-store/ with:

  • upload(bucket, key, body)
  • download(bucket, key)
  • list(bucket, prefix)
  • presign(bucket, key, expiry)
  • deleteObject(bucket, key)

Implementation contract:

  • SDK: @aws-sdk/client-s3
  • Local config: S3_ENDPOINT=https://aistor-s3-api.aistor:443 (or http://minio:9000 while rollback path is active)
  • Cloud config: omit S3_ENDPOINT and use native AWS resolution
  • Event payload object reference schema:
{ bucket, key, etag, size, contentType }

Bucket Taxonomy

Naming convention:

  • joelclaw-{env}-usw2-{domain}
  • lowercase, hyphenated, no dots, AWS-compatible

Initial bucket set:

  • joelclaw-local-usw2-media-raw
  • joelclaw-local-usw2-media-derived
  • joelclaw-local-usw2-docs-raw
  • joelclaw-local-usw2-docs-derived
  • joelclaw-local-usw2-session-archive
  • joelclaw-local-usw2-otel-archive
  • joelclaw-local-usw2-vault-backups
  • joelclaw-local-usw2-agent-loop-artifacts
  • joelclaw-local-usw2-discovery-raw

Lifecycle policy bands:

  • Hot mutable (7–30d TTL): loop temp artifacts, build caches
  • Warm replayable (90–365d TTL): session archives, discovery captures
  • Cold immutable (1y+): OTEL exports, Vault backups, release artifacts

Phase 1: Foundation (1–2 days)

  • Create @joelclaw/object-store
  • Create initial buckets and lifecycle policies
  • Move MinIO credentials to joelclaw secrets
  • Smoke test upload/download from worker pod

Phase 2: Archives First (3–5 days)

  • Session rotation writes to S3 instead of NFS copy
  • OTEL daily export writes gzipped NDJSON partitions to S3
  • Vault backup snapshots write to S3
  • Run dual-write (NAS + S3) during confidence window

Phase 3: Media + Docs (1–2 weeks)

  • Replace media pipeline SCP steps with S3 uploads
  • Update docs/pdf-brain events to accept objectRef alongside nasPath shim
  • Archive discovery raw payloads to S3 before Vault note creation

Phase 4: Runtime Artifacts (1 week)

  • Persist agent loop artifacts to S3
  • Publish build artifacts via S3
  • Back hot image OCI registry with MinIO (ADR-0206)

Storage Forecast

  • Year 1: 1–3 TB (media + hot images dominate)
  • Year 2: 3–8 TB
  • NAS capacity: 64 TB RAID5 (not a near-term constraint)

Consequences

Positive

  • Canonical S3 API across all subsystems
  • Same storage code path for local AIStor and AWS S3
  • NAS capacity exposed behind standard object API instead of ad-hoc paths
  • Lifecycle policies automate retention and cleanup
  • Cloud portability via endpoint/config swap only

Negative

  • S3 SDK becomes a shared dependency across more packages
  • AIStor adds one more service surface to monitor
  • NFS-backed MinIO latency is higher than native S3 for small objects
  • Dual-write window adds temporary operational complexity

Verification

  • @joelclaw/object-store package exists and compiles
  • Initial buckets are created with lifecycle policies
  • MinIO credentials are stored in joelclaw secrets (not manifest defaults)
  • Session rotation writes to S3
  • OTEL daily export writes to S3
  • Media pipeline uploads via S3 (no SCP)
  • Inngest events use the { bucket, key, etag, size, contentType } object reference schema

Related ADRs