Workload Planner — Explicit Stages, DAG Support, File Decomposition

Status

proposed

Context

joelclaw workload plan auto-generates stages from templates based on shape. For simple work this is fine. For complex multi-step projects (ADR-0230: 10 steps, dependency DAG, mixed execution modes), the auto-generated stages are generic blobs that don’t capture the real work structure.

Additionally, packages/cli/src/commands/workload.ts is 4850 lines in one file. It contains types, utility functions, planning logic, dispatch logic, run logic, sandbox management commands, inference engines, guidance builders, and the CLI command definitions — all fused together.

Decision

Two changes:

1. Add --stages-from <file> to the planner

When provided, the planner uses the caller’s explicit stage definitions instead of auto-generating from templates.

Stage definition schema:

type ExplicitStage = {
  id: string;                          // "fix-admin-port"
  name: string;                        // human-readable title
  executionMode?: "manual" | "codex" | "sandbox" | "microvm" | "inline";
  dependsOn?: string[];                // stage IDs (DAG, not just linear)
  acceptance: string[];                // testable criteria per stage
  files?: string[];                    // paths this stage touches
  artifacts?: string[];                // expected outputs
  owner?: string;                      // "worker" | "reviewer" | "planner" | agent name
  tool?: "pi" | "codex" | "claude";
  timeout?: number;                    // seconds
  phase?: string;                      // grouping label ("A", "B")
  notes?: string;
};

File format: JSON array. Example:

[
  {
    "id": "fix-admin-port",
    "name": "Fix Restate admin reachability",
    "executionMode": "manual",
    "phase": "A",
    "acceptance": ["joelclaw jobs status reports restate healthy"],
    "files": ["k8s/restate.yaml"]
  },
  {
    "id": "build-rootfs",
    "name": "Build agent rootfs",
    "executionMode": "sandbox",
    "dependsOn": ["install-firecracker"],
    "phase": "B",
    "acceptance": ["microVM boots, tools work inside VM"],
    "files": ["infra/firecracker/"],
    "tool": "codex"
  }
]

DAG validation: Topological sort, cycle detection. Auto-infer shape from DAG topology (linear → serial, branches → parallel, mixed → chained).

Dependency gate in workload run: Refuses to run a stage whose dependsOn stages don’t have terminal inbox results.

2. Decompose workload.ts (4850 lines → ~5 files)

Current single file contains six distinct concerns:

Rules:

• workload.ts stays as the public entry point — exports workloadCmd
• All other files export pure functions, no CLI/Effect coupling
• Types shared across files go in workload-types.ts
• Each file is testable independently
• No circular imports — dependency flows: types ← utils ← plan/dispatch/run/stages ← workload.ts

Implementation

Phase 1: File decomposition (boy scout)

Extract types, utils, planning logic, dispatch logic, run logic into separate files. Zero behavior change. Verify with existing tests + bunx tsc --noEmit + pnpm biome check.

Phase 2: Explicit stages

Add workload-stages.ts with:

• JSON stage file parser + schema validation
• DAG cycle detection (Kahn’s algorithm)
• DAG shape inference (serial/parallel/chained)
• Critical path calculation

Wire into plan command via --stages-from option.

Phase 3: Dependency gate

In workload run, when the selected stage has dependsOn:

• Check inbox for terminal results of each dependency stage
• Refuse with clear error if any dependency is incomplete
• Report which dependencies are satisfied vs pending

Consequences

Positive

• Complex projects get first-class planner support with per-stage acceptance, execution modes, and dependencies
• 4850-line file becomes 5-6 manageable files (~500-1400 lines each)
• Each concern is independently testable
• DAG validation catches impossible dependency chains before execution
• Dependency gate prevents out-of-order stage execution

Negative

• More files to navigate (mitigated by clear naming and single entry point)
• Two planning modes (auto-generate vs explicit) to maintain
• Stage files are another artifact alongside ADRs and project notes

Risks

• Decomposition could break the existing test suite if imports aren’t updated carefully
• Stage files could become stale if the project evolves mid-flight