feat(run-ops): webapp write path — trigger/batch minting, idempotency routing, run lifecycle

[d-cs]

What

Routes the webapp write path through the run-ops split seam: trigger/batch minting, idempotency-key resolution, and the run-lifecycle services now determine residency and dispatch writes to the correct store.

• Trigger & batch (runEngine/services/triggerTask.server.ts, batchTrigger.server.ts, createBatch.server.ts, streamBatchItems.server.ts, v3/services/batchTriggerV3.server.ts): mint ids with the run-ops-aware minting and route creation/streaming through the store; batch children inherit the parent's residency.
• Idempotency (runEngine/concerns/idempotencyKeys.server.ts + new idempotencyResidency.server.ts): idempotency-key lookup/dedup is residency-aware so a keyed retrigger resolves against the store that owns the original run.
• Run lifecycle services (createCheckpoint, createTaskRunAttempt, enqueueDelayedRun, expireEnqueuedRun, finalizeTaskRun, resumeBatchRun, cancelDevSessionRuns, executeTasksWaitingForDeploy, triggerFailedTask): resolve their target run through the store rather than a fixed client.
• Reads that fan out from writes (runsRepository + clickhouseRunsRepository, BulkActionV2 + batch read-through, realtime sessions/runReader, alerts deliverAlert/performTaskRunAlerts): route through the read-through resolver.
• 9535ae63d — resolves the parent run through an injectable run store in TriggerFailedTaskService.
• bf8f7c881 — drops the "known-migrated" concept from write-path and read repos; residency is id-shape only.
• 515b897ea — self-defaults resolveWaitpointThroughReadThrough to the safe run-ops clients.

Why

PR6 of the run-ops split stack. This is the write-path counterpart to the read foundation in the previous PRs: with it in place, both reads and writes route through the seam. Additive when the split is disabled (id-shape resolution collapses to the control-plane client); behavior-changing on the minting, idempotency, and lifecycle paths when enabled.

Tests

Large new/expanded vitest suite under apps/webapp/test/ and colocated service tests: trigger-task and batch-trigger store routing, residency inheritance, idempotency dedup residency + legacy-authority, bulk-action read routing, cancel-dev-session routing, alerts store routing, runs-repository read-through, realtime session/run-reader read-through and stream-registration routing, and the waitpoint read-through default. Testcontainers-backed; no mocks.

Notes

Draft, stacked on #4117 (runops/pr05-webapp-foundation). Review that first; this diff is against it.

Server-change / changeset note to be added at stack-assembly time.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: aa55b6b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 20c6d1a3-4024-44ce-843c-9b22910baac1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

This PR migrates numerous webapp services from direct Prisma queries to a RunStore/controlPlaneResolver abstraction, enabling split reads/writes between legacy and new ("run-ops") Postgres databases. It introduces residency-aware idempotency dedup client resolution, waitpoint read-through resolution, and mint-kind-based friendly ID generation (ksuid vs cuid) for run, batch, and failed-task creation that preserves parent/child residency lineage. Trigger, batch trigger, alert, session serialization, run listing, and cancellation services are updated to resolve environments and locked workers via controlPlaneResolver instead of eager Prisma includes, and to read/write batch and run records through RunStore methods rather than direct Prisma calls. Extensive new integration tests validate store routing behavior across heterogeneous Postgres containers (legacy vs new), residency inheritance, and passthrough single-database scenarios.

Changes

Area	Change
Idempotency	New resolveIdempotencyDedupClient resolves dedup client by split mode/mint kind/parent residency; wired into IdempotencyKeyConcern
Waitpoints	New resolveWaitpointThroughReadThrough helper for read-through waitpoint lookups
Trigger services	RunEngineTriggerTaskService and TriggerFailedTaskService mint friendly IDs (ksuid/cuid) based on residency inheritance; RunStore injectable
Batch services	RunEngineBatchTriggerService, CreateBatchService, StreamBatchItemsService, BatchTriggerV3Service migrated to runStore/controlPlaneResolver for batch/item persistence and friendly-id minting
Lifecycle services	createTaskRunAttempt, enqueueDelayedRun, expireEnqueuedRun, executeTasksWaitingForDeploy, finalizeTaskRun, deliverAlert, performTaskRunAlerts, resumeBatchRun, createCheckpoint, cancelDevSessionRuns resolve environment/lock via controlPlaneResolver/runStore
Reads/repositories	runsRepository, clickhouseRunsRepository, sessions.server, dashboardAgent, runReader, BulkActionV2 use runStore-based hydration/read-through seams
Project deletion	deleteProject.server.ts invokes RunOpsCascadeCleanupService.cleanupProject
Tests	Extensive new unit and heterogeneous-database integration tests validating store routing, residency inheritance, and passthrough behavior

Related PRs: None identified.

Suggested labels: run-engine, database, testing

Suggested reviewers: matt-aitken, ericallam

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is useful but does not follow the required template and is missing Closes, checklist, Changelog, and Screenshots sections.	Add the required template sections (Closes #..., checklist, Changelog, Screenshots) and keep the current What/Why/Tests summary.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and matches the main write-path routing changes in the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch runops/pr06-write-path

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 1 potential issue.

[pkg-pr-new]

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@a8068e9

trigger.dev

npm i https://pkg.pr.new/trigger.dev@a8068e9

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@a8068e9

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@a8068e9

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@a8068e9

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@a8068e9

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@a8068e9

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@a8068e9

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@a8068e9

commit: a8068e9

[coderabbitai]

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

apps/webapp/app/runEngine/concerns/idempotencyKeys.server.ts (1)

245-294: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

blockRunWithWaitpoint still writes via this.prisma, not the resolved dedupClient.

dedupClient (computed above at line 155-174) is derived using parentRunFriendlyId: request.body.options?.parentRunId — the exact same parentRunId used here to block the parent run's waitpoint (line 246, 279). dedupClient is precisely the client that owns this parent run's residency, yet the transaction at line 290 still passes tx: this.prisma, the (possibly wrong) fallback client.

If split mode is enabled and the parent run resides on the "new" store while this.prisma targets the legacy store (or vice versa), this write would target the wrong database, failing to find the parent run row or silently writing state to a store that doesn't own it — contradicting the PR's core objective of routing writes to the store that owns the target run.

🐛 Proposed fix

             await this.engine.blockRunWithWaitpoint({
               runId: RunId.fromFriendlyId(parentRunId),
               waitpoints: associatedWaitpoint!.id,
               spanIdToComplete: spanId,
               batch: request.options?.batchId
                 ? {
                     id: request.options.batchId,
                     index: request.options.batchIndex ?? 0,
                   }
                 : undefined,
               projectId: request.environment.projectId,
               organizationId: request.environment.organizationId,
-              tx: this.prisma,
+              tx: dedupClient,
             });

🧹 Nitpick comments (8)

apps/webapp/app/runEngine/concerns/resolveWaitpointThroughReadThrough.server.ts (1)

44-49: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Consider forwarding logger/onLegacyReplicaRead for parity with other read-through consumers.

ReadThroughDeps supports logger and onLegacyReplicaRead (saturation-signal hook), but ResolveWaitpointDeps/this wrapper drop both, so legacy-replica reads for waitpoints won't emit the saturation signal that other read-through call sites presumably rely on for monitoring split-read health.

apps/webapp/app/runEngine/services/batchTrigger.server.ts (2)

92-99: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Discards the minted id and re-derives it via BatchId.fromFriendlyId.

mintBatchFriendlyId returns { id, friendlyId }, but only friendlyId is kept here; id is recomputed later via BatchId.fromFriendlyId(batchId) (lines 175, 266). createBatch.server.ts uses the returned id directly instead of re-deriving it. Functionally likely equivalent if BatchId.fromFriendlyId is a lossless decode, but it's redundant work and an inconsistency between two services from the same PR doing the same job.

♻️ Proposed consistency fix

-          const { friendlyId } = await mintBatchFriendlyId({
+          const { id, friendlyId } = await mintBatchFriendlyId({
             environment: {
               organizationId: environment.organizationId,
               id: environment.id,
               orgFeatureFlags: environment.organization.featureFlags,
             },
             parentRunFriendlyId: body.parentRunId,
           });

Then thread id through to #createAndProcessBatchTaskRun and use it directly instead of BatchId.fromFriendlyId(batchId).

Please confirm BatchId.fromFriendlyId reliably reconstructs the same id for both ksuid- and cuid-shaped friendly ids before treating this purely as a style nit.

Also applies to: 169-184, 265-275

---

359-374: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Missing batch is silently ignored — no log emitted.

When findBatchTaskRunById returns nothing, the function returns silently, unlike the environment miss two lines below which logs an error. Given store-routing bugs could make a batch invisible from the wrong store, this failure mode deserves the same observability.

🔍 Proposed fix

     const batch = await this._engine.runStore.findBatchTaskRunById(options.batchId);

     if (!batch) {
+      logger.error("[RunEngineBatchTrigger][processBatchTaskRun] Batch not found", {
+        options,
+      });
       return;
     }

apps/webapp/test/engine/streamBatchItems.test.ts (1)

655-662: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Repeated PostgresRunStore wiring across 7 test cases.

The same 4-line block assigning engine.runStore = new PostgresRunStore({ prisma: racingPrisma, readOnlyPrisma: racingPrisma }) appears 7 times. A small helper (e.g. attachRacingRunStore(engine, racingPrisma)) would reduce duplication and centralize any future changes to how the racing store is wired.

Also applies to: 787-794, 919-926, 1052-1059, 1272-1279, 1411-1418, 1600-1604

apps/webapp/app/v3/services/createCheckpoint.server.ts (1)

149-154: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Batch lookup correctly routed through RunStore.

Both WAIT_FOR_BATCH lookups now call runStore.findBatchTaskRunByFriendlyId(friendlyId, environmentId), matching the upstream contract that probes NEW then LEGACY sub-stores scoped by (friendlyId, environmentId). This correctly fixes the pre-existing gap where a raw single-DB Prisma query would miss a NEW-resident (ksuid) batch.

The identical 5-line lookup + comment block is duplicated at both call sites (149-154 and 364-369). Extracting a small private helper would remove the duplication and centralize any future changes to the routing logic.

♻️ Proposed extraction

+  // Routed by friendlyId so a ksuid (NEW-resident) batch is found on the owning DB;
+  // env-scoped to the dependent attempt's run (a batch shares its dependent's env).
+  private async findWaitForBatchRun(batchFriendlyId: string, environmentId: string) {
+    return this.runStore.findBatchTaskRunByFriendlyId(batchFriendlyId, environmentId);
+  }

Then replace both call sites with:

-        // Routed by friendlyId so a ksuid (NEW-resident) batch is found on the owning DB;
-        // env-scoped to the dependent attempt's run (a batch shares its dependent's env).
-        const batchRun = await this.runStore.findBatchTaskRunByFriendlyId(
-          reason.batchFriendlyId,
-          attempt.taskRun.runtimeEnvironmentId
-        );
+        const batchRun = await this.findWaitForBatchRun(
+          reason.batchFriendlyId,
+          attempt.taskRun.runtimeEnvironmentId
+        );

Also applies to: 364-369

apps/webapp/app/v3/services/executeTasksWaitingForDeploy.ts (1)

74-111: 🩺 Stability & Availability | 🔵 Trivial

Solid defense-in-depth split; consider a quarantine path for stuck NEW-resident runs.

The NEW/legacy split correctly prevents a control-plane updateMany/enqueue from touching runs it can't actually own. One residual risk: if a NEW-resident run keeps getting selected by findRuns (e.g. from a real misconfiguration), it will never transition out of WAITING_FOR_DEPLOY, so this job will re-log the same error and potentially keep rescheduling itself (via the runsWaitingForDeploy.length > maxCount reschedule) on every poll, indefinitely.

Consider adding a metric/alert-worthy signal or a way to skip re-selecting known-stuck NEW-resident runs.

apps/webapp/test/engine/triggerTask.test.ts (1)

2393-2402: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Redundant dynamic import() — generateKsuidId is already statically imported.

generateKsuidId is imported statically at the top of the file (line 33) and used directly a few lines later (line 2432: generateKsuidId()). The dynamic await import("@trigger.dev/core/v3/isomorphic") here is unnecessary and inconsistent with the static-import usage elsewhere in the same file.

♻️ Proposed fix

       const parentFriendlyId = RunId.toFriendlyId(
-        // 27-char ksuid → classifies NEW
-        (await import("`@trigger.dev/core/v3/isomorphic`")).generateKsuidId()
+        // 27-char ksuid → classifies NEW
+        generateKsuidId()
       );

As per coding guidelines: "Prefer static imports over dynamic import(), and only use dynamic imports for unresolved circular dependencies, genuine code-splitting needs, or conditional runtime loading."

Source: Coding guidelines

apps/webapp/test/idempotencyDedupResidency.test.ts (1)

45-103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate seeding helpers across split-seam test files.

seedOrgProjectEnv/seedRun here are re-implemented nearly verbatim in apps/webapp/test/idempotencyKeyConcernLegacyAuthority.test.ts and apps/webapp/test/resetIdempotencyKeyLegacyAuthority.test.ts. Consider extracting a shared heteroPostgresTest fixture-seeding helper module as this residency test suite keeps growing.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: d490b21c-676d-49ba-83a2-67771105b181

📥 Commits

Reviewing files that changed from the base of the PR and between 413a945 and 515b897.

📒 Files selected for processing (50)

• apps/webapp/app/runEngine/concerns/idempotencyKeys.server.ts
• apps/webapp/app/runEngine/concerns/idempotencyResidency.server.test.ts
• apps/webapp/app/runEngine/concerns/idempotencyResidency.server.ts
• apps/webapp/app/runEngine/concerns/resolveWaitpointThroughReadThrough.server.ts
• apps/webapp/app/runEngine/services/batchTrigger.server.ts
• apps/webapp/app/runEngine/services/createBatch.server.ts
• apps/webapp/app/runEngine/services/streamBatchItems.server.ts
• apps/webapp/app/runEngine/services/triggerFailedTask.server.ts
• apps/webapp/app/runEngine/services/triggerTask.server.test.ts
• apps/webapp/app/runEngine/services/triggerTask.server.ts
• apps/webapp/app/services/archiveBranch.server.ts
• apps/webapp/app/services/dashboardAgent.server.ts
• apps/webapp/app/services/deleteProject.server.ts
• apps/webapp/app/services/realtime/runReader.server.ts
• apps/webapp/app/services/realtime/sessions.server.ts
• apps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
• apps/webapp/app/services/runsRepository/runsRepository.server.ts
• apps/webapp/app/v3/services/alerts/deliverAlert.server.ts
• apps/webapp/app/v3/services/alerts/performTaskRunAlerts.server.ts
• apps/webapp/app/v3/services/batchTriggerV3.server.ts
• apps/webapp/app/v3/services/bulk/BulkActionV2.batchReadThrough.server.test.ts
• apps/webapp/app/v3/services/bulk/BulkActionV2.batchReadThrough.server.ts
• apps/webapp/app/v3/services/bulk/BulkActionV2.server.ts
• apps/webapp/app/v3/services/cancelDevSessionRuns.server.ts
• apps/webapp/app/v3/services/createCheckpoint.server.ts
• apps/webapp/app/v3/services/createTaskRunAttempt.server.ts
• apps/webapp/app/v3/services/enqueueDelayedRun.server.ts
• apps/webapp/app/v3/services/executeTasksWaitingForDeploy.ts
• apps/webapp/app/v3/services/expireEnqueuedRun.server.ts
• apps/webapp/app/v3/services/finalizeTaskRun.server.ts
• apps/webapp/app/v3/services/resumeBatchRun.server.ts
• apps/webapp/test/batchTriggerV3ResidencyInheritance.test.ts
• apps/webapp/test/batchTriggerV3StoreRouting.test.ts
• apps/webapp/test/bulkActionV2ReadRouting.test.ts
• apps/webapp/test/cancelDevSessionRunsStoreRouting.test.ts
• apps/webapp/test/engine/streamBatchItems.test.ts
• apps/webapp/test/engine/triggerFailedTask.test.ts
• apps/webapp/test/engine/triggerTask.test.ts
• apps/webapp/test/idempotencyDedupResidency.test.ts
• apps/webapp/test/idempotencyKeyConcernLegacyAuthority.test.ts
• apps/webapp/test/performTaskRunAlertsStoreRouting.test.ts
• apps/webapp/test/realtime/runReaderReadThrough.test.ts
• apps/webapp/test/realtime/streamRegistrationRouting.test.ts
• apps/webapp/test/resetIdempotencyKeyLegacyAuthority.test.ts
• apps/webapp/test/resolveWaitpointThroughReadThrough.readthrough.test.ts
• apps/webapp/test/runEngineBatchTriggerStoreRouting.test.ts
• apps/webapp/test/runsRepository.readthrough.test.ts
• apps/webapp/test/runsRepositoryCpres.test.ts
• apps/webapp/test/sessions.readthrough.test.ts
• apps/webapp/test/streamLoader.controlPlane.test.ts

[d-cs]

Addressed the outside-diff note on idempotencyKeys.server.ts (blockRunWithWaitpoint writing via this.prisma): the block now passes the residency-resolved dedupClient as tx, so the idempotent parent run's waitpoint write lands on the store that owns that parent run rather than the fallback client.