workouts/PERSISTENCE-MIGRATION.md

Persistence Migration Plan: iCloud Drive Documents → Pure Database

Status: proposal — not started. This documents the plan for retiring the iCloud-Drive-documents-as-source-of-truth architecture (IndieSync, SyncEngine, JSON files + rebuildable SwiftData cache) in favor of a database as the sole source of truth. Nothing in this file is committed-to; it exists so the shift can be evaluated and, if approved, executed in well-scoped phases.

---

1. Where we are (summary of the current architecture)

• JSON documents in the ubiquity container iCloud.dev.rzen.indie.Workouts are the source of truth: Splits/<ULID>.json (routines), Workouts/YYYY/MM/<ULID>.json, Schedules/…, Stubs/<id>.json tombstones.
• SwiftData (WorkoutsModelContainer, cloudKitDatabase: .none) is a rebuildable read-through cache, wiped on schema bump or account change.
• One-way flow: view → SyncEngine.save(document) → file write → NSMetadataQuery observer → CacheMapper upsert → @Query refresh.
• The phone is the sole writer; the watch is a WatchConnectivity relay using the same @Model entities, fed only by phone pushes.
• Supporting machinery this architecture required us to build: tombstones + resurrection veto, seed reconcile planner, write backlog + flush hooks, settle-delay auto-seeding, conflict resolution via NSFileVersion, eviction-safe reads, schema-skip forward gate, duplicate cleanup tooling, a diagnostics subsystem, and a hard iCloud gate (RootGateView) because the app cannot function without the container.

That last list is the honest motivation for this migration: most of the system's complexity exists to compensate for filesystem-as-database.

2. Target architecture (recommendation)

SwiftData as the sole source of truth, with CloudKit mirroring for cross-device sync and reinstall durability. Documents survive only as DTOs: the WatchConnectivity wire format, the HealthKit-mapping input, and the view-layer mutation payload.

2.1 Database choice: SwiftData vs Core Data

Consideration	SwiftData	Core Data (NSPersistentCloudKitContainer)
Existing code	The 5 @Model entities, @Query views, and the container already exist — the cache becomes the database	Full rewrite of the entity layer + view fetch layer
Swift 6 strict concurrency	Native (@Model, ModelActor)	Achievable but fighting ObjC-era API
CloudKit sync status / observability	Opaque. No public sync-event API; diagnostics limited to network + account status	eventChangedNotification gives import/export/setup events with errors
Deduplication after concurrent inserts	HistoryDescriptor (iOS 18+) or fetch-based dedup passes	Persistent history transactions (Apple's canonical dedup sample)
Sharing / public DB (future)	Not exposed	Supported
Migration tooling	VersionedSchema + SchemaMigrationPlan	Mapping models / lightweight migration

Recommendation: SwiftData. The entities, container, and every list view already speak SwiftData; iOS 26 SwiftData is mature enough for this app's simple graph (5 models, 2 relationships). The one real cost — opaque sync status — is acceptable for a fitness app and is partly mitigated in §6 (Diagnostics). Core Data is the fallback only if a hard requirement for sync observability or CloudKit sharing emerges; the escape hatch stays open because SwiftData and Core Data can read the same store, but plan as if the choice is final.

2.2 Sync choice: CloudKit mirroring vs local-only

Two legitimate options; decide before Phase 3:

• Option A — CloudKit mirroring (recommended). cloudKitDatabase: .automatic (private DB). Keeps multi-device sync and free reinstall restore. Offline-first is preserved (mirroring queues locally). Deletes propagate natively — the entire tombstone subsystem disappears.
• Option B — local-only database. Simplest possible system; the watch already syncs via WatchConnectivity, not iCloud, so watch support is unaffected. But: no reinstall restore, no iPad/second-phone story, and backup becomes the only durability mechanism. Only choose this if multi-device and reinstall-restore are explicitly non-goals.

The rest of this plan assumes Option A, and notes where Option B would simplify a step.

2.3 Entity changes required for CloudKit

From the coupling inventory (Shared/Model/Entities.swift):

1. Drop @Attribute(.unique) from id on all 5 models — CloudKit does not support unique constraints; the container will refuse to mirror otherwise. Uniqueness becomes an app-level invariant: every insert path goes through an upsert helper (fetch-by-ULID first), and a dedup pass (§2.4) handles remote races.
2. Relationships must be optional: Routine.exercises: [Exercise] and Workout.logs: [WorkoutLog] become [Exercise]? / [WorkoutLog]? (computed non-optional accessors can hide this from views). The existing .cascade delete rules are fine; the optional to-ones (Exercise.routine, WorkoutLog.workout) are already compatible.
3. All other stored properties already carry defaults or are optional — ✅ compatible today (good news from the inventory).
4. Keep ULID string ids. They remain the stable identity across devices, the watch wire, and HealthKit metadata — nothing about CloudKit changes that.
5. Schedule.routineID stays a string join (no live relationship) — consistent with the existing denormalized Workout.routineID design and avoids CK relationship-integrity headaches.

2.4 Deduplication (CloudKit-specific, unavoidable)

Without unique constraints, two devices can insert the same logical record (same ULID) before syncing — most likely for seeds (both devices seed on first launch) and for the initial migration import (§4). Required:

• An idempotent dedup pass keyed by ULID: on remote-change import (or, at minimum, on app foreground), fetch ids with count > 1, keep the row with the newest updatedAt, merge child rows, delete the rest.
• Seeds specifically: fixed ULIDs make the merge trivial (identical content).

3. The new write path: WorkoutStore replaces SyncEngine

A new @Observable @MainActor type (working name WorkoutStore) with a deliberately identical public surface to today's SyncEngine where possible, so the ~40 view call sites (inventoried below) change their environment key and little else:

• save(routine:), save(workout:), save(schedule:), delete(…) — same signatures, still taking Document DTOs. Internally: map DTO → upsert @Model via CacheMapper (which already exists and stays), save context. No file write, no observer round-trip — @Query views update immediately.
• ingestFromWatch(_:) — same per-log merge logic (WorkoutMergePlanner survives unchanged; it operates on documents), then upsert.
• writeBackMachineSettings(…), scanForDuplicates() — port as-is.
• currentRoutineID(for:) — survives only as long as clone-on-edit does (see §5); ideally becomes the identity function and is then deleted.
• Gone entirely: connect(), iCloudStatus, abandonWaiting(), flushPendingWrites() + WriteBacklog (writes are local and synchronous — there is nothing to backlog), handle(_:) observer deltas, MetadataObserver, tombstone plumbing, settle delays.

Keeping documents as the mutation DTO is the load-bearing decision that makes this migration tractable: the inventory shows document types are pervasive in views, the watch app, WCPayload, and HealthKitMapping — none of that churns. Documents.swift gets retitled in comments from "on-disk format" to "wire/DTO format"; VersionedDocument conformance and relativePath move behind a legacy-import-only extension (needed only by §4).

4. One-time migration of existing user data

On first launch of the new version (guarded by a persisted migration flag):

1. Import: enumerate the ubiquity container exactly as reconcile() does today (IndieSync's placeholder-aware enumeration — this is the one place IndieSync is still linked), decode every live document, and upsert into SwiftData via CacheMapper. Honor tombstones: a stub for id X means X is not imported. Record deleted-seed stubs into the new seed-veto store (§5).
2. Do not delete the files. Leave the container intact for ≥2 release cycles as a recovery escape hatch. A later release removes the CloudDocuments entitlement and (optionally) offers a cleanup.
3. Multi-device staggering: device 1 upgrades and imports at time T; device 2 keeps writing files until it upgrades, then imports its view of the container. Both imports upsert by ULID into the same CloudKit private DB — last-writer-wins per record, dedup pass (§2.4) cleans up races. Edits made on the not-yet-upgraded device after T that never reached the container before device 1's import are picked up when device 2 itself imports. The convergence guarantee is: every device imports its own container replica once, and ULID-upsert makes that idempotent.
4. Cache-wipe semantics change permanently: the store is no longer disposable. wipeIfNeeded() / schema-version-bump-wipes are removed; from this point on, schema changes require real VersionedSchema migrations (§7). The account-change wipe also goes away — CloudKit mirroring handles account switching itself (the mirror re-syncs; local store is per-account managed by the system).
5. Failure handling: import is all-or-nothing per document but tolerant overall (a corrupt file is logged and skipped — same policy as today's reconcile). The migration flag is set only after the enumeration completes; a crash mid-import re-runs it (idempotent by ULID upsert).

5. Subsystem-by-subsystem disposition

Grounded in the coupling inventory (working tree, post Splits→Routines rename):

Subsystem	Disposition
SyncEngine (~1000 lines)	Replaced by WorkoutStore (§3); expect it to shrink to ~⅓ the size
IndieSync SPM package	Dropped from all 4 targets after the legacy import window closes; during the window, linked only for the import path (DocumentFileStore enumeration + Tombstone decode)
Tombstones / TombstoneStore	Deleted — CloudKit propagates deletes. One residue: the seed veto (below)
Seed system (SeedLibrary, SeedReconcilePlanner, auto-seed, restore)	Simplifies drastically. Keep fixed ULIDs. Seeding = upsert seed entities if absent and not vetoed. The resurrection veto becomes a tiny synced record (e.g. a deletedSeedIDs list on a singleton AppState model, or a SeedVeto model) instead of stub files. reconcileSeeds() becomes "upsert newer seed content by ULID unless user-forked". Restore Starter Routines stays: clear veto + re-upsert
Clone-on-edit (cloneSeedOnEdit, cloneRedirects, currentRoutineID(for:), repointWorkouts)	Decision point. It existed because a fixed-ULID file could be resurrected by reconcile, so user edits had to fork away from seed identity. With a DB + veto record, an edited seed can simply… be edited (mark it userModified: Bool so reconcile skips upgrading it). Recommendation: retire clone-on-edit; currentRoutineID(for:) becomes identity and its 15 call sites collapse. This deletes the subtlest code in the app
WriteBacklog + flushPendingWrites + SyncStatusBanner write-queue state	Deleted — no async write path to backlog. Banner either goes away or repurposes for CK account status
RootGateView iCloud gate	Deleted. The app works offline/signed-out; CloudKit mirrors when it can. This removes a whole class of first-run failure (today's "no iCloud → no app")
Diagnostics (ContainerStatus, DocumentSyncInspector, SchemaSkipScanner, DiagnosticsReport)	Gutted. File/metadata/schema-skip diagnostics are meaningless. Keep NetworkReadiness; add CKContainer.accountStatus. Accept that SwiftData mirroring offers no per-record sync visibility (revisit Core Data only if this proves unacceptable in practice)
DuplicateCleanup dev tool	Replace with the §2.4 dedup pass; the Settings dev screen can surface its results
IndieBackup (AppBackupConfiguration)	Keep — it's persistence-agnostic. backupRoot changes from the ubiquity Documents/ tree to a staged JSON export: prepareForBackup serializes all entities to documents (the mappers already exist) into a local folder; restore imports them and rebuildCacheAfterRestore becomes "import the JSON". Backup remains human-readable JSON — a deliberate property worth preserving
Watch (WatchConnectivityBridge, WatchCacheApplier, WCPayload)	Unchanged. The watch keeps its local non-CK SwiftData store fed by phone pushes; wire format stays documents; watch entitlements untouched. (Option considered and rejected: giving the watch its own CloudKit mirror — worse latency for live workouts, new entitlement, and the WC relay already works)
HealthKitMapping	Unchanged (consumes documents; documents survive as DTOs)
ScreenshotSeed / ScreenshotRootView	Unchanged (already constructs entities directly; jsonRelativePath args drop out when the field is removed)
jsonRelativePath on entities	Removed (a schema migration, §7)
Scripts/generate_starter_splits.swift	Survives — seeds stay bundled canonical JSON with fixed ULIDs; only the consumer changes (decode → upsert entities instead of writing bytes to the container)
WorkoutsModelContainer	Loses wipeIfNeeded/wipeIfAccountChanged; gains cloudKitDatabase: .private("iCloud.dev.rzen.indie.Workouts") on iOS and keeps .none on watchOS (one make() with a per-platform flag)
Entitlements / project.yml	iOS: add CloudKit to icloud-services (keep CloudDocuments during the import window, remove later). Watch: no change. Drop IndieSync package refs at the end

6. CloudKit operational realities (Option A)

• Schema is additive-only in production. Once the CK schema deploys, record fields can never be removed or renamed on the server — only added. Local SwiftData migrations stay flexible; the CK record type accretes. This makes §2.3's field cleanup (e.g. dropping jsonRelativePath) something to do before first CK deployment, not after.
• Deploy the schema to the production CK environment before App Store release (Development → Production promotion in CK Console); TestFlight builds use the production environment — sequence the rollout accordingly.
• First sync of a large history: month-bucketed workout history could be years of records; initial mirroring is background and throttled. Set expectations: no progress UI is possible with SwiftData (see Diagnostics).
• Quota: private-DB data counts against the user's iCloud storage, same as the current documents — no change in story.
• The current WorkoutDocument.relativePath month-bucketing rationale disappears entirely — CK doesn't care; ULIDs already sort chronologically.

7. Schema versioning going forward

Today: bump WorkoutsModelContainer.currentSchemaVersion (now 8) → wipe → rebuild from files. That option dies with this migration. Replacement:

• Freeze the current entity shape as SchemaV1 (VersionedSchema), define a SchemaMigrationPlan, and route all future shape changes through lightweight (preferred) or custom migration stages.
• Document schema versions (RoutineDocument.currentSchemaVersion etc.) stay, but now gate only the wire/DTO/backup formats (watch payloads, backup exports, legacy import) — decoupled from the store schema.
• The forward-compat quarantine ("file written by a newer app version") has no CK equivalent; CloudKit handles unknown-field tolerance natively (unknown record fields are preserved, not decoded). Older app + newer schema coexistence is governed by CK's additive-only rule instead.

8. Execution phases

Each phase is independently shippable; stop-points between all of them.

• Phase 0 — prerequisites (do first, ships with current architecture)
  1. Land the in-flight UX redesign / Routines rename. Do not start this migration on top of the current uncommitted tree.
  2. Introduce the upsert-by-ULID helper and non-optional relationship accessors so later diffs are mechanical.
  3. Decide §2.2 (CloudKit vs local-only) and the clone-on-edit retirement (§5) — the two genuine product decisions in this plan.
• Phase 1 — the flip (biggest single change) WorkoutStore replaces SyncEngine; entity changes (§2.3, minus CK enablement); one-time import (§4); delete tombstones/backlog/gate/seed machinery per §5; rewire Backup; gut Diagnostics. CK not yet enabled (cloudKitDatabase still .none) — this ships as a local-only build behind full regression testing, or goes straight to Phase 2 in the same release if confidence is high. Watch untouched; all watch tests must pass unmodified.
• Phase 2 — enable CloudKit mirroring Entitlement + container config, .private(...) database, dedup pass, seed-veto record, CK Console schema deploy to production. Multi-device testing matrix: fresh install, upgrade-with-data, two-device stagger (§4.3), account switch, airplane-mode edits on both devices then reconnect.
• Phase 3 — decommission (a release or two later) Drop IndieSync from project.yml, remove the legacy import path and CloudDocuments entitlement, optionally offer container cleanup, delete relativePath/VersionedDocument residue from Documents.swift.

9. Test strategy

• The pure planners (WorkoutMergePlanner, seed decision logic) keep their unit tests nearly verbatim — they operate on documents.
• New unit targets: upsert-by-ULID idempotence, dedup pass, legacy import (fixture container → expected entity graph, including tombstone honoring and corrupt-file skip), backup export/import round-trip.
• Watch test suites (WatchCacheApplierTests, WatchConnectivityBridgeTests, SessionEndPlannerTests) must pass unchanged — they are the proof the watch boundary held.
• Device protocol (extend BULLETPROOFING.md): the Phase 2 multi-device matrix above, plus reinstall-restore and iCloud-signed-out operation (which must now work instead of gating).

10. Risks & open questions

Risk	Mitigation
SwiftData CK mirroring is opaque (no sync events, no conflict hooks)	Accept for v1; Core Data fallback documented (§2.1); dedup pass covers the main correctness hole
CK schema additive-only lock-in	Clean the entity shape (drop dead fields) before Phase 2; review every field name once more at that gate
Migration bugs eat user data	Files left intact ≥2 releases (§4.2); import idempotent; backup feature works before Phase 1 ships
Duplicate records from device staggering	ULID upsert + dedup pass; seeds are the worst case and merge trivially
Losing human-readable files as a user-facing property	Backup export keeps JSON portability; consider a manual "Export data" share action
Retiring clone-on-edit changes seed-upgrade semantics (a user-edited seed no longer receives content upgrades)	That is arguably the correct behavior; userModified flag makes it explicit. Confirm before Phase 0 exit
SwiftData maturity surprises (mirroring edge cases)	Phase 1/Phase 2 split means CK can be delayed indefinitely without blocking the architectural cleanup

11. What gets deleted (the payoff)

Tombstone store + resurrection veto files, reconcile settle-delays, write backlog + flush hooks, NSMetadataQuery observers (both of them), eviction handling, NSFileVersion conflict resolution, the iCloud hard gate, the schema-skip scanner, duplicate cleanup tooling, clone-on-edit + redirect resolution (15 call sites), five independent ubiquity-container resolutions, and the entire class of "file arrived while / metadata index hasn't settled" race conditions. Estimated net: -2,000–3,000 lines of the app's subtlest code, in exchange for one dedup pass and a real schema-migration discipline.