workouts/PERSISTENCE-MIGRATION.md

# Persistence Migration Plan: iCloud Drive Documents → Pure Database

Status: **proposal — not started.** This documents the plan for retiring the
iCloud-Drive-documents-as-source-of-truth architecture (IndieSync, `SyncEngine`,
JSON files + rebuildable SwiftData cache) in favor of a database as the sole
source of truth. Nothing in this file is committed-to; it exists so the shift
can be evaluated and, if approved, executed in well-scoped phases.

---

## 1. Where we are (summary of the current architecture)

- JSON documents in the ubiquity container `iCloud.dev.rzen.indie.Workouts`
  are the source of truth: `Splits/<ULID>.json` (routines), `Workouts/YYYY/MM/<ULID>.json`,
  `Schedules/…`, `Stubs/<id>.json` tombstones.
- SwiftData (`WorkoutsModelContainer`, `cloudKitDatabase: .none`) is a
  **rebuildable read-through cache**, wiped on schema bump or account change.
- One-way flow: view → `SyncEngine.save(document)` → file write →
  `NSMetadataQuery` observer → `CacheMapper` upsert → `@Query` refresh.
- The phone is the sole writer; the watch is a WatchConnectivity relay using
  the same `@Model` entities, fed only by phone pushes.
- Supporting machinery this architecture *required us to build*: tombstones +
  resurrection veto, seed reconcile planner, write backlog + flush hooks,
  settle-delay auto-seeding, conflict resolution via `NSFileVersion`,
  eviction-safe reads, schema-skip forward gate, duplicate cleanup tooling,
  a diagnostics subsystem, and a hard iCloud gate (`RootGateView`) because the
  app cannot function without the container.

That last list is the honest motivation for this migration: most of the
system's complexity exists to compensate for filesystem-as-database.

## 2. Target architecture (recommendation)

**SwiftData as the sole source of truth, with CloudKit mirroring for
cross-device sync and reinstall durability.** Documents survive only as DTOs:
the WatchConnectivity wire format, the HealthKit-mapping input, and the
view-layer mutation payload.

### 2.1 Database choice: SwiftData vs Core Data

| Consideration | SwiftData | Core Data (`NSPersistentCloudKitContainer`) |
|---|---|---|
| Existing code | The 5 `@Model` entities, `@Query` views, and the container already exist — the cache *becomes* the database | Full rewrite of the entity layer + view fetch layer |
| Swift 6 strict concurrency | Native (`@Model`, `ModelActor`) | Achievable but fighting ObjC-era API |
| CloudKit sync status / observability | **Opaque.** No public sync-event API; diagnostics limited to network + account status | `eventChangedNotification` gives import/export/setup events with errors |
| Deduplication after concurrent inserts | `HistoryDescriptor` (iOS 18+) or fetch-based dedup passes | Persistent history transactions (Apple's canonical dedup sample) |
| Sharing / public DB (future) | Not exposed | Supported |
| Migration tooling | `VersionedSchema` + `SchemaMigrationPlan` | Mapping models / lightweight migration |

**Recommendation: SwiftData.** The entities, container, and every list view
already speak SwiftData; iOS 26 SwiftData is mature enough for this app's
simple graph (5 models, 2 relationships). The one real cost — opaque sync
status — is acceptable for a fitness app and is partly mitigated in §6
(Diagnostics). Core Data is the fallback **only if** a hard requirement for
sync observability or CloudKit sharing emerges; the escape hatch stays open
because SwiftData and Core Data can read the same store, but plan as if the
choice is final.

### 2.2 Sync choice: CloudKit mirroring vs local-only

Two legitimate options; decide before Phase 3:

- **Option A — CloudKit mirroring (recommended).** `cloudKitDatabase: .automatic`
  (private DB). Keeps multi-device sync and free reinstall restore. Offline-first
  is preserved (mirroring queues locally). Deletes propagate natively —
  **the entire tombstone subsystem disappears.**
- **Option B — local-only database.** Simplest possible system; the watch
  already syncs via WatchConnectivity, not iCloud, so watch support is
  unaffected. But: no reinstall restore, no iPad/second-phone story, and
  backup becomes the *only* durability mechanism. Only choose this if
  multi-device and reinstall-restore are explicitly non-goals.

The rest of this plan assumes **Option A**, and notes where Option B would
simplify a step.

### 2.3 Entity changes required for CloudKit

From the coupling inventory (`Shared/Model/Entities.swift`):

1. **Drop `@Attribute(.unique)` from `id` on all 5 models** — CloudKit does
   not support unique constraints; the container will refuse to mirror
   otherwise. Uniqueness becomes an app-level invariant: every insert path
   goes through an upsert helper (fetch-by-ULID first), and a dedup pass
   (§2.4) handles remote races.
2. **Relationships must be optional**: `Routine.exercises: [Exercise]` and
   `Workout.logs: [WorkoutLog]` become `[Exercise]?` / `[WorkoutLog]?`
   (computed non-optional accessors can hide this from views). The existing
   `.cascade` delete rules are fine; the optional to-ones (`Exercise.routine`,
   `WorkoutLog.workout`) are already compatible.
3. All other stored properties already carry defaults or are optional — ✅
   compatible today (good news from the inventory).
4. Keep ULID string `id`s. They remain the stable identity across devices,
   the watch wire, and HealthKit metadata — nothing about CloudKit changes
   that.
5. **`Schedule.routineID` stays a string join** (no live relationship) —
   consistent with the existing denormalized `Workout.routineID` design and
   avoids CK relationship-integrity headaches.

### 2.4 Deduplication (CloudKit-specific, unavoidable)

Without unique constraints, two devices can insert the same logical record
(same ULID) before syncing — most likely for **seeds** (both devices seed on
first launch) and for the initial migration import (§4). Required:

- An idempotent **dedup pass keyed by ULID**: on remote-change import (or, at
  minimum, on app foreground), fetch ids with count > 1, keep the row with
  the newest `updatedAt`, merge child rows, delete the rest.
- Seeds specifically: fixed ULIDs make the merge trivial (identical content).

## 3. The new write path: `WorkoutStore` replaces `SyncEngine`

A new `@Observable @MainActor` type (working name `WorkoutStore`) with a
**deliberately identical public surface** to today's `SyncEngine` where
possible, so the ~40 view call sites (inventoried below) change their
environment key and little else:

- `save(routine:)`, `save(workout:)`, `save(schedule:)`, `delete(…)` — same
  signatures, still taking **Document DTOs**. Internally: map DTO → upsert
  `@Model` via `CacheMapper` (which already exists and stays), save context.
  No file write, no observer round-trip — `@Query` views update immediately.
- `ingestFromWatch(_:)` — same per-log merge logic (`WorkoutMergePlanner`
  survives unchanged; it operates on documents), then upsert.
- `writeBackMachineSettings(…)`, `scanForDuplicates()` — port as-is.
- `currentRoutineID(for:)` — survives only as long as clone-on-edit does
  (see §5); ideally becomes the identity function and is then deleted.
- **Gone entirely**: `connect()`, `iCloudStatus`, `abandonWaiting()`,
  `flushPendingWrites()` + `WriteBacklog` (writes are local and synchronous —
  there is nothing to backlog), `handle(_:)` observer deltas,
  `MetadataObserver`, tombstone plumbing, settle delays.

Keeping documents as the mutation DTO is the load-bearing decision that makes
this migration tractable: the inventory shows document types are pervasive in
views, the watch app, `WCPayload`, and `HealthKitMapping` — none of that
churns. `Documents.swift` gets retitled in comments from "on-disk format" to
"wire/DTO format"; `VersionedDocument` conformance and `relativePath` move
behind a legacy-import-only extension (needed only by §4).

## 4. One-time migration of existing user data

On first launch of the new version (guarded by a persisted migration flag):

1. **Import**: enumerate the ubiquity container exactly as `reconcile()` does
   today (IndieSync's placeholder-aware enumeration — this is the one place
   IndieSync is still linked), decode every live document, and upsert into
   SwiftData via `CacheMapper`. Honor tombstones: a stub for id X means X is
   not imported. Record deleted-seed stubs into the new seed-veto store (§5).
2. **Do not delete the files.** Leave the container intact for ≥2 release
   cycles as a recovery escape hatch. A later release removes the
   `CloudDocuments` entitlement and (optionally) offers a cleanup.
3. **Multi-device staggering**: device 1 upgrades and imports at time T;
   device 2 keeps writing *files* until it upgrades, then imports *its* view
   of the container. Both imports upsert by ULID into the same CloudKit
   private DB — last-writer-wins per record, dedup pass (§2.4) cleans up
   races. Edits made on the not-yet-upgraded device after T that never
   reached the container before device 1's import are picked up when device
   2 itself imports. The convergence guarantee is: every device imports its
   own container replica once, and ULID-upsert makes that idempotent.
4. **Cache-wipe semantics change permanently**: the store is no longer
   disposable. `wipeIfNeeded()` / schema-version-bump-wipes are removed;
   from this point on, schema changes require real `VersionedSchema`
   migrations (§7). The account-change wipe also goes away — CloudKit
   mirroring handles account switching itself (the mirror re-syncs; local
   store is per-account managed by the system).
5. **Failure handling**: import is all-or-nothing per document but tolerant
   overall (a corrupt file is logged and skipped — same policy as today's
   reconcile). The migration flag is set only after the enumeration
   completes; a crash mid-import re-runs it (idempotent by ULID upsert).

## 5. Subsystem-by-subsystem disposition

Grounded in the coupling inventory (working tree, post Splits→Routines rename):

| Subsystem | Disposition |
|---|---|
| `SyncEngine` (~1000 lines) | Replaced by `WorkoutStore` (§3); expect it to shrink to ~⅓ the size |
| IndieSync SPM package | Dropped from all 4 targets after the legacy import window closes; during the window, linked only for the import path (`DocumentFileStore` enumeration + `Tombstone` decode) |
| Tombstones / `TombstoneStore` | **Deleted** — CloudKit propagates deletes. One residue: the seed veto (below) |
| Seed system (`SeedLibrary`, `SeedReconcilePlanner`, auto-seed, restore) | Simplifies drastically. Keep fixed ULIDs. Seeding = upsert seed entities if absent **and not vetoed**. The resurrection veto becomes a tiny synced record (e.g. a `deletedSeedIDs` list on a singleton `AppState` model, or a `SeedVeto` model) instead of stub files. `reconcileSeeds()` becomes "upsert newer seed content by ULID unless user-forked". **Restore Starter Routines** stays: clear veto + re-upsert |
| Clone-on-edit (`cloneSeedOnEdit`, `cloneRedirects`, `currentRoutineID(for:)`, `repointWorkouts`) | **Decision point.** It existed because a fixed-ULID *file* could be resurrected by reconcile, so user edits had to fork away from seed identity. With a DB + veto record, an edited seed can simply… be edited (mark it `userModified: Bool` so reconcile skips upgrading it). Recommendation: retire clone-on-edit; `currentRoutineID(for:)` becomes identity and its 15 call sites collapse. This deletes the subtlest code in the app |
| `WriteBacklog` + `flushPendingWrites` + `SyncStatusBanner` write-queue state | **Deleted** — no async write path to backlog. Banner either goes away or repurposes for CK account status |
| `RootGateView` iCloud gate | **Deleted.** The app works offline/signed-out; CloudKit mirrors when it can. This removes a whole class of first-run failure (today's "no iCloud → no app") |
| Diagnostics (`ContainerStatus`, `DocumentSyncInspector`, `SchemaSkipScanner`, `DiagnosticsReport`) | Gutted. File/metadata/schema-skip diagnostics are meaningless. Keep `NetworkReadiness`; add `CKContainer.accountStatus`. Accept that SwiftData mirroring offers no per-record sync visibility (revisit Core Data only if this proves unacceptable in practice) |
| `DuplicateCleanup` dev tool | Replace with the §2.4 dedup pass; the Settings dev screen can surface its results |
| IndieBackup (`AppBackupConfiguration`) | Keep — it's persistence-agnostic. `backupRoot` changes from the ubiquity `Documents/` tree to a **staged JSON export**: `prepareForBackup` serializes all entities to documents (the mappers already exist) into a local folder; restore imports them and `rebuildCacheAfterRestore` becomes "import the JSON". Backup remains human-readable JSON — a deliberate property worth preserving |
| Watch (`WatchConnectivityBridge`, `WatchCacheApplier`, `WCPayload`) | **Unchanged.** The watch keeps its local non-CK SwiftData store fed by phone pushes; wire format stays documents; watch entitlements untouched. (Option considered and rejected: giving the watch its own CloudKit mirror — worse latency for live workouts, new entitlement, and the WC relay already works) |
| `HealthKitMapping` | Unchanged (consumes documents; documents survive as DTOs) |
| `ScreenshotSeed` / `ScreenshotRootView` | Unchanged (already constructs entities directly; `jsonRelativePath` args drop out when the field is removed) |
| `jsonRelativePath` on entities | Removed (a schema migration, §7) |
| `Scripts/generate_starter_splits.swift` | Survives — seeds stay bundled canonical JSON with fixed ULIDs; only the *consumer* changes (decode → upsert entities instead of writing bytes to the container) |
| `WorkoutsModelContainer` | Loses `wipeIfNeeded`/`wipeIfAccountChanged`; gains `cloudKitDatabase: .private("iCloud.dev.rzen.indie.Workouts")` on iOS and keeps `.none` on watchOS (one `make()` with a per-platform flag) |
| Entitlements / `project.yml` | iOS: add `CloudKit` to `icloud-services` (keep `CloudDocuments` during the import window, remove later). Watch: no change. Drop IndieSync package refs at the end |

## 6. CloudKit operational realities (Option A)

- **Schema is additive-only in production.** Once the CK schema deploys,
  record fields can never be removed or renamed on the server — only added.
  Local SwiftData migrations stay flexible; the CK record type accretes.
  This makes §2.3's field cleanup (e.g. dropping `jsonRelativePath`)
  something to do **before** first CK deployment, not after.
- **Deploy the schema to the production CK environment before App Store
  release** (Development → Production promotion in CK Console); TestFlight
  builds use the production environment — sequence the rollout accordingly.
- **First sync of a large history**: month-bucketed workout history could be
  years of records; initial mirroring is background and throttled. Set
  expectations: no progress UI is possible with SwiftData (see Diagnostics).
- **Quota**: private-DB data counts against the user's iCloud storage, same
  as the current documents — no change in story.
- The current `WorkoutDocument.relativePath` month-bucketing rationale
  disappears entirely — CK doesn't care; ULIDs already sort chronologically.

## 7. Schema versioning going forward

Today: bump `WorkoutsModelContainer.currentSchemaVersion` (now 8) → wipe →
rebuild from files. That option dies with this migration. Replacement:

- Freeze the current entity shape as `SchemaV1` (`VersionedSchema`), define a
  `SchemaMigrationPlan`, and route all future shape changes through
  lightweight (preferred) or custom migration stages.
- Document schema versions (`RoutineDocument.currentSchemaVersion` etc.) stay,
  but now gate only the **wire/DTO/backup** formats (watch payloads, backup
  exports, legacy import) — decoupled from the store schema.
- The forward-compat quarantine ("file written by a newer app version") has
  no CK equivalent; CloudKit handles unknown-field tolerance natively
  (unknown record fields are preserved, not decoded). Older app + newer
  schema coexistence is governed by CK's additive-only rule instead.

## 8. Execution phases

Each phase is independently shippable; stop-points between all of them.

- **Phase 0 — prerequisites (do first, ships with current architecture)**
  1. Land the in-flight UX redesign / Routines rename. **Do not start this
     migration on top of the current uncommitted tree.**
  2. Introduce the upsert-by-ULID helper and non-optional relationship
     accessors so later diffs are mechanical.
  3. Decide §2.2 (CloudKit vs local-only) and the clone-on-edit retirement
     (§5) — the two genuine product decisions in this plan.
- **Phase 1 — the flip (biggest single change)**
  `WorkoutStore` replaces `SyncEngine`; entity changes (§2.3, minus CK
  enablement); one-time import (§4); delete tombstones/backlog/gate/seed
  machinery per §5; rewire Backup; gut Diagnostics. CK **not yet enabled**
  (`cloudKitDatabase` still `.none`) — this ships as a local-only build
  behind full regression testing, or goes straight to Phase 2 in the same
  release if confidence is high. Watch untouched; all watch tests must pass
  unmodified.
- **Phase 2 — enable CloudKit mirroring**
  Entitlement + container config, `.private(...)` database, dedup pass,
  seed-veto record, CK Console schema deploy to production. Multi-device
  testing matrix: fresh install, upgrade-with-data, two-device stagger (§4.3),
  account switch, airplane-mode edits on both devices then reconnect.
- **Phase 3 — decommission (a release or two later)**
  Drop IndieSync from `project.yml`, remove the legacy import path and
  `CloudDocuments` entitlement, optionally offer container cleanup, delete
  `relativePath`/`VersionedDocument` residue from `Documents.swift`.

## 9. Test strategy

- The pure planners (`WorkoutMergePlanner`, seed decision logic) keep their
  unit tests nearly verbatim — they operate on documents.
- New unit targets: upsert-by-ULID idempotence, dedup pass, legacy import
  (fixture container → expected entity graph, including tombstone honoring
  and corrupt-file skip), backup export/import round-trip.
- Watch test suites (`WatchCacheApplierTests`, `WatchConnectivityBridgeTests`,
  `SessionEndPlannerTests`) must pass **unchanged** — they are the proof the
  watch boundary held.
- Device protocol (extend BULLETPROOFING.md): the Phase 2 multi-device matrix
  above, plus reinstall-restore and iCloud-signed-out operation (which must
  now *work* instead of gating).

## 10. Risks & open questions

| Risk | Mitigation |
|---|---|
| SwiftData CK mirroring is opaque (no sync events, no conflict hooks) | Accept for v1; Core Data fallback documented (§2.1); dedup pass covers the main correctness hole |
| CK schema additive-only lock-in | Clean the entity shape (drop dead fields) *before* Phase 2; review every field name once more at that gate |
| Migration bugs eat user data | Files left intact ≥2 releases (§4.2); import idempotent; backup feature works before Phase 1 ships |
| Duplicate records from device staggering | ULID upsert + dedup pass; seeds are the worst case and merge trivially |
| Losing human-readable files as a user-facing property | Backup export keeps JSON portability; consider a manual "Export data" share action |
| Retiring clone-on-edit changes seed-upgrade semantics (a user-edited seed no longer receives content upgrades) | That is arguably the *correct* behavior; `userModified` flag makes it explicit. Confirm before Phase 0 exit |
| SwiftData maturity surprises (mirroring edge cases) | Phase 1/Phase 2 split means CK can be delayed indefinitely without blocking the architectural cleanup |

## 11. What gets deleted (the payoff)

Tombstone store + resurrection veto files, reconcile settle-delays, write
backlog + flush hooks, `NSMetadataQuery` observers (both of them), eviction
handling, `NSFileVersion` conflict resolution, the iCloud hard gate, the
schema-skip scanner, duplicate cleanup tooling, clone-on-edit + redirect
resolution (15 call sites), five independent ubiquity-container resolutions,
and the entire class of "file arrived while / metadata index hasn't settled"
race conditions. Estimated net: **-2,000–3,000 lines** of the app's subtlest
code, in exchange for one dedup pass and a real schema-migration discipline.