diff --git a/PERSISTENCE-MIGRATION.md b/PERSISTENCE-MIGRATION.md new file mode 100644 index 0000000..84a41c6 --- /dev/null +++ b/PERSISTENCE-MIGRATION.md @@ -0,0 +1,291 @@ +# Persistence Migration Plan: iCloud Drive Documents → Pure Database + +Status: **proposal — not started.** This documents the plan for retiring the +iCloud-Drive-documents-as-source-of-truth architecture (IndieSync, `SyncEngine`, +JSON files + rebuildable SwiftData cache) in favor of a database as the sole +source of truth. Nothing in this file is committed-to; it exists so the shift +can be evaluated and, if approved, executed in well-scoped phases. + +--- + +## 1. Where we are (summary of the current architecture) + +- JSON documents in the ubiquity container `iCloud.dev.rzen.indie.Workouts` + are the source of truth: `Splits/.json` (routines), `Workouts/YYYY/MM/.json`, + `Schedules/…`, `Stubs/.json` tombstones. +- SwiftData (`WorkoutsModelContainer`, `cloudKitDatabase: .none`) is a + **rebuildable read-through cache**, wiped on schema bump or account change. +- One-way flow: view → `SyncEngine.save(document)` → file write → + `NSMetadataQuery` observer → `CacheMapper` upsert → `@Query` refresh. +- The phone is the sole writer; the watch is a WatchConnectivity relay using + the same `@Model` entities, fed only by phone pushes. +- Supporting machinery this architecture *required us to build*: tombstones + + resurrection veto, seed reconcile planner, write backlog + flush hooks, + settle-delay auto-seeding, conflict resolution via `NSFileVersion`, + eviction-safe reads, schema-skip forward gate, duplicate cleanup tooling, + a diagnostics subsystem, and a hard iCloud gate (`RootGateView`) because the + app cannot function without the container. + +That last list is the honest motivation for this migration: most of the +system's complexity exists to compensate for filesystem-as-database. + +## 2. Target architecture (recommendation) + +**SwiftData as the sole source of truth, with CloudKit mirroring for +cross-device sync and reinstall durability.** Documents survive only as DTOs: +the WatchConnectivity wire format, the HealthKit-mapping input, and the +view-layer mutation payload. + +### 2.1 Database choice: SwiftData vs Core Data + +| Consideration | SwiftData | Core Data (`NSPersistentCloudKitContainer`) | +|---|---|---| +| Existing code | The 5 `@Model` entities, `@Query` views, and the container already exist — the cache *becomes* the database | Full rewrite of the entity layer + view fetch layer | +| Swift 6 strict concurrency | Native (`@Model`, `ModelActor`) | Achievable but fighting ObjC-era API | +| CloudKit sync status / observability | **Opaque.** No public sync-event API; diagnostics limited to network + account status | `eventChangedNotification` gives import/export/setup events with errors | +| Deduplication after concurrent inserts | `HistoryDescriptor` (iOS 18+) or fetch-based dedup passes | Persistent history transactions (Apple's canonical dedup sample) | +| Sharing / public DB (future) | Not exposed | Supported | +| Migration tooling | `VersionedSchema` + `SchemaMigrationPlan` | Mapping models / lightweight migration | + +**Recommendation: SwiftData.** The entities, container, and every list view +already speak SwiftData; iOS 26 SwiftData is mature enough for this app's +simple graph (5 models, 2 relationships). The one real cost — opaque sync +status — is acceptable for a fitness app and is partly mitigated in §6 +(Diagnostics). Core Data is the fallback **only if** a hard requirement for +sync observability or CloudKit sharing emerges; the escape hatch stays open +because SwiftData and Core Data can read the same store, but plan as if the +choice is final. + +### 2.2 Sync choice: CloudKit mirroring vs local-only + +Two legitimate options; decide before Phase 3: + +- **Option A — CloudKit mirroring (recommended).** `cloudKitDatabase: .automatic` + (private DB). Keeps multi-device sync and free reinstall restore. Offline-first + is preserved (mirroring queues locally). Deletes propagate natively — + **the entire tombstone subsystem disappears.** +- **Option B — local-only database.** Simplest possible system; the watch + already syncs via WatchConnectivity, not iCloud, so watch support is + unaffected. But: no reinstall restore, no iPad/second-phone story, and + backup becomes the *only* durability mechanism. Only choose this if + multi-device and reinstall-restore are explicitly non-goals. + +The rest of this plan assumes **Option A**, and notes where Option B would +simplify a step. + +### 2.3 Entity changes required for CloudKit + +From the coupling inventory (`Shared/Model/Entities.swift`): + +1. **Drop `@Attribute(.unique)` from `id` on all 5 models** — CloudKit does + not support unique constraints; the container will refuse to mirror + otherwise. Uniqueness becomes an app-level invariant: every insert path + goes through an upsert helper (fetch-by-ULID first), and a dedup pass + (§2.4) handles remote races. +2. **Relationships must be optional**: `Routine.exercises: [Exercise]` and + `Workout.logs: [WorkoutLog]` become `[Exercise]?` / `[WorkoutLog]?` + (computed non-optional accessors can hide this from views). The existing + `.cascade` delete rules are fine; the optional to-ones (`Exercise.routine`, + `WorkoutLog.workout`) are already compatible. +3. All other stored properties already carry defaults or are optional — ✅ + compatible today (good news from the inventory). +4. Keep ULID string `id`s. They remain the stable identity across devices, + the watch wire, and HealthKit metadata — nothing about CloudKit changes + that. +5. **`Schedule.routineID` stays a string join** (no live relationship) — + consistent with the existing denormalized `Workout.routineID` design and + avoids CK relationship-integrity headaches. + +### 2.4 Deduplication (CloudKit-specific, unavoidable) + +Without unique constraints, two devices can insert the same logical record +(same ULID) before syncing — most likely for **seeds** (both devices seed on +first launch) and for the initial migration import (§4). Required: + +- An idempotent **dedup pass keyed by ULID**: on remote-change import (or, at + minimum, on app foreground), fetch ids with count > 1, keep the row with + the newest `updatedAt`, merge child rows, delete the rest. +- Seeds specifically: fixed ULIDs make the merge trivial (identical content). + +## 3. The new write path: `WorkoutStore` replaces `SyncEngine` + +A new `@Observable @MainActor` type (working name `WorkoutStore`) with a +**deliberately identical public surface** to today's `SyncEngine` where +possible, so the ~40 view call sites (inventoried below) change their +environment key and little else: + +- `save(routine:)`, `save(workout:)`, `save(schedule:)`, `delete(…)` — same + signatures, still taking **Document DTOs**. Internally: map DTO → upsert + `@Model` via `CacheMapper` (which already exists and stays), save context. + No file write, no observer round-trip — `@Query` views update immediately. +- `ingestFromWatch(_:)` — same per-log merge logic (`WorkoutMergePlanner` + survives unchanged; it operates on documents), then upsert. +- `writeBackMachineSettings(…)`, `scanForDuplicates()` — port as-is. +- `currentRoutineID(for:)` — survives only as long as clone-on-edit does + (see §5); ideally becomes the identity function and is then deleted. +- **Gone entirely**: `connect()`, `iCloudStatus`, `abandonWaiting()`, + `flushPendingWrites()` + `WriteBacklog` (writes are local and synchronous — + there is nothing to backlog), `handle(_:)` observer deltas, + `MetadataObserver`, tombstone plumbing, settle delays. + +Keeping documents as the mutation DTO is the load-bearing decision that makes +this migration tractable: the inventory shows document types are pervasive in +views, the watch app, `WCPayload`, and `HealthKitMapping` — none of that +churns. `Documents.swift` gets retitled in comments from "on-disk format" to +"wire/DTO format"; `VersionedDocument` conformance and `relativePath` move +behind a legacy-import-only extension (needed only by §4). + +## 4. One-time migration of existing user data + +On first launch of the new version (guarded by a persisted migration flag): + +1. **Import**: enumerate the ubiquity container exactly as `reconcile()` does + today (IndieSync's placeholder-aware enumeration — this is the one place + IndieSync is still linked), decode every live document, and upsert into + SwiftData via `CacheMapper`. Honor tombstones: a stub for id X means X is + not imported. Record deleted-seed stubs into the new seed-veto store (§5). +2. **Do not delete the files.** Leave the container intact for ≥2 release + cycles as a recovery escape hatch. A later release removes the + `CloudDocuments` entitlement and (optionally) offers a cleanup. +3. **Multi-device staggering**: device 1 upgrades and imports at time T; + device 2 keeps writing *files* until it upgrades, then imports *its* view + of the container. Both imports upsert by ULID into the same CloudKit + private DB — last-writer-wins per record, dedup pass (§2.4) cleans up + races. Edits made on the not-yet-upgraded device after T that never + reached the container before device 1's import are picked up when device + 2 itself imports. The convergence guarantee is: every device imports its + own container replica once, and ULID-upsert makes that idempotent. +4. **Cache-wipe semantics change permanently**: the store is no longer + disposable. `wipeIfNeeded()` / schema-version-bump-wipes are removed; + from this point on, schema changes require real `VersionedSchema` + migrations (§7). The account-change wipe also goes away — CloudKit + mirroring handles account switching itself (the mirror re-syncs; local + store is per-account managed by the system). +5. **Failure handling**: import is all-or-nothing per document but tolerant + overall (a corrupt file is logged and skipped — same policy as today's + reconcile). The migration flag is set only after the enumeration + completes; a crash mid-import re-runs it (idempotent by ULID upsert). + +## 5. Subsystem-by-subsystem disposition + +Grounded in the coupling inventory (working tree, post Splits→Routines rename): + +| Subsystem | Disposition | +|---|---| +| `SyncEngine` (~1000 lines) | Replaced by `WorkoutStore` (§3); expect it to shrink to ~⅓ the size | +| IndieSync SPM package | Dropped from all 4 targets after the legacy import window closes; during the window, linked only for the import path (`DocumentFileStore` enumeration + `Tombstone` decode) | +| Tombstones / `TombstoneStore` | **Deleted** — CloudKit propagates deletes. One residue: the seed veto (below) | +| Seed system (`SeedLibrary`, `SeedReconcilePlanner`, auto-seed, restore) | Simplifies drastically. Keep fixed ULIDs. Seeding = upsert seed entities if absent **and not vetoed**. The resurrection veto becomes a tiny synced record (e.g. a `deletedSeedIDs` list on a singleton `AppState` model, or a `SeedVeto` model) instead of stub files. `reconcileSeeds()` becomes "upsert newer seed content by ULID unless user-forked". **Restore Starter Routines** stays: clear veto + re-upsert | +| Clone-on-edit (`cloneSeedOnEdit`, `cloneRedirects`, `currentRoutineID(for:)`, `repointWorkouts`) | **Decision point.** It existed because a fixed-ULID *file* could be resurrected by reconcile, so user edits had to fork away from seed identity. With a DB + veto record, an edited seed can simply… be edited (mark it `userModified: Bool` so reconcile skips upgrading it). Recommendation: retire clone-on-edit; `currentRoutineID(for:)` becomes identity and its 15 call sites collapse. This deletes the subtlest code in the app | +| `WriteBacklog` + `flushPendingWrites` + `SyncStatusBanner` write-queue state | **Deleted** — no async write path to backlog. Banner either goes away or repurposes for CK account status | +| `RootGateView` iCloud gate | **Deleted.** The app works offline/signed-out; CloudKit mirrors when it can. This removes a whole class of first-run failure (today's "no iCloud → no app") | +| Diagnostics (`ContainerStatus`, `DocumentSyncInspector`, `SchemaSkipScanner`, `DiagnosticsReport`) | Gutted. File/metadata/schema-skip diagnostics are meaningless. Keep `NetworkReadiness`; add `CKContainer.accountStatus`. Accept that SwiftData mirroring offers no per-record sync visibility (revisit Core Data only if this proves unacceptable in practice) | +| `DuplicateCleanup` dev tool | Replace with the §2.4 dedup pass; the Settings dev screen can surface its results | +| IndieBackup (`AppBackupConfiguration`) | Keep — it's persistence-agnostic. `backupRoot` changes from the ubiquity `Documents/` tree to a **staged JSON export**: `prepareForBackup` serializes all entities to documents (the mappers already exist) into a local folder; restore imports them and `rebuildCacheAfterRestore` becomes "import the JSON". Backup remains human-readable JSON — a deliberate property worth preserving | +| Watch (`WatchConnectivityBridge`, `WatchCacheApplier`, `WCPayload`) | **Unchanged.** The watch keeps its local non-CK SwiftData store fed by phone pushes; wire format stays documents; watch entitlements untouched. (Option considered and rejected: giving the watch its own CloudKit mirror — worse latency for live workouts, new entitlement, and the WC relay already works) | +| `HealthKitMapping` | Unchanged (consumes documents; documents survive as DTOs) | +| `ScreenshotSeed` / `ScreenshotRootView` | Unchanged (already constructs entities directly; `jsonRelativePath` args drop out when the field is removed) | +| `jsonRelativePath` on entities | Removed (a schema migration, §7) | +| `Scripts/generate_starter_splits.swift` | Survives — seeds stay bundled canonical JSON with fixed ULIDs; only the *consumer* changes (decode → upsert entities instead of writing bytes to the container) | +| `WorkoutsModelContainer` | Loses `wipeIfNeeded`/`wipeIfAccountChanged`; gains `cloudKitDatabase: .private("iCloud.dev.rzen.indie.Workouts")` on iOS and keeps `.none` on watchOS (one `make()` with a per-platform flag) | +| Entitlements / `project.yml` | iOS: add `CloudKit` to `icloud-services` (keep `CloudDocuments` during the import window, remove later). Watch: no change. Drop IndieSync package refs at the end | + +## 6. CloudKit operational realities (Option A) + +- **Schema is additive-only in production.** Once the CK schema deploys, + record fields can never be removed or renamed on the server — only added. + Local SwiftData migrations stay flexible; the CK record type accretes. + This makes §2.3's field cleanup (e.g. dropping `jsonRelativePath`) + something to do **before** first CK deployment, not after. +- **Deploy the schema to the production CK environment before App Store + release** (Development → Production promotion in CK Console); TestFlight + builds use the production environment — sequence the rollout accordingly. +- **First sync of a large history**: month-bucketed workout history could be + years of records; initial mirroring is background and throttled. Set + expectations: no progress UI is possible with SwiftData (see Diagnostics). +- **Quota**: private-DB data counts against the user's iCloud storage, same + as the current documents — no change in story. +- The current `WorkoutDocument.relativePath` month-bucketing rationale + disappears entirely — CK doesn't care; ULIDs already sort chronologically. + +## 7. Schema versioning going forward + +Today: bump `WorkoutsModelContainer.currentSchemaVersion` (now 8) → wipe → +rebuild from files. That option dies with this migration. Replacement: + +- Freeze the current entity shape as `SchemaV1` (`VersionedSchema`), define a + `SchemaMigrationPlan`, and route all future shape changes through + lightweight (preferred) or custom migration stages. +- Document schema versions (`RoutineDocument.currentSchemaVersion` etc.) stay, + but now gate only the **wire/DTO/backup** formats (watch payloads, backup + exports, legacy import) — decoupled from the store schema. +- The forward-compat quarantine ("file written by a newer app version") has + no CK equivalent; CloudKit handles unknown-field tolerance natively + (unknown record fields are preserved, not decoded). Older app + newer + schema coexistence is governed by CK's additive-only rule instead. + +## 8. Execution phases + +Each phase is independently shippable; stop-points between all of them. + +- **Phase 0 — prerequisites (do first, ships with current architecture)** + 1. Land the in-flight UX redesign / Routines rename. **Do not start this + migration on top of the current uncommitted tree.** + 2. Introduce the upsert-by-ULID helper and non-optional relationship + accessors so later diffs are mechanical. + 3. Decide §2.2 (CloudKit vs local-only) and the clone-on-edit retirement + (§5) — the two genuine product decisions in this plan. +- **Phase 1 — the flip (biggest single change)** + `WorkoutStore` replaces `SyncEngine`; entity changes (§2.3, minus CK + enablement); one-time import (§4); delete tombstones/backlog/gate/seed + machinery per §5; rewire Backup; gut Diagnostics. CK **not yet enabled** + (`cloudKitDatabase` still `.none`) — this ships as a local-only build + behind full regression testing, or goes straight to Phase 2 in the same + release if confidence is high. Watch untouched; all watch tests must pass + unmodified. +- **Phase 2 — enable CloudKit mirroring** + Entitlement + container config, `.private(...)` database, dedup pass, + seed-veto record, CK Console schema deploy to production. Multi-device + testing matrix: fresh install, upgrade-with-data, two-device stagger (§4.3), + account switch, airplane-mode edits on both devices then reconnect. +- **Phase 3 — decommission (a release or two later)** + Drop IndieSync from `project.yml`, remove the legacy import path and + `CloudDocuments` entitlement, optionally offer container cleanup, delete + `relativePath`/`VersionedDocument` residue from `Documents.swift`. + +## 9. Test strategy + +- The pure planners (`WorkoutMergePlanner`, seed decision logic) keep their + unit tests nearly verbatim — they operate on documents. +- New unit targets: upsert-by-ULID idempotence, dedup pass, legacy import + (fixture container → expected entity graph, including tombstone honoring + and corrupt-file skip), backup export/import round-trip. +- Watch test suites (`WatchCacheApplierTests`, `WatchConnectivityBridgeTests`, + `SessionEndPlannerTests`) must pass **unchanged** — they are the proof the + watch boundary held. +- Device protocol (extend BULLETPROOFING.md): the Phase 2 multi-device matrix + above, plus reinstall-restore and iCloud-signed-out operation (which must + now *work* instead of gating). + +## 10. Risks & open questions + +| Risk | Mitigation | +|---|---| +| SwiftData CK mirroring is opaque (no sync events, no conflict hooks) | Accept for v1; Core Data fallback documented (§2.1); dedup pass covers the main correctness hole | +| CK schema additive-only lock-in | Clean the entity shape (drop dead fields) *before* Phase 2; review every field name once more at that gate | +| Migration bugs eat user data | Files left intact ≥2 releases (§4.2); import idempotent; backup feature works before Phase 1 ships | +| Duplicate records from device staggering | ULID upsert + dedup pass; seeds are the worst case and merge trivially | +| Losing human-readable files as a user-facing property | Backup export keeps JSON portability; consider a manual "Export data" share action | +| Retiring clone-on-edit changes seed-upgrade semantics (a user-edited seed no longer receives content upgrades) | That is arguably the *correct* behavior; `userModified` flag makes it explicit. Confirm before Phase 0 exit | +| SwiftData maturity surprises (mirroring edge cases) | Phase 1/Phase 2 split means CK can be delayed indefinitely without blocking the architectural cleanup | + +## 11. What gets deleted (the payoff) + +Tombstone store + resurrection veto files, reconcile settle-delays, write +backlog + flush hooks, `NSMetadataQuery` observers (both of them), eviction +handling, `NSFileVersion` conflict resolution, the iCloud hard gate, the +schema-skip scanner, duplicate cleanup tooling, clone-on-edit + redirect +resolution (15 call sites), five independent ubiquity-container resolutions, +and the entire class of "file arrived while / metadata index hasn't settled" +race conditions. Estimated net: **-2,000–3,000 lines** of the app's subtlest +code, in exchange for one dedup pass and a real schema-migration discipline.