# Persistence Migration Plan: iCloud Drive Documents → Pure Database Status: **proposal — not started.** This documents the plan for retiring the iCloud-Drive-documents-as-source-of-truth architecture (IndieSync, `SyncEngine`, JSON files + rebuildable SwiftData cache) in favor of a database as the sole source of truth. Nothing in this file is committed-to; it exists so the shift can be evaluated and, if approved, executed in well-scoped phases. --- ## 1. Where we are (summary of the current architecture) - JSON documents in the ubiquity container `iCloud.dev.rzen.indie.Workouts` are the source of truth: `Splits/.json` (routines), `Workouts/YYYY/MM/.json`, `Schedules/…`, `Stubs/.json` tombstones. - SwiftData (`WorkoutsModelContainer`, `cloudKitDatabase: .none`) is a **rebuildable read-through cache**, wiped on schema bump or account change. - One-way flow: view → `SyncEngine.save(document)` → file write → `NSMetadataQuery` observer → `CacheMapper` upsert → `@Query` refresh. - The phone is the sole writer; the watch is a WatchConnectivity relay using the same `@Model` entities, fed only by phone pushes. - Supporting machinery this architecture *required us to build*: tombstones + resurrection veto, seed reconcile planner, write backlog + flush hooks, settle-delay auto-seeding, conflict resolution via `NSFileVersion`, eviction-safe reads, schema-skip forward gate, duplicate cleanup tooling, a diagnostics subsystem, and a hard iCloud gate (`RootGateView`) because the app cannot function without the container. That last list is the honest motivation for this migration: most of the system's complexity exists to compensate for filesystem-as-database. ## 2. Target architecture (recommendation) **SwiftData as the sole source of truth, with CloudKit mirroring for cross-device sync and reinstall durability.** Documents survive only as DTOs: the WatchConnectivity wire format, the HealthKit-mapping input, and the view-layer mutation payload. ### 2.1 Database choice: SwiftData vs Core Data | Consideration | SwiftData | Core Data (`NSPersistentCloudKitContainer`) | |---|---|---| | Existing code | The 5 `@Model` entities, `@Query` views, and the container already exist — the cache *becomes* the database | Full rewrite of the entity layer + view fetch layer | | Swift 6 strict concurrency | Native (`@Model`, `ModelActor`) | Achievable but fighting ObjC-era API | | CloudKit sync status / observability | **Opaque.** No public sync-event API; diagnostics limited to network + account status | `eventChangedNotification` gives import/export/setup events with errors | | Deduplication after concurrent inserts | `HistoryDescriptor` (iOS 18+) or fetch-based dedup passes | Persistent history transactions (Apple's canonical dedup sample) | | Sharing / public DB (future) | Not exposed | Supported | | Migration tooling | `VersionedSchema` + `SchemaMigrationPlan` | Mapping models / lightweight migration | **Recommendation: SwiftData.** The entities, container, and every list view already speak SwiftData; iOS 26 SwiftData is mature enough for this app's simple graph (5 models, 2 relationships). The one real cost — opaque sync status — is acceptable for a fitness app and is partly mitigated in §6 (Diagnostics). Core Data is the fallback **only if** a hard requirement for sync observability or CloudKit sharing emerges; the escape hatch stays open because SwiftData and Core Data can read the same store, but plan as if the choice is final. ### 2.2 Sync choice: CloudKit mirroring vs local-only Two legitimate options; decide before Phase 3: - **Option A — CloudKit mirroring (recommended).** `cloudKitDatabase: .automatic` (private DB). Keeps multi-device sync and free reinstall restore. Offline-first is preserved (mirroring queues locally). Deletes propagate natively — **the entire tombstone subsystem disappears.** - **Option B — local-only database.** Simplest possible system; the watch already syncs via WatchConnectivity, not iCloud, so watch support is unaffected. But: no reinstall restore, no iPad/second-phone story, and backup becomes the *only* durability mechanism. Only choose this if multi-device and reinstall-restore are explicitly non-goals. The rest of this plan assumes **Option A**, and notes where Option B would simplify a step. ### 2.3 Entity changes required for CloudKit From the coupling inventory (`Shared/Model/Entities.swift`): 1. **Drop `@Attribute(.unique)` from `id` on all 5 models** — CloudKit does not support unique constraints; the container will refuse to mirror otherwise. Uniqueness becomes an app-level invariant: every insert path goes through an upsert helper (fetch-by-ULID first), and a dedup pass (§2.4) handles remote races. 2. **Relationships must be optional**: `Routine.exercises: [Exercise]` and `Workout.logs: [WorkoutLog]` become `[Exercise]?` / `[WorkoutLog]?` (computed non-optional accessors can hide this from views). The existing `.cascade` delete rules are fine; the optional to-ones (`Exercise.routine`, `WorkoutLog.workout`) are already compatible. 3. All other stored properties already carry defaults or are optional — ✅ compatible today (good news from the inventory). 4. Keep ULID string `id`s. They remain the stable identity across devices, the watch wire, and HealthKit metadata — nothing about CloudKit changes that. 5. **`Schedule.routineID` stays a string join** (no live relationship) — consistent with the existing denormalized `Workout.routineID` design and avoids CK relationship-integrity headaches. ### 2.4 Deduplication (CloudKit-specific, unavoidable) Without unique constraints, two devices can insert the same logical record (same ULID) before syncing — most likely for **seeds** (both devices seed on first launch) and for the initial migration import (§4). Required: - An idempotent **dedup pass keyed by ULID**: on remote-change import (or, at minimum, on app foreground), fetch ids with count > 1, keep the row with the newest `updatedAt`, merge child rows, delete the rest. - Seeds specifically: fixed ULIDs make the merge trivial (identical content). ## 3. The new write path: `WorkoutStore` replaces `SyncEngine` A new `@Observable @MainActor` type (working name `WorkoutStore`) with a **deliberately identical public surface** to today's `SyncEngine` where possible, so the ~40 view call sites (inventoried below) change their environment key and little else: - `save(routine:)`, `save(workout:)`, `save(schedule:)`, `delete(…)` — same signatures, still taking **Document DTOs**. Internally: map DTO → upsert `@Model` via `CacheMapper` (which already exists and stays), save context. No file write, no observer round-trip — `@Query` views update immediately. - `ingestFromWatch(_:)` — same per-log merge logic (`WorkoutMergePlanner` survives unchanged; it operates on documents), then upsert. - `writeBackMachineSettings(…)`, `scanForDuplicates()` — port as-is. - `currentRoutineID(for:)` — survives only as long as clone-on-edit does (see §5); ideally becomes the identity function and is then deleted. - **Gone entirely**: `connect()`, `iCloudStatus`, `abandonWaiting()`, `flushPendingWrites()` + `WriteBacklog` (writes are local and synchronous — there is nothing to backlog), `handle(_:)` observer deltas, `MetadataObserver`, tombstone plumbing, settle delays. Keeping documents as the mutation DTO is the load-bearing decision that makes this migration tractable: the inventory shows document types are pervasive in views, the watch app, `WCPayload`, and `HealthKitMapping` — none of that churns. `Documents.swift` gets retitled in comments from "on-disk format" to "wire/DTO format"; `VersionedDocument` conformance and `relativePath` move behind a legacy-import-only extension (needed only by §4). ## 4. One-time migration of existing user data On first launch of the new version (guarded by a persisted migration flag): 1. **Import**: enumerate the ubiquity container exactly as `reconcile()` does today (IndieSync's placeholder-aware enumeration — this is the one place IndieSync is still linked), decode every live document, and upsert into SwiftData via `CacheMapper`. Honor tombstones: a stub for id X means X is not imported. Record deleted-seed stubs into the new seed-veto store (§5). 2. **Do not delete the files.** Leave the container intact for ≥2 release cycles as a recovery escape hatch. A later release removes the `CloudDocuments` entitlement and (optionally) offers a cleanup. 3. **Multi-device staggering**: device 1 upgrades and imports at time T; device 2 keeps writing *files* until it upgrades, then imports *its* view of the container. Both imports upsert by ULID into the same CloudKit private DB — last-writer-wins per record, dedup pass (§2.4) cleans up races. Edits made on the not-yet-upgraded device after T that never reached the container before device 1's import are picked up when device 2 itself imports. The convergence guarantee is: every device imports its own container replica once, and ULID-upsert makes that idempotent. 4. **Cache-wipe semantics change permanently**: the store is no longer disposable. `wipeIfNeeded()` / schema-version-bump-wipes are removed; from this point on, schema changes require real `VersionedSchema` migrations (§7). The account-change wipe also goes away — CloudKit mirroring handles account switching itself (the mirror re-syncs; local store is per-account managed by the system). 5. **Failure handling**: import is all-or-nothing per document but tolerant overall (a corrupt file is logged and skipped — same policy as today's reconcile). The migration flag is set only after the enumeration completes; a crash mid-import re-runs it (idempotent by ULID upsert). ## 5. Subsystem-by-subsystem disposition Grounded in the coupling inventory (working tree, post Splits→Routines rename): | Subsystem | Disposition | |---|---| | `SyncEngine` (~1000 lines) | Replaced by `WorkoutStore` (§3); expect it to shrink to ~⅓ the size | | IndieSync SPM package | Dropped from all 4 targets after the legacy import window closes; during the window, linked only for the import path (`DocumentFileStore` enumeration + `Tombstone` decode) | | Tombstones / `TombstoneStore` | **Deleted** — CloudKit propagates deletes. One residue: the seed veto (below) | | Seed system (`SeedLibrary`, `SeedReconcilePlanner`, auto-seed, restore) | Simplifies drastically. Keep fixed ULIDs. Seeding = upsert seed entities if absent **and not vetoed**. The resurrection veto becomes a tiny synced record (e.g. a `deletedSeedIDs` list on a singleton `AppState` model, or a `SeedVeto` model) instead of stub files. `reconcileSeeds()` becomes "upsert newer seed content by ULID unless user-forked". **Restore Starter Routines** stays: clear veto + re-upsert | | Clone-on-edit (`cloneSeedOnEdit`, `cloneRedirects`, `currentRoutineID(for:)`, `repointWorkouts`) | **Decision point.** It existed because a fixed-ULID *file* could be resurrected by reconcile, so user edits had to fork away from seed identity. With a DB + veto record, an edited seed can simply… be edited (mark it `userModified: Bool` so reconcile skips upgrading it). Recommendation: retire clone-on-edit; `currentRoutineID(for:)` becomes identity and its 15 call sites collapse. This deletes the subtlest code in the app | | `WriteBacklog` + `flushPendingWrites` + `SyncStatusBanner` write-queue state | **Deleted** — no async write path to backlog. Banner either goes away or repurposes for CK account status | | `RootGateView` iCloud gate | **Deleted.** The app works offline/signed-out; CloudKit mirrors when it can. This removes a whole class of first-run failure (today's "no iCloud → no app") | | Diagnostics (`ContainerStatus`, `DocumentSyncInspector`, `SchemaSkipScanner`, `DiagnosticsReport`) | Gutted. File/metadata/schema-skip diagnostics are meaningless. Keep `NetworkReadiness`; add `CKContainer.accountStatus`. Accept that SwiftData mirroring offers no per-record sync visibility (revisit Core Data only if this proves unacceptable in practice) | | `DuplicateCleanup` dev tool | Replace with the §2.4 dedup pass; the Settings dev screen can surface its results | | IndieBackup (`AppBackupConfiguration`) | Keep — it's persistence-agnostic. `backupRoot` changes from the ubiquity `Documents/` tree to a **staged JSON export**: `prepareForBackup` serializes all entities to documents (the mappers already exist) into a local folder; restore imports them and `rebuildCacheAfterRestore` becomes "import the JSON". Backup remains human-readable JSON — a deliberate property worth preserving | | Watch (`WatchConnectivityBridge`, `WatchCacheApplier`, `WCPayload`) | **Unchanged.** The watch keeps its local non-CK SwiftData store fed by phone pushes; wire format stays documents; watch entitlements untouched. (Option considered and rejected: giving the watch its own CloudKit mirror — worse latency for live workouts, new entitlement, and the WC relay already works) | | `HealthKitMapping` | Unchanged (consumes documents; documents survive as DTOs) | | `ScreenshotSeed` / `ScreenshotRootView` | Unchanged (already constructs entities directly; `jsonRelativePath` args drop out when the field is removed) | | `jsonRelativePath` on entities | Removed (a schema migration, §7) | | `Scripts/generate_starter_splits.swift` | Survives — seeds stay bundled canonical JSON with fixed ULIDs; only the *consumer* changes (decode → upsert entities instead of writing bytes to the container) | | `WorkoutsModelContainer` | Loses `wipeIfNeeded`/`wipeIfAccountChanged`; gains `cloudKitDatabase: .private("iCloud.dev.rzen.indie.Workouts")` on iOS and keeps `.none` on watchOS (one `make()` with a per-platform flag) | | Entitlements / `project.yml` | iOS: add `CloudKit` to `icloud-services` (keep `CloudDocuments` during the import window, remove later). Watch: no change. Drop IndieSync package refs at the end | ## 6. CloudKit operational realities (Option A) - **Schema is additive-only in production.** Once the CK schema deploys, record fields can never be removed or renamed on the server — only added. Local SwiftData migrations stay flexible; the CK record type accretes. This makes §2.3's field cleanup (e.g. dropping `jsonRelativePath`) something to do **before** first CK deployment, not after. - **Deploy the schema to the production CK environment before App Store release** (Development → Production promotion in CK Console); TestFlight builds use the production environment — sequence the rollout accordingly. - **First sync of a large history**: month-bucketed workout history could be years of records; initial mirroring is background and throttled. Set expectations: no progress UI is possible with SwiftData (see Diagnostics). - **Quota**: private-DB data counts against the user's iCloud storage, same as the current documents — no change in story. - The current `WorkoutDocument.relativePath` month-bucketing rationale disappears entirely — CK doesn't care; ULIDs already sort chronologically. ## 7. Schema versioning going forward Today: bump `WorkoutsModelContainer.currentSchemaVersion` (now 8) → wipe → rebuild from files. That option dies with this migration. Replacement: - Freeze the current entity shape as `SchemaV1` (`VersionedSchema`), define a `SchemaMigrationPlan`, and route all future shape changes through lightweight (preferred) or custom migration stages. - Document schema versions (`RoutineDocument.currentSchemaVersion` etc.) stay, but now gate only the **wire/DTO/backup** formats (watch payloads, backup exports, legacy import) — decoupled from the store schema. - The forward-compat quarantine ("file written by a newer app version") has no CK equivalent; CloudKit handles unknown-field tolerance natively (unknown record fields are preserved, not decoded). Older app + newer schema coexistence is governed by CK's additive-only rule instead. ## 8. Execution phases Each phase is independently shippable; stop-points between all of them. - **Phase 0 — prerequisites (do first, ships with current architecture)** 1. Land the in-flight UX redesign / Routines rename. **Do not start this migration on top of the current uncommitted tree.** 2. Introduce the upsert-by-ULID helper and non-optional relationship accessors so later diffs are mechanical. 3. Decide §2.2 (CloudKit vs local-only) and the clone-on-edit retirement (§5) — the two genuine product decisions in this plan. - **Phase 1 — the flip (biggest single change)** `WorkoutStore` replaces `SyncEngine`; entity changes (§2.3, minus CK enablement); one-time import (§4); delete tombstones/backlog/gate/seed machinery per §5; rewire Backup; gut Diagnostics. CK **not yet enabled** (`cloudKitDatabase` still `.none`) — this ships as a local-only build behind full regression testing, or goes straight to Phase 2 in the same release if confidence is high. Watch untouched; all watch tests must pass unmodified. - **Phase 2 — enable CloudKit mirroring** Entitlement + container config, `.private(...)` database, dedup pass, seed-veto record, CK Console schema deploy to production. Multi-device testing matrix: fresh install, upgrade-with-data, two-device stagger (§4.3), account switch, airplane-mode edits on both devices then reconnect. - **Phase 3 — decommission (a release or two later)** Drop IndieSync from `project.yml`, remove the legacy import path and `CloudDocuments` entitlement, optionally offer container cleanup, delete `relativePath`/`VersionedDocument` residue from `Documents.swift`. ## 9. Test strategy - The pure planners (`WorkoutMergePlanner`, seed decision logic) keep their unit tests nearly verbatim — they operate on documents. - New unit targets: upsert-by-ULID idempotence, dedup pass, legacy import (fixture container → expected entity graph, including tombstone honoring and corrupt-file skip), backup export/import round-trip. - Watch test suites (`WatchCacheApplierTests`, `WatchConnectivityBridgeTests`, `SessionEndPlannerTests`) must pass **unchanged** — they are the proof the watch boundary held. - Device protocol (extend BULLETPROOFING.md): the Phase 2 multi-device matrix above, plus reinstall-restore and iCloud-signed-out operation (which must now *work* instead of gating). ## 10. Risks & open questions | Risk | Mitigation | |---|---| | SwiftData CK mirroring is opaque (no sync events, no conflict hooks) | Accept for v1; Core Data fallback documented (§2.1); dedup pass covers the main correctness hole | | CK schema additive-only lock-in | Clean the entity shape (drop dead fields) *before* Phase 2; review every field name once more at that gate | | Migration bugs eat user data | Files left intact ≥2 releases (§4.2); import idempotent; backup feature works before Phase 1 ships | | Duplicate records from device staggering | ULID upsert + dedup pass; seeds are the worst case and merge trivially | | Losing human-readable files as a user-facing property | Backup export keeps JSON portability; consider a manual "Export data" share action | | Retiring clone-on-edit changes seed-upgrade semantics (a user-edited seed no longer receives content upgrades) | That is arguably the *correct* behavior; `userModified` flag makes it explicit. Confirm before Phase 0 exit | | SwiftData maturity surprises (mirroring edge cases) | Phase 1/Phase 2 split means CK can be delayed indefinitely without blocking the architectural cleanup | ## 11. What gets deleted (the payoff) Tombstone store + resurrection veto files, reconcile settle-delays, write backlog + flush hooks, `NSMetadataQuery` observers (both of them), eviction handling, `NSFileVersion` conflict resolution, the iCloud hard gate, the schema-skip scanner, duplicate cleanup tooling, clone-on-edit + redirect resolution (15 call sites), five independent ubiquity-container resolutions, and the entire class of "file arrived while / metadata index hasn't settled" race conditions. Estimated net: **-2,000–3,000 lines** of the app's subtlest code, in exchange for one dedup pass and a real schema-migration discipline.