Record the persistence-migration proposal (documents -> pure database)
Claude-Session: https://claude.ai/code/session_012qw2itfzKyEJ1HpsFt8Ex4
This commit is contained in:
@@ -0,0 +1,291 @@
|
||||
# Persistence Migration Plan: iCloud Drive Documents → Pure Database
|
||||
|
||||
Status: **proposal — not started.** This documents the plan for retiring the
|
||||
iCloud-Drive-documents-as-source-of-truth architecture (IndieSync, `SyncEngine`,
|
||||
JSON files + rebuildable SwiftData cache) in favor of a database as the sole
|
||||
source of truth. Nothing in this file is committed-to; it exists so the shift
|
||||
can be evaluated and, if approved, executed in well-scoped phases.
|
||||
|
||||
---
|
||||
|
||||
## 1. Where we are (summary of the current architecture)
|
||||
|
||||
- JSON documents in the ubiquity container `iCloud.dev.rzen.indie.Workouts`
|
||||
are the source of truth: `Splits/<ULID>.json` (routines), `Workouts/YYYY/MM/<ULID>.json`,
|
||||
`Schedules/…`, `Stubs/<id>.json` tombstones.
|
||||
- SwiftData (`WorkoutsModelContainer`, `cloudKitDatabase: .none`) is a
|
||||
**rebuildable read-through cache**, wiped on schema bump or account change.
|
||||
- One-way flow: view → `SyncEngine.save(document)` → file write →
|
||||
`NSMetadataQuery` observer → `CacheMapper` upsert → `@Query` refresh.
|
||||
- The phone is the sole writer; the watch is a WatchConnectivity relay using
|
||||
the same `@Model` entities, fed only by phone pushes.
|
||||
- Supporting machinery this architecture *required us to build*: tombstones +
|
||||
resurrection veto, seed reconcile planner, write backlog + flush hooks,
|
||||
settle-delay auto-seeding, conflict resolution via `NSFileVersion`,
|
||||
eviction-safe reads, schema-skip forward gate, duplicate cleanup tooling,
|
||||
a diagnostics subsystem, and a hard iCloud gate (`RootGateView`) because the
|
||||
app cannot function without the container.
|
||||
|
||||
That last list is the honest motivation for this migration: most of the
|
||||
system's complexity exists to compensate for filesystem-as-database.
|
||||
|
||||
## 2. Target architecture (recommendation)
|
||||
|
||||
**SwiftData as the sole source of truth, with CloudKit mirroring for
|
||||
cross-device sync and reinstall durability.** Documents survive only as DTOs:
|
||||
the WatchConnectivity wire format, the HealthKit-mapping input, and the
|
||||
view-layer mutation payload.
|
||||
|
||||
### 2.1 Database choice: SwiftData vs Core Data
|
||||
|
||||
| Consideration | SwiftData | Core Data (`NSPersistentCloudKitContainer`) |
|
||||
|---|---|---|
|
||||
| Existing code | The 5 `@Model` entities, `@Query` views, and the container already exist — the cache *becomes* the database | Full rewrite of the entity layer + view fetch layer |
|
||||
| Swift 6 strict concurrency | Native (`@Model`, `ModelActor`) | Achievable but fighting ObjC-era API |
|
||||
| CloudKit sync status / observability | **Opaque.** No public sync-event API; diagnostics limited to network + account status | `eventChangedNotification` gives import/export/setup events with errors |
|
||||
| Deduplication after concurrent inserts | `HistoryDescriptor` (iOS 18+) or fetch-based dedup passes | Persistent history transactions (Apple's canonical dedup sample) |
|
||||
| Sharing / public DB (future) | Not exposed | Supported |
|
||||
| Migration tooling | `VersionedSchema` + `SchemaMigrationPlan` | Mapping models / lightweight migration |
|
||||
|
||||
**Recommendation: SwiftData.** The entities, container, and every list view
|
||||
already speak SwiftData; iOS 26 SwiftData is mature enough for this app's
|
||||
simple graph (5 models, 2 relationships). The one real cost — opaque sync
|
||||
status — is acceptable for a fitness app and is partly mitigated in §6
|
||||
(Diagnostics). Core Data is the fallback **only if** a hard requirement for
|
||||
sync observability or CloudKit sharing emerges; the escape hatch stays open
|
||||
because SwiftData and Core Data can read the same store, but plan as if the
|
||||
choice is final.
|
||||
|
||||
### 2.2 Sync choice: CloudKit mirroring vs local-only
|
||||
|
||||
Two legitimate options; decide before Phase 3:
|
||||
|
||||
- **Option A — CloudKit mirroring (recommended).** `cloudKitDatabase: .automatic`
|
||||
(private DB). Keeps multi-device sync and free reinstall restore. Offline-first
|
||||
is preserved (mirroring queues locally). Deletes propagate natively —
|
||||
**the entire tombstone subsystem disappears.**
|
||||
- **Option B — local-only database.** Simplest possible system; the watch
|
||||
already syncs via WatchConnectivity, not iCloud, so watch support is
|
||||
unaffected. But: no reinstall restore, no iPad/second-phone story, and
|
||||
backup becomes the *only* durability mechanism. Only choose this if
|
||||
multi-device and reinstall-restore are explicitly non-goals.
|
||||
|
||||
The rest of this plan assumes **Option A**, and notes where Option B would
|
||||
simplify a step.
|
||||
|
||||
### 2.3 Entity changes required for CloudKit
|
||||
|
||||
From the coupling inventory (`Shared/Model/Entities.swift`):
|
||||
|
||||
1. **Drop `@Attribute(.unique)` from `id` on all 5 models** — CloudKit does
|
||||
not support unique constraints; the container will refuse to mirror
|
||||
otherwise. Uniqueness becomes an app-level invariant: every insert path
|
||||
goes through an upsert helper (fetch-by-ULID first), and a dedup pass
|
||||
(§2.4) handles remote races.
|
||||
2. **Relationships must be optional**: `Routine.exercises: [Exercise]` and
|
||||
`Workout.logs: [WorkoutLog]` become `[Exercise]?` / `[WorkoutLog]?`
|
||||
(computed non-optional accessors can hide this from views). The existing
|
||||
`.cascade` delete rules are fine; the optional to-ones (`Exercise.routine`,
|
||||
`WorkoutLog.workout`) are already compatible.
|
||||
3. All other stored properties already carry defaults or are optional — ✅
|
||||
compatible today (good news from the inventory).
|
||||
4. Keep ULID string `id`s. They remain the stable identity across devices,
|
||||
the watch wire, and HealthKit metadata — nothing about CloudKit changes
|
||||
that.
|
||||
5. **`Schedule.routineID` stays a string join** (no live relationship) —
|
||||
consistent with the existing denormalized `Workout.routineID` design and
|
||||
avoids CK relationship-integrity headaches.
|
||||
|
||||
### 2.4 Deduplication (CloudKit-specific, unavoidable)
|
||||
|
||||
Without unique constraints, two devices can insert the same logical record
|
||||
(same ULID) before syncing — most likely for **seeds** (both devices seed on
|
||||
first launch) and for the initial migration import (§4). Required:
|
||||
|
||||
- An idempotent **dedup pass keyed by ULID**: on remote-change import (or, at
|
||||
minimum, on app foreground), fetch ids with count > 1, keep the row with
|
||||
the newest `updatedAt`, merge child rows, delete the rest.
|
||||
- Seeds specifically: fixed ULIDs make the merge trivial (identical content).
|
||||
|
||||
## 3. The new write path: `WorkoutStore` replaces `SyncEngine`
|
||||
|
||||
A new `@Observable @MainActor` type (working name `WorkoutStore`) with a
|
||||
**deliberately identical public surface** to today's `SyncEngine` where
|
||||
possible, so the ~40 view call sites (inventoried below) change their
|
||||
environment key and little else:
|
||||
|
||||
- `save(routine:)`, `save(workout:)`, `save(schedule:)`, `delete(…)` — same
|
||||
signatures, still taking **Document DTOs**. Internally: map DTO → upsert
|
||||
`@Model` via `CacheMapper` (which already exists and stays), save context.
|
||||
No file write, no observer round-trip — `@Query` views update immediately.
|
||||
- `ingestFromWatch(_:)` — same per-log merge logic (`WorkoutMergePlanner`
|
||||
survives unchanged; it operates on documents), then upsert.
|
||||
- `writeBackMachineSettings(…)`, `scanForDuplicates()` — port as-is.
|
||||
- `currentRoutineID(for:)` — survives only as long as clone-on-edit does
|
||||
(see §5); ideally becomes the identity function and is then deleted.
|
||||
- **Gone entirely**: `connect()`, `iCloudStatus`, `abandonWaiting()`,
|
||||
`flushPendingWrites()` + `WriteBacklog` (writes are local and synchronous —
|
||||
there is nothing to backlog), `handle(_:)` observer deltas,
|
||||
`MetadataObserver`, tombstone plumbing, settle delays.
|
||||
|
||||
Keeping documents as the mutation DTO is the load-bearing decision that makes
|
||||
this migration tractable: the inventory shows document types are pervasive in
|
||||
views, the watch app, `WCPayload`, and `HealthKitMapping` — none of that
|
||||
churns. `Documents.swift` gets retitled in comments from "on-disk format" to
|
||||
"wire/DTO format"; `VersionedDocument` conformance and `relativePath` move
|
||||
behind a legacy-import-only extension (needed only by §4).
|
||||
|
||||
## 4. One-time migration of existing user data
|
||||
|
||||
On first launch of the new version (guarded by a persisted migration flag):
|
||||
|
||||
1. **Import**: enumerate the ubiquity container exactly as `reconcile()` does
|
||||
today (IndieSync's placeholder-aware enumeration — this is the one place
|
||||
IndieSync is still linked), decode every live document, and upsert into
|
||||
SwiftData via `CacheMapper`. Honor tombstones: a stub for id X means X is
|
||||
not imported. Record deleted-seed stubs into the new seed-veto store (§5).
|
||||
2. **Do not delete the files.** Leave the container intact for ≥2 release
|
||||
cycles as a recovery escape hatch. A later release removes the
|
||||
`CloudDocuments` entitlement and (optionally) offers a cleanup.
|
||||
3. **Multi-device staggering**: device 1 upgrades and imports at time T;
|
||||
device 2 keeps writing *files* until it upgrades, then imports *its* view
|
||||
of the container. Both imports upsert by ULID into the same CloudKit
|
||||
private DB — last-writer-wins per record, dedup pass (§2.4) cleans up
|
||||
races. Edits made on the not-yet-upgraded device after T that never
|
||||
reached the container before device 1's import are picked up when device
|
||||
2 itself imports. The convergence guarantee is: every device imports its
|
||||
own container replica once, and ULID-upsert makes that idempotent.
|
||||
4. **Cache-wipe semantics change permanently**: the store is no longer
|
||||
disposable. `wipeIfNeeded()` / schema-version-bump-wipes are removed;
|
||||
from this point on, schema changes require real `VersionedSchema`
|
||||
migrations (§7). The account-change wipe also goes away — CloudKit
|
||||
mirroring handles account switching itself (the mirror re-syncs; local
|
||||
store is per-account managed by the system).
|
||||
5. **Failure handling**: import is all-or-nothing per document but tolerant
|
||||
overall (a corrupt file is logged and skipped — same policy as today's
|
||||
reconcile). The migration flag is set only after the enumeration
|
||||
completes; a crash mid-import re-runs it (idempotent by ULID upsert).
|
||||
|
||||
## 5. Subsystem-by-subsystem disposition
|
||||
|
||||
Grounded in the coupling inventory (working tree, post Splits→Routines rename):
|
||||
|
||||
| Subsystem | Disposition |
|
||||
|---|---|
|
||||
| `SyncEngine` (~1000 lines) | Replaced by `WorkoutStore` (§3); expect it to shrink to ~⅓ the size |
|
||||
| IndieSync SPM package | Dropped from all 4 targets after the legacy import window closes; during the window, linked only for the import path (`DocumentFileStore` enumeration + `Tombstone` decode) |
|
||||
| Tombstones / `TombstoneStore` | **Deleted** — CloudKit propagates deletes. One residue: the seed veto (below) |
|
||||
| Seed system (`SeedLibrary`, `SeedReconcilePlanner`, auto-seed, restore) | Simplifies drastically. Keep fixed ULIDs. Seeding = upsert seed entities if absent **and not vetoed**. The resurrection veto becomes a tiny synced record (e.g. a `deletedSeedIDs` list on a singleton `AppState` model, or a `SeedVeto` model) instead of stub files. `reconcileSeeds()` becomes "upsert newer seed content by ULID unless user-forked". **Restore Starter Routines** stays: clear veto + re-upsert |
|
||||
| Clone-on-edit (`cloneSeedOnEdit`, `cloneRedirects`, `currentRoutineID(for:)`, `repointWorkouts`) | **Decision point.** It existed because a fixed-ULID *file* could be resurrected by reconcile, so user edits had to fork away from seed identity. With a DB + veto record, an edited seed can simply… be edited (mark it `userModified: Bool` so reconcile skips upgrading it). Recommendation: retire clone-on-edit; `currentRoutineID(for:)` becomes identity and its 15 call sites collapse. This deletes the subtlest code in the app |
|
||||
| `WriteBacklog` + `flushPendingWrites` + `SyncStatusBanner` write-queue state | **Deleted** — no async write path to backlog. Banner either goes away or repurposes for CK account status |
|
||||
| `RootGateView` iCloud gate | **Deleted.** The app works offline/signed-out; CloudKit mirrors when it can. This removes a whole class of first-run failure (today's "no iCloud → no app") |
|
||||
| Diagnostics (`ContainerStatus`, `DocumentSyncInspector`, `SchemaSkipScanner`, `DiagnosticsReport`) | Gutted. File/metadata/schema-skip diagnostics are meaningless. Keep `NetworkReadiness`; add `CKContainer.accountStatus`. Accept that SwiftData mirroring offers no per-record sync visibility (revisit Core Data only if this proves unacceptable in practice) |
|
||||
| `DuplicateCleanup` dev tool | Replace with the §2.4 dedup pass; the Settings dev screen can surface its results |
|
||||
| IndieBackup (`AppBackupConfiguration`) | Keep — it's persistence-agnostic. `backupRoot` changes from the ubiquity `Documents/` tree to a **staged JSON export**: `prepareForBackup` serializes all entities to documents (the mappers already exist) into a local folder; restore imports them and `rebuildCacheAfterRestore` becomes "import the JSON". Backup remains human-readable JSON — a deliberate property worth preserving |
|
||||
| Watch (`WatchConnectivityBridge`, `WatchCacheApplier`, `WCPayload`) | **Unchanged.** The watch keeps its local non-CK SwiftData store fed by phone pushes; wire format stays documents; watch entitlements untouched. (Option considered and rejected: giving the watch its own CloudKit mirror — worse latency for live workouts, new entitlement, and the WC relay already works) |
|
||||
| `HealthKitMapping` | Unchanged (consumes documents; documents survive as DTOs) |
|
||||
| `ScreenshotSeed` / `ScreenshotRootView` | Unchanged (already constructs entities directly; `jsonRelativePath` args drop out when the field is removed) |
|
||||
| `jsonRelativePath` on entities | Removed (a schema migration, §7) |
|
||||
| `Scripts/generate_starter_splits.swift` | Survives — seeds stay bundled canonical JSON with fixed ULIDs; only the *consumer* changes (decode → upsert entities instead of writing bytes to the container) |
|
||||
| `WorkoutsModelContainer` | Loses `wipeIfNeeded`/`wipeIfAccountChanged`; gains `cloudKitDatabase: .private("iCloud.dev.rzen.indie.Workouts")` on iOS and keeps `.none` on watchOS (one `make()` with a per-platform flag) |
|
||||
| Entitlements / `project.yml` | iOS: add `CloudKit` to `icloud-services` (keep `CloudDocuments` during the import window, remove later). Watch: no change. Drop IndieSync package refs at the end |
|
||||
|
||||
## 6. CloudKit operational realities (Option A)
|
||||
|
||||
- **Schema is additive-only in production.** Once the CK schema deploys,
|
||||
record fields can never be removed or renamed on the server — only added.
|
||||
Local SwiftData migrations stay flexible; the CK record type accretes.
|
||||
This makes §2.3's field cleanup (e.g. dropping `jsonRelativePath`)
|
||||
something to do **before** first CK deployment, not after.
|
||||
- **Deploy the schema to the production CK environment before App Store
|
||||
release** (Development → Production promotion in CK Console); TestFlight
|
||||
builds use the production environment — sequence the rollout accordingly.
|
||||
- **First sync of a large history**: month-bucketed workout history could be
|
||||
years of records; initial mirroring is background and throttled. Set
|
||||
expectations: no progress UI is possible with SwiftData (see Diagnostics).
|
||||
- **Quota**: private-DB data counts against the user's iCloud storage, same
|
||||
as the current documents — no change in story.
|
||||
- The current `WorkoutDocument.relativePath` month-bucketing rationale
|
||||
disappears entirely — CK doesn't care; ULIDs already sort chronologically.
|
||||
|
||||
## 7. Schema versioning going forward
|
||||
|
||||
Today: bump `WorkoutsModelContainer.currentSchemaVersion` (now 8) → wipe →
|
||||
rebuild from files. That option dies with this migration. Replacement:
|
||||
|
||||
- Freeze the current entity shape as `SchemaV1` (`VersionedSchema`), define a
|
||||
`SchemaMigrationPlan`, and route all future shape changes through
|
||||
lightweight (preferred) or custom migration stages.
|
||||
- Document schema versions (`RoutineDocument.currentSchemaVersion` etc.) stay,
|
||||
but now gate only the **wire/DTO/backup** formats (watch payloads, backup
|
||||
exports, legacy import) — decoupled from the store schema.
|
||||
- The forward-compat quarantine ("file written by a newer app version") has
|
||||
no CK equivalent; CloudKit handles unknown-field tolerance natively
|
||||
(unknown record fields are preserved, not decoded). Older app + newer
|
||||
schema coexistence is governed by CK's additive-only rule instead.
|
||||
|
||||
## 8. Execution phases
|
||||
|
||||
Each phase is independently shippable; stop-points between all of them.
|
||||
|
||||
- **Phase 0 — prerequisites (do first, ships with current architecture)**
|
||||
1. Land the in-flight UX redesign / Routines rename. **Do not start this
|
||||
migration on top of the current uncommitted tree.**
|
||||
2. Introduce the upsert-by-ULID helper and non-optional relationship
|
||||
accessors so later diffs are mechanical.
|
||||
3. Decide §2.2 (CloudKit vs local-only) and the clone-on-edit retirement
|
||||
(§5) — the two genuine product decisions in this plan.
|
||||
- **Phase 1 — the flip (biggest single change)**
|
||||
`WorkoutStore` replaces `SyncEngine`; entity changes (§2.3, minus CK
|
||||
enablement); one-time import (§4); delete tombstones/backlog/gate/seed
|
||||
machinery per §5; rewire Backup; gut Diagnostics. CK **not yet enabled**
|
||||
(`cloudKitDatabase` still `.none`) — this ships as a local-only build
|
||||
behind full regression testing, or goes straight to Phase 2 in the same
|
||||
release if confidence is high. Watch untouched; all watch tests must pass
|
||||
unmodified.
|
||||
- **Phase 2 — enable CloudKit mirroring**
|
||||
Entitlement + container config, `.private(...)` database, dedup pass,
|
||||
seed-veto record, CK Console schema deploy to production. Multi-device
|
||||
testing matrix: fresh install, upgrade-with-data, two-device stagger (§4.3),
|
||||
account switch, airplane-mode edits on both devices then reconnect.
|
||||
- **Phase 3 — decommission (a release or two later)**
|
||||
Drop IndieSync from `project.yml`, remove the legacy import path and
|
||||
`CloudDocuments` entitlement, optionally offer container cleanup, delete
|
||||
`relativePath`/`VersionedDocument` residue from `Documents.swift`.
|
||||
|
||||
## 9. Test strategy
|
||||
|
||||
- The pure planners (`WorkoutMergePlanner`, seed decision logic) keep their
|
||||
unit tests nearly verbatim — they operate on documents.
|
||||
- New unit targets: upsert-by-ULID idempotence, dedup pass, legacy import
|
||||
(fixture container → expected entity graph, including tombstone honoring
|
||||
and corrupt-file skip), backup export/import round-trip.
|
||||
- Watch test suites (`WatchCacheApplierTests`, `WatchConnectivityBridgeTests`,
|
||||
`SessionEndPlannerTests`) must pass **unchanged** — they are the proof the
|
||||
watch boundary held.
|
||||
- Device protocol (extend BULLETPROOFING.md): the Phase 2 multi-device matrix
|
||||
above, plus reinstall-restore and iCloud-signed-out operation (which must
|
||||
now *work* instead of gating).
|
||||
|
||||
## 10. Risks & open questions
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| SwiftData CK mirroring is opaque (no sync events, no conflict hooks) | Accept for v1; Core Data fallback documented (§2.1); dedup pass covers the main correctness hole |
|
||||
| CK schema additive-only lock-in | Clean the entity shape (drop dead fields) *before* Phase 2; review every field name once more at that gate |
|
||||
| Migration bugs eat user data | Files left intact ≥2 releases (§4.2); import idempotent; backup feature works before Phase 1 ships |
|
||||
| Duplicate records from device staggering | ULID upsert + dedup pass; seeds are the worst case and merge trivially |
|
||||
| Losing human-readable files as a user-facing property | Backup export keeps JSON portability; consider a manual "Export data" share action |
|
||||
| Retiring clone-on-edit changes seed-upgrade semantics (a user-edited seed no longer receives content upgrades) | That is arguably the *correct* behavior; `userModified` flag makes it explicit. Confirm before Phase 0 exit |
|
||||
| SwiftData maturity surprises (mirroring edge cases) | Phase 1/Phase 2 split means CK can be delayed indefinitely without blocking the architectural cleanup |
|
||||
|
||||
## 11. What gets deleted (the payoff)
|
||||
|
||||
Tombstone store + resurrection veto files, reconcile settle-delays, write
|
||||
backlog + flush hooks, `NSMetadataQuery` observers (both of them), eviction
|
||||
handling, `NSFileVersion` conflict resolution, the iCloud hard gate, the
|
||||
schema-skip scanner, duplicate cleanup tooling, clone-on-edit + redirect
|
||||
resolution (15 call sites), five independent ubiquity-container resolutions,
|
||||
and the entire class of "file arrived while / metadata index hasn't settled"
|
||||
race conditions. Estimated net: **-2,000–3,000 lines** of the app's subtlest
|
||||
code, in exchange for one dedup pass and a real schema-migration discipline.
|
||||
Reference in New Issue
Block a user