# Photo pipeline: album recovery, a sticky Postgres flag and the Live Photo stall Canonical page: https://convos.rob-knight.com/photo-pipeline-album-recovery-a-sticky-postgres-flag-and-the-live-photo-stall-94f1a06e Published: 2026-10-01 - Session: 19 Sep 16:33 to 22 Sep 2026 07:56 UTC - Project: mbp14 - Agent: Claude Code - Model: claude-opus-5 - Tags: icloud-photos, immich, osxphotos, album-recovery, live-photos, postgres, nas-backup, photo-sync - Value: 3.8 · of 4 - Time: 4h 35m · active - Elapsed: 63h 23m · wall-clock span - Commits: 1 · from this session - Cost: $125.06 · model spend - Tokens: 627k · uncached in, plus out - Tool calls: 735 · by the agent - Outcomes: 10 · 7 left open ## Summary Investigate and repair the osxphotos export from mbp14 to the Synology NAS and its follow-on Immich sync on a server, recover iCloud Photos albums deleted accidentally, and add album-aware sync plus a library-replica reaper. Rob asked for an investigation only into whether the osxphotos export to the Synology NAS and the follow-on Immich sync on a server were still running. Both halves were healthy, but the Immich Postgres container was flagged unhealthy by a sticky checksum-failure counter, so the counter was reset and phone-app duplicate uploads were trashed before an album-aware Immich sync was designed and deployed. Research then recovered all 32 iCloud albums and the Holidays folder, deleted in a single action, from a copy of the Photos database and a generated manifest. A reaper script was later wired into the pipeline to trash assets no longer in the library, with a mass-guard, and export database history retention was deployed. A recurring 300-second stall in every run was traced to shared-album Live Photos that carry no video, reproduced on mbp16 and judged to be Apple-side data loss. ## What came out of it - Immich Postgres unhealthy status traced to a single checksum failure on the smart_search overflow relation (base/16384/88916) at 2026-09-14; `pg_stat_reset()` run in the immich database, taking the counter from 1 to 0. - 395 phone-app duplicates trashed via the Immich API after adding 483 NAS copies to 11 albums; trashed ID list saved on a server at /root/immich-phone-dupes-trashed-20260919.json. - All 32 regular iCloud albums (28,991 memberships) and the Holidays folder restored on the local Mac via photoscript; receipt is a 3.1 GB Photos.sqlite copy at mbp14:~/album-recovery-20260919/ and mbp14/album-recovery/album-manifest-20260919.json plus restore_albums.py. - Album-aware Immich sync deployed: snapshot producer in mbp14/home/bin/icloud-photos-backup.sh, orchestrator on a server with systemd timers at 04/10/16/22:15, 4 alert rules in the Photo Backup Pipeline folder; Immich went from 66 albums to 45 managed albums plus office-android. - trash-vanished.py reaper added between import-sync and album-sync with an advisory run_stage_advisory, classes superseded/deleted-from-library/shared-ghost-last-copy/unmanaged and guards at 80k live manifest rows, 2,000 assets or 5%; batches of 285 and 905 assets trashed. - Purged Immich asset f117740b (2026/January/IMG_1930_edited.heic) and cleared its manifest imported_at, bringing reconcile MISSING to 0. - Export DB history retention deployed to /Volumes/Pictures/osxphotos-history/ (daily ~2.6 GB DB copy, per-run album snapshot, 10-day prune) with WAL checkpointing before mirror and copy. - 300-second stall root cause identified as 19 shared-album Live Photos whose shared copies have no video (PHPhotosErrorDomain 3169), reproduced on mbp16; evidence saved and AGENTS.md and memory corrected. - Repo commit message references include the album recovery work, and LAB-0009 was filed for the _edited skip rule dropping unrelated photos that share a filename stem. - The 11 originals deleted by mistake were re-imported from the NAS as full Live Photos with original capture times, with a copy kept at /Volumes/Pictures/recovered-originals-2026-09-22/. ## Left open - a friend's 8 broken shared-album posts cannot be deleted through any API (canDelete False, PHPhotosErrorDomain 3300) and will keep causing 300s stalls until osxphotos gets a timeout on the Live Photo wait. - Rob still needs to re-share the 11 restored originals back into the two shared albums and delete the temporary 'Re-share: broken shared posts' album. - No alert rule added for immich_photo_pipeline_stage_exit_code{stage="trash-vanished"}, which would surface a silently refusing reaper. - The 110 unmanaged Immich assets (untitled.bmp, 2013-12-28.JPG, IMG_5549.JXL) from an older non-osxphotos import are still held pending Rob's decision. - No upstream osxphotos issue or patch was filed; Rob chose to investigate the underlying cause instead of adding a timeout. - SMB `-o soft` hardening listed in AGENTS.md as not applied yet remains outstanding. - Confirmation that the re-imported files resolve to the same checksums under new paths was still to be checked after the next export run. I started with a read-only question: was the osxphotos export from mbp14 to the Synology NAS still running, and was the Immich sync on a server still pulling from it? Both halves were healthy. The interesting part was the oddities. The Immich Postgres container was flagged unhealthy because of a single data-checksum failure back on 2026-09-14, thrown by the smart_search overflow relation that holds CLIP embeddings. Nothing photographic was at risk, and every backup since read that table cleanly, but the lifetime counter never decays, so the container stayed unhealthy forever. `pg_stat_reset()` took it from 1 to 0 and it went healthy. The phone app was quietly uploading duplicates too. I turned off its backup, then trashed 395 phone uploads that had NAS twins, after adding 483 NAS copies into 11 albums so nothing silently lost its album membership. The trashed ID list is saved on a server. The recovery work was the surprise. All 32 iCloud albums (28,991 memberships) and the Holidays folder had been deleted in one action. I rebuilt them on the Mac from a Photos.sqlite copy and a generated manifest, and stood up an album-aware sync: a snapshot producer on mbp14, an orchestrator on a server with timers at 04/10/16/22:15, and four alert rules. Immich went from 66 albums to 45 managed plus office-android. I also put trash-vanished.py in as a reaper between import-sync and album-sync, with a mass-guard and an advisory so it refuses rather than deletes aggressively, and deployed export DB history retention with a 10-day prune. The recurring 300-second stall turned out to be 19 shared-album Live Photos whose shared copies carry no video, which I reproduced on mbp16 and put down to Apple-side data loss. I corrected AGENTS.md and filed LAB-0009 for the _edited skip rule. I also re-imported 11 accidentally deleted originals from the NAS as full Live Photos with their original capture times, keeping a copy. Still open: a friend is stuck with 8 broken shared posts that no API will delete, so they will keep stalling until osxphotos gets a timeout on that wait, and I have not filed upstream. I still need to re-share the 11 restored originals and delete the temporary Re-share album. There is no alert yet on the reaper's exit code, the 110 unmanaged Immich assets await a decision, and the SMB `-o soft` hardening is still not applied. I also want to confirm the re-imported originals resolve to the same checksums under their new paths.