rzen/unique
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

pipeline: autonomous operation — shortlist stage, auto modes, daily-pipeline orchestrator

Browse Source 
- daily-finds: feed-first fetch strategy (fetch: feed + feedUrl), cadence-aware
  scanning, new score-and-shortlist stage (shortlist: true, up to 12/day, top
  1-3 flagged as review candidates), Amazon affiliate lookup for shortlisted
  physical products, and authority to apply mechanical sources.json maintenance
  (skip dead sources, demote dormant ones, record working fetch paths) instead
  of logging recommendations nobody executes.
- build-bundle: auto mode (no confirmation gate when unattended) and a 'weekly'
  date-based catch mode so shortlisted finds never sit invisible for weeks.
- cluster-bundles: auto mode.
- build-review: hero.ts registration step removed (heroes auto-discovered from
  hero.<ext>); mandatory fact-check against the product's own site in auto
  mode; amazonLink carry-through; auto mode publishes and logs instead of
  pausing twice for approval.
- New daily-pipeline skill: capture → bundle → reviews → pick of the day →
  build → commit → push → digest, with granular per-stage commits. This is the
  entrypoint for scheduled runs.
- Project CLAUDE.md documenting the pipeline and its foot-gun conventions.
This commit is contained in:
rzen
2026-07-03 15:21:27 -04:00
parent 80b3920715
commit 622d9d63fd
6 changed files with 228 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/daily-finds/SKILL.md
+73 -29
View File
@@ -1,18 +1,20 @@
---
name: daily-finds
description: Capture-only daily sweep for unique.rzen.dev. Scans every source in sources.json, writes one find file per qualifying item to src/content/find/, and nominates any new sources cited in the items it reads (plus optionally via WebSearch). Does NOT pick a winner, write a bundle, or address the user/audience. Picking happens via /build-bundle and /build-review. Use when the user asks for "today's finds", "morning sweep", "capture", or runs /daily-finds.
description: Capture-only daily sweep for unique.rzen.dev. Scans every source in sources.json, writes one find file per qualifying item to src/content/find/, marks the day's shortlist, adds Amazon affiliate links where the product is confidently matched, performs mechanical source maintenance, and nominates new sources. Does NOT write bundles or reviews — that's /build-bundle and /build-review (or /daily-pipeline for the whole chain). Use when the user asks for "today's finds", "morning sweep", "capture", or runs /daily-finds.
disable-model-invocation: true
---
# daily-finds
Pure capture: every item that passes the bar becomes a `find` file. No daily cap, no theme detection, no audience-facing post. Whether any of these finds appears in a `bundle` or graduates to a review is a separate editorial decision.
Pure capture plus triage: every item that passes the bar becomes a `find` file, and the run ends by ranking the day's captures into a **shortlist** (up to 12) that drives downstream automation. Whether a find appears in a `bundle` or graduates to a review is still a separate step — but the shortlist is the pipeline's selection signal.
This skill is designed to run unattended. Never pause for confirmation mid-run; make the calls, log them, and surface anything odd in the end-of-run digest.
Invoke from inside the `unique.rzen.dev` repo.
## Inputs
- `sources.json` at the project root — curated source list with per-source `fetch` strategy (`webfetch` / `curl` / `skip`) and optional `fetchUrl`.
- `sources.json` at the project root — curated source list with per-source fetch strategy (`feed` / `webfetch` / `curl` / `skip`), optional `feedUrl` (RSS/Atom) and `fetchUrl`.
- `daily-finds.log.md` — running log of every find slug ever written (used to skip duplicates).
- `src/content/find/<year>/<month>/<slug>/index.mdx` — already-captured finds (every existing slug is a "do not re-suggest" item; slugs are globally unique across years/months even though folders are nested by date).
- `src/content/reviews/<slug>/index.mdx` — already-reviewed items (do not re-suggest).
@@ -22,7 +24,7 @@ Invoke from inside the `unique.rzen.dev` repo.
A find qualifies if and only if it is **(a)** delightful, unusual, or distinctive in a way the audience would care about *and* **(b)** usable daily (relax to "memorable for a visitor" for travel). Drop generic mass-market items, ad-copy listicles, speculative / not-yet-shipping products, self-promo, and anything not traceable to a real currently-buyable / currently-visitable thing.
Crucially: do **not** filter by "is it the best of the day." Capture broadly. Editorial selection happens later.
Crucially: do **not** filter by "is it the best of the day." Capture broadly. Selection happens in the shortlist step, and editorial treatment later.
## Procedure
@@ -30,13 +32,16 @@ Crucially: do **not** filter by "is it the best of the day." Capture broadly. Ed
For each source in `sources.json`:
- **Default (`fetch: "webfetch"` or no field)**: WebFetch against `fetchUrl` if present, else `url`. Extract recently-featured items (name, link, one-line description).
- **`fetch: "curl"`**: shell out to `curl -sL -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" --max-time 25 "<fetchUrl|url>"` and parse the HTML body. For Reddit JSON endpoints, pipe through `python3 -c '…'` to extract titles, links, and selftext. Treat HTTP 403 with a non-empty HTML body as a content response (Atlas Obscura quirk) — only treat empty bodies or Cloudflare challenge markers as blocked.
- **`fetch: "skip"`**: log the source as skipped (with `skipReason`) and move on. Do not retry.
- **`fetch: "feed"` (preferred)**: `curl -sL --max-time 25 -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" "<feedUrl>"` and parse the RSS/Atom entries (title, link, summary, date). Consider only entries newer than the source's last productive run (when in doubt, the last ~30 days). Feeds bypass Cloudflare/HTML breakage — when a `webfetch`/`curl` source keeps failing, look for a feed and switch it over (see step 6).
- **`fetch: "webfetch"` or no field**: WebFetch against `fetchUrl` if present, else `url`. Extract recently-featured items (name, link, one-line description).
- **`fetch: "curl"`**: shell out to curl with the Safari UA above against `fetchUrl|url` and parse the HTML (or JSON — e.g. Shopify `products.json` endpoints). Treat HTTP 403 with a non-empty HTML body as a content response — only treat empty bodies or Cloudflare challenge markers as blocked.
- **`fetch: "skip"`**: log the source as skipped (with `skipReason`) and move on. Do not retry (re-probing skipped sources is a step-6 quarterly job, not a per-run one).
Parallelize WebFetch in batches of ~8 concurrent requests; finish all sources before proceeding.
Respect `cadence`: sources marked `weekly` / `monthly` / `as-needed` only need scanning when their cadence window has elapsed since the last productive scan — check the log; skip (and log as `cadence-skipped`) otherwise. This cuts run cost and rate-limit exposure.
For each source record: items extracted, or `empty` / `blocked` / `skipped`.
Parallelize fetches in batches of ~8 concurrent requests; finish all sources before proceeding.
For each source record: items extracted, or `empty` / `blocked` / `skipped` / `cadence-skipped`.
### 2. Filter
@@ -49,7 +54,7 @@ For each extracted item, drop:
### 3. Write find files
For every survivor, write one file at `src/content/find/<year>/<month>/<slug>/index.mdx` where `<year>` and `<month>` come from today's date (e.g. `src/content/find/2026/05/<slug>/index.mdx`). Hero image (when present) sits next to it as `src/content/find/<year>/<month>/<slug>/hero.<ext>`. The leaf folder name is the slug; URLs stay flat at `/find/<slug>/` because the content-collection loader uses a `leafSlug` id derivation.
For every survivor, write one file at `src/content/find/<year>/<month>/<slug>/index.mdx` where `<year>` and `<month>` come from today's date. Hero image (when present) sits next to it as `src/content/find/<year>/<month>/<slug>/hero.<ext>`. The leaf folder name is the slug; URLs stay flat at `/find/<slug>/` because the content-collection loader uses a `leafSlug` id derivation.
- Slug: kebab-case form of the name (lowercase, ASCII, no leading numbers).
- Frontmatter: `name`, `subtitle` (optional, one-liner), `date` (today), `link` (external URL — required), `linkText` (optional), `source` (the source name from sources.json — required, exact match), `topics` (copied from the source's `topics`), `tags` (free editorial tags inferred from the item — keep to 2-4), `description` (optional meta override).
@@ -63,60 +68,99 @@ If `link` for a Reddit-discovered item points at a v.redd.it / i.redd.it URL, fe
For every find, try to retrieve a single representative image from the linked page so the find has visual support if it later graduates to a review (or gets surfaced on a tile).
- **Where to look (in priority order):** `og:image` meta tag → JSON-LD `image` field → main product photo / hero image markup → first article image.
- **Where to save:** `src/content/find/<year>/<month>/<slug>/hero.<ext>` — colocated next to `index.mdx`. Preserve the original extension (`.jpg`, `.png`, `.webp`, `.gif`). One file per find — `hero` is the canonical name.
- **Where to save:** `src/content/find/<year>/<month>/<slug>/hero.<ext>` — colocated next to `index.mdx`. Preserve the original extension (`.jpg`, `.png`, `.webp`, `.gif`, `.avif`). One file per find — `hero` is the canonical name.
- **What to skip:** images with marketing/sales overlay text (Amazon-style "BEST PRESS / NO GRIT" tags), tiny thumbnails (<400px on the long edge), generic site logos, paywalled CDN URLs that 403 to curl. If the only available shots are marketing collages, skip rather than save a noisy image — a missing hero is fine, a bad one isn't.
- **For Mac/iOS apps:** prefer the App Store hero or developer-site screenshot. For physical products: prefer a clean studio shot or lifestyle photo without overlay copy. For travel destinations: prefer a Wikimedia Commons / Wikipedia image (free-to-use only). For text-only finds (quotes, dad jokes, essays): no media expected — skip.
- **Fetch strategy:** `curl -sL --max-time 25 -A "Mozilla/5.0 (...Safari...)" "<image-url>" -o <path>`. WebFetch is for HTML, not binaries.
- **Don't register heroes in `src/lib/hero.ts` at this stage.** That's a review-time decision (see `build-review`). The asset just needs to exist on disk so it's there if/when the find graduates.
Log image-fetch failures (404, 403, marketing-only) as a one-line note in the capture-run section of `daily-finds.log.md` so we can revisit broken media-source patterns later — same way we log fetch issues for source pages.
Log image-fetch failures (404, 403, marketing-only) as a one-line note in the capture-run section of `daily-finds.log.md` so we can revisit broken media-source patterns later.
### 4. Nominate new sources
### 4. Score and shortlist
Watch for new sources organically and via search. No cap on nominees per run — if a day's reading surfaces five plausible new sources, capture all five.
After all finds are written, rank the day's captures and mark the shortlist — the up-to-a-dozen finds the rest of the pipeline should act on.
- **From inline references in source posts.** When a post on a known source credits or links out to another publication as where they originally discovered the thing, capture that publication as a candidate source. Trigger phrases to watch for: *"via X"*, *"first seen on Y"*, *"spotted by Z"*, *"h/t X"*, *"originally posted on Y"*, *"thanks to X for the find"*, or any byline / sidebar link pointing to a different domain that itself looks like a curated feed. Cool Tools, Tools and Toys, Recomendo, Kottke, and Daring Fireball especially tend to credit their finds — read posts carefully enough to catch these credits.
- Score each of today's finds on: **delight** (would the audience light up), **specificity of virtue** (does the body name something concrete), **media** (clean hero present), and **variety** (the shortlist should not be twelve variations of one topic — spread across topics when quality allows).
- Set `shortlist: true` in the frontmatter of the top finds. Aim for 6–12 on a normal day; fewer on a thin day. **Never pad** — a 4-item shortlist beats a 12-item one with filler.
- Record the shortlist in the log, ranked, with a one-line rationale each. Note the top 1–3 as **review candidates** (strong media + strong story) — `/build-review` (or the pipeline) picks from these.
#### Amazon affiliate lookup
For every **shortlisted physical product** (tools, gifts, clothing, kitchen, EDC — not apps, not travel, not quotes/jokes/web pages):
- Check whether the exact product is sold on Amazon: WebSearch `site:amazon.com "<maker> <product name>"`, or curl the Amazon search page.
- Only when the hit is unambiguously the **same product** (same maker, same model — not a lookalike, not a marketplace knockoff), add `amazonLink: https://www.amazon.com/dp/<ASIN>` to the find's frontmatter. Use the canonical `/dp/<ASIN>` form with no query parameters — the site's affiliate layer (`src/lib/affiliate.ts`) appends the tracking ID, badge, and disclosure at render time.
- When in doubt, omit. A wrong affiliate link is worse than none. The site's focus is interesting items, not monetizable ones — this is a bonus, not a filter.
- Non-shortlisted physical finds may get the same treatment when the match is trivially confident; don't burn time on them otherwise.
### 5. Nominate new sources
Watch for new sources organically and via search. No cap on nominees per run.
- **From inline references in source posts.** When a post on a known source credits or links out to another publication as where they originally discovered the thing, capture that publication as a candidate source. Trigger phrases: *"via X"*, *"first seen on Y"*, *"spotted by Z"*, *"h/t X"*, *"originally posted on Y"*, or any byline / sidebar link pointing to a different domain that itself looks like a curated feed. Cool Tools, Recomendo, Kottke, and Daring Fireball especially tend to credit their finds.
- **From general observation.** When a publication is mentioned in a way that suggests it'd fit the brief (curated, taste-driven, focused on delightful or unusual items), nominate it.
- **From targeted search.** If the run produced few or no organic candidates, run one `WebSearch` for `"best <topic> newsletter"` or `"<topic> curator blog"`.
Append every nominee to `candidate-sources.md` under today's date heading. For each: name, URL, a one-line case for adding, and how it was surfaced — name the originating source post when applicable (e.g. *"via a Cool Tools post about the Spyderco Dragonfly"*), or *"WebSearch"* for search-derived candidates.
Append every nominee to `candidate-sources.md` under today's date heading. For each: name, URL, a one-line case for adding, and how it was surfaced.
Don't auto-add to `sources.json` — that's a user decision.
Adding new sources to `sources.json` remains a **user decision** — never auto-add.
### 5. Update the log
### 6. Source maintenance (mechanical edits — apply, don't recommend)
This skill has authority over mechanical `sources.json` hygiene. Past runs accumulated "overdue" recommendations nobody executed; that pattern is over. Apply these directly during the run:
- **Hard-blocked source** (Cloudflare challenge, empty 403s, dead DNS) for the 3rd consecutive run: before skipping, probe for a feed (`<link rel="alternate">` on the homepage, or common paths `/feed`, `/rss.xml`, `/atom.xml`, `/index.xml`). A working feed rescues the source — set `fetch: "feed"` + `feedUrl`. Otherwise set `fetch: "skip"` with a dated `skipReason`.
- **Dormant source** (live but no new content across 3+ runs): demote `cadence` to `as-needed`; if the newest content is over a year old, `fetch: "skip"` with reason "dormant since <date>".
- **Better fetch path discovered** (feed URL, Shopify `/collections/new/products.json`, article-level URL pattern that bypasses an index-level block): record it in `fetch`/`feedUrl`/`fetchUrl` immediately.
- **Quarterly re-probe**: if any `fetch: "skip"` entry has a `skipReason` dated 90+ days ago, re-test it this run; blocks get lifted. Remove the skip when the source is back.
- Update the top-level `updated` field when you touch the file.
Every mechanical change gets one line in the log's maintenance section. Only *new* sources need the user (step 5).
Note on Reddit: anonymous JSON/RSS endpoints have been blocked since ~June 2026. If `REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` are available in the environment (or `.env`), obtain an app-only OAuth token (`https://www.reddit.com/api/v1/access_token`, grant_type=client_credentials) and fetch `https://oauth.reddit.com/r/<sub>/top?t=week` with it. If no credentials exist, log the Reddit sources as blocked-pending-oauth and move on — don't retry anonymous endpoints every run.
### 7. Update the log
Append to `daily-finds.log.md`:
```
## YYYY-MM-DD — capture run
Sources scanned: <N> total — <hits> hits, <empty> empty, <blocked> blocked, <skipped> skipped.
Sources scanned: <N> total — <hits> hits, <empty> empty, <blocked> blocked, <skipped> skipped, <cadence-skipped> cadence-skipped.
Productive sources: <comma-separated list of sources that produced at least one written find>.
Captured finds:
- <slug> — <name> — source: <source>
- <slug> — <name> — source: <source>
- ...
Shortlist (<N>, ranked):
1. <slug> — <one-line rationale> [review candidate]
2. <slug> — <one-line rationale>
...
Amazon links added: <N> (<slug>, <slug>, ...) — or "none".
Source maintenance:
- <one line per sources.json change> — or "none".
New source nominees: <count> (see candidate-sources.md).
```
### 6. Report to the user
### 8. Report to the user
Short summary in chat:
Short digest in chat (or, when running as part of `/daily-pipeline`, feed this into the pipeline digest):
```
Captured <N> finds across <M> productive sources today.
Blocked: <K> sources. New source nominees: <count>.
Run /build-bundle to draft a bundle from the recent pool, or /build-review <slug> to graduate one to a review.
Captured <N> finds across <M> productive sources; shortlisted <K>.
Review candidates: <slug>, <slug>.
Amazon links: <N>. Source maintenance: <N> changes. New nominees: <count>.
Blocked: <list or none>.
```
That's it. No theme, no candidates list, no "pick a winner."
## Notes
- "Why it fits" / find descriptions should name a *specific* virtue. Bad: "great design". Good: "magnetic connector means it never strains the cable when a foot catches it".
- Travel destinations relax the daily-use criterion — a place qualifies if a visitor would remember it years later.
- If a source's homepage is too noisy, prefer adding a `fetchUrl` to `sources.json` rather than working around it inside the skill.
- If a source's homepage is too noisy, prefer adding a `feedUrl`/`fetchUrl` to `sources.json` rather than working around it inside the skill.
- Never invent slugs that collide with existing finds or reviews. If a name would produce a colliding slug, append a year or distinguishing word.
- Source attribution must be exact — the `source` field in each find file must match a `name` in `sources.json` so the `/sources` page cross-reference works.
- For large sweeps, fan out: scanner subagents per source cluster → central dedup/bar/shortlist pass → writer subagents for files + heroes. Keep the shortlist ranking in the main context — it's the editorial judgment call of the run.