Lenny's Peer Review

Built for Lenny's Buildathon — three weeks (May 6–27, 2026) where Lenny Rachitsky opened up his podcast and newsletter archive to builders, with the brief to do something useful or creative with the dataset. I'd been sitting on the feedback-loop problem for a while without a forcing function — the buildathon was the excuse to actually build it, and the archive turned out to be the exact corpus I'd want a reviewer trained on.

Getting senior feedback on a PRD or strategy memo is one of the hardest scarcities to solve as a PM — especially early in your career. The people whose pushback would actually sharpen the doc are the people with the least time to read it. Lenny's archive has 60+ hours of exactly those people — the best operators and thinkers in product — working out loud on the decisions PMs make every day. I wanted a way to get a directional first pass from them on a draft, so that when I do get time with the people closer to the work, we spend it on the texture and tradeoffs only they can see, not on the obvious gaps.

Scoping a buildathon project. The temptation in a buildathon is to build the platonic version — every guest, every doc type, a polished UI with saved reviews and a Chrome extension. I cut everything that wasn't load-bearing for the core moment: paste a Google Doc URL, watch comments get written directly into the doc, each one cited to a specific guest and episode. The thing I refused to cut was citation. Without the quote and the source link, the critique is just another LLM doing its thing — and the whole reason I was building this was to get away from that.

Building the critic index. Indexing is one Haiku pass per source, prompted to extract 3–5 principle-bearing moments — the passages where a guest is making a transferable point, not telling a personal story or doing a sponsor read. Each chunk comes back as a verbatim quote, a one-sentence principle, an applies_when line that tells the reviewer when to cite it, and a timestamp deep-link back to the original episode. The output is a single principles.json (~150 entries, ~100K tokens) that lives in the system prompt and gets cached on Anthropic's side. After the first call, every subsequent review reads from cache — a few cents per analysis instead of forty.

Voicing the critics. The reviewer isn't a chain — it's an agent with tools. Sonnet 4.6 gets the doc, gets the cached principles, and gets four tools: list_articles, read_article, write_comment, and finish. It decides on its own which guests to consult, what to cite, and when to stop. Two design choices made the difference between this feeling like a real reviewer and feeling like a chatbot. First, the tools are defensive: write_comment rejects any anchor that isn't a verbatim substring of the doc, and any evidence quote that isn't a verbatim substring of the cited principle. Hallucinations can't make it past the tool boundary. Second, every tool call streams to the dashboard in real time — "Consulting Shreyas Doshi on metrics… Writing comment on 'we expect activation to increase by 20%'…" — so the user watches the model think instead of watching a spinner. The latency stopped being a cost and started being the demo.

Specific over generic. The whole reason this exists is that generic LLM feedback isn't useful. The design choices that forced specificity: every critique is voiced by a named guest (not "an experienced PM"), every claim is anchored to a verbatim sentence from your doc (not a paraphrase), and every position is backed by a real quote from that guest's own archive (not the model making something up that sounds like them).

Cited or it didn't happen. Every comment includes the verbatim quote it draws from plus a deep link back to the original episode or newsletter — and the citation is enforced at the tool boundary, not asked for in the prompt. The write_comment tool rejects any evidence quote that isn't a verbatim substring of the source it claims to cite. The model can't write a citation it doesn't have. The cost is that some good critiques get dropped when the supporting quote isn't sharp enough — and that's the right tradeoff. Better to leave four well-cited comments than five with one that fabricates.

Inline over summary. Critique anchors to the specific sentence it's responding to — not a wall of feedback at the bottom of the doc, not a high-level summary that makes you map "be more specific" back onto whichever claim was vague. Inline pushback is harder to ignore because it's pointed at one thing, and easier to address because you don't have to guess what the reviewer meant. Where the problem lives is where the fix should land.

Meet PMs where they work. PMs do all their drafting and revising in Google Docs — that's where the doc lives, where teammates already leave comments, and where the next revision is going to happen. Most AI feedback tools force a context switch: paste into a separate UI, read the critique there, port the takeaways back into the actual doc. The round-trip is enough friction that the feedback gets skimmed and forgotten. Writing critique back as Google Docs comments — formatted the way a human reviewer's comments would be — slots the system into a workflow PMs already trust, instead of asking them to adopt a new one.

The first architecture was wrong. My initial build plan was a full RAG pipeline — embeddings, vector search, reranking — because "RAG" is the reflex when you have a body of text to search. A day in, I did the math: ~50 podcasts × 4 principle-bearing moments × ~500 tokens, plus 10 newsletters, came out to ~150K tokens. The entire usable corpus fit inside Claude Sonnet 4.6's 200K context window — there was nothing to retrieve from. Pivoting to long-context cut about 500 lines of infrastructure and dropped per-analysis cost by ~10x once the corpus cache warmed up. The lesson I keep relearning at buildathons: the right scoping question isn't "what should I build," it's "what does the actual data demand."

Hallucinated quotes. The two failure modes I had to design against: anchors that don't actually exist in the doc, and evidence quotes the model paraphrased instead of pulled. Asking the model nicely not to hallucinate (the v1 approach) worked most of the time, which is the same as failing badly some of the time. The fix moved the enforcement from the prompt into the tool: write_comment validates that the anchor is a verbatim substring of the doc, and that the evidence quote is a verbatim substring of the cited principle's source quote. Anything that doesn't pass returns an error the model has to address before it can try again. Hallucinations can't make it past the tool boundary.

The principle library was the bottleneck. The model only sounds as sharp as the index it's reading from. After the Haiku extraction pass, I had ~200 candidate principle chunks and the quality was uneven — some were genuinely transferable insights, others were just guests being articulate about their own company's situation. I spent about an hour reading through and cutting the weak ones down to ~150. That hour was the single highest-leverage piece of work in the build: critique quality is bottlenecked on the principle index, and there's no model trick that fixes a corpus full of mid-quality chunks. It's also the part of the system that doesn't get more impressive with a better model — it just needs a human who knows what good looks like.