Research
The 4-Phase Research Pipeline: From Topic to S-Rank Report
March 29, 2026 · 6 min read
Decomposition, distributed extraction, statistics, and synthesis — how Clerk turns a single topic into a statistically-backed research paper.
Every Clerk research run follows the same four-phase pipeline, designed to combine breadth with rigor.
Phase I is Decomposition. The Commander agent takes a topic like "Web3" and breaks it into 100 specific sub-niches — each with its own ID, timeframe, and search parameters. The output is a Research Map: the blueprint for the entire investigation.
Phase II is Distributed Extraction. Up to 100 worker workflows fan out in parallel, one per sub-niche. Each worker runs on Temporal.io, so a rate limit, ban, or server restart never loses progress. Every result passes through a Pydantic schema — a reliability score between 0 and 1, an array of technical claims, and a sentiment index.
Phase III is Aggregation and Statistics. All results are loaded into DuckDB, where correlation matrices, trend vectors, and predictive odds are computed. Duplicates are removed across sub-niches so the final analysis reflects genuinely unique signal.
Phase IV is Synthesis. The Synthesizer agent — Claude 3.5 — reads the statistics and compiles the S-Rank report: executive summary, citations, visualizations, and predictive odds with confidence scores.
The result is research that computes rather than guesses — and the entire pipeline is auditable commit by commit.