Clerk
Architecture
A five-layer system designed to survive infrastructure failure while delivering enterprise-grade research.
Presentation Layer
Next.js · FastAPI · WebSockets
- Real-time research progress dashboard
- REST API for research management
- Redis Pub/Sub + WebSocket live streaming
Orchestration Layer
Temporal.io · LangGraph
- Durable execution — tasks survive restarts for 1+ weeks
- Commander / Worker / Analyst / Synthesizer workflows
- State machine with checkpoints and error recovery
Agent Layer
o3 · DeepSeek-R1 · Claude 3.5 · GPT-4o-mini
- Commander decomposes topics into 100+ sub-niches
- Workers execute parallel research with schema extraction
- Synthesizer compiles the final S-Rank report
Data Layer
MinIO/S3 · PostgreSQL + pgvector · DuckDB
- Raw JSON research data lake
- Semantic memory and metadata
- In-process analytics engine for statistics
Infrastructure Layer
Kubernetes · Docker · Prometheus · Grafana
- Horizontal scaling to 100+ worker pods
- Containerized services
- Metrics, dashboards, and observability
Network Security Layer
- WireGuard / OpenVPN secondary shell tunneling
- Bright Data / Oxylabs residential proxy mesh
- Floating IP shell with auto-swapper (Linux IPRoute2)
- Rate-limit resilience with automatic retry