InterviewsVector
Original Academy

Legacy mirror · noindex · upstream phase 15

Autonomous Systems

Long-horizon agents, self-improvement, and the 2026 safety stack.

Provenance: this phase outline and its lesson readings are preserved from ai-engineering-from-scratch by Rohit Ghumare under the MIT License. InterviewsVector does not claim authorship. These archive pages remain available for old links and progress, but are excluded from indexing.

Attributed readings

  1. 01From Chatbots to Long-Horizon Agents (METR)In 2023 a chatbot answered a question in one turn. In 2026 a frontier model routinely runs minutes to hours on a single task. METR's Time Horizon 1.1 benchmark (January 2026) pu…
  2. 02STaR, V-STaR, Quiet-STaR: Self-Taught ReasoningThe smallest possible self-improvement loop sits inside the rationale. A model generates a chain of thought, keeps the ones that land on correct answers, and fine-tunes on those…
  3. 03AlphaEvolve: Evolutionary Coding AgentsPair a frontier coding model with an evolutionary loop and a machine-checkable evaluator. Let the loop run long enough. It discovers a 4x4 complex-matrix multiplication procedur…
  4. 04Darwin Gödel Machine: Self-Modifying AgentsSchmidhuber's 2003 Godel Machine required a formal proof that any self-modification was beneficial before accepting it. That proof is impossible in practice. Darwin Godel Machin…
  5. 05AI Scientist v2: Workshop-Level ResearchSakana's AI Scientist v2 (Yamada et al., arXiv:2504.08066) runs the full research loop: hypothesis, code, experiments, figures, writeup, submission. It is the first system to ha…
  6. 06Automated Alignment Research (Anthropic AAR)Anthropic ran parallel teams of Claude Opus 4.6 Autonomous Alignment Researchers in independent sandboxes, coordinating via a shared forum whose logs live outside any sandbox (s…
  7. 07Recursive Self-Improvement: Capability vs AlignmentRecursive self-improvement (RSI) is no longer speculation. The ICLR 2026 RSI Workshop in Rio (April 23-27) framed it as an engineering problem with concrete tooling. Demis Hassa…
  8. 08Bounded Self-Improvement DesignsResearch has converged on four primitives for bounding a self-improvement loop. Formal invariants that must hold across every edit. Alignment anchors that cannot be modified. Mu…
  9. 09Autonomous Coding Agent Landscape (SWE-bench, CodeAct)SWE-bench Verified went from 4% to 80.9% in under three years. Same Claude Sonnet 4.5 scored 43.2% on SWE-agent v1 and 59.8% on Cline autonomous — the scaffolding around the mod…
  10. 10Claude Code Permission Modes and Auto ModeClaude Code exposes seven permission modes. "plan" asks before every action, "default" asks only for risky ones, "acceptEdits" auto-approves file writes but still confirms shell…
  11. 11Browser Agents and Indirect Prompt InjectionChatGPT agent (July 2025) merged Operator and deep research into one browser/terminal agent and set BrowseComp SOTA at 68.9%. OpenAI shut Operator down August 31, 2025 — consoli…
  12. 12Durable Execution for Long-Running AgentsProduction long-horizon agents do not run in `while True`. Every LLM call becomes an activity with checkpoint, retry, and replay. Temporal's OpenAI Agents SDK integration went G…
  13. 13Action Budgets, Iteration Caps, Cost GovernorsA mid-sized e-commerce agent's monthly LLM cost jumped from $1,200 to $4,800 after its team enabled the "order-tracking" skill. That is not a pricing bug. That is an agent that …
  14. 14Kill Switches, Circuit Breakers, Canary TokensA kill switch is a boolean held outside the agent's edit surface — a Redis key, a feature flag, a signed config — that disables the agent entirely. A circuit breaker is finer-gr…
  15. 15HITL: Propose-Then-CommitThe 2026 consensus on HITL is specific. It is not "the agent asks, the user clicks Approve." It is propose-then-commit: the proposed action is persisted to a durable store with …
  16. 16Checkpoints and RollbackEvery graph-state transition persists. When a worker crashes, its lease expires and another worker picks up at the latest checkpoint. Cloudflare Durable Objects hold state acros…
  17. 17Constitutional AI and Rule OverridesAnthropic's January 22, 2026 Claude Constitution runs 79 pages and is CC0. It moves from rule-based to reason-based alignment and establishes a four-tier priority hierarchy: (1)…
  18. 18Llama Guard and Input/Output ClassificationLlama Guard 3 (Meta, Llama-3.1-8B base, fine-tuned for content safety) classifies both LLM inputs and outputs against an MLCommons 13-hazard taxonomy across 8 languages. A 1B-IN…
  19. 19Anthropic Responsible Scaling Policy v3.0RSP v3.0 went into effect February 24, 2026, replacing the 2023 policy. Two-tier mitigation: what Anthropic will do unilaterally vs what is framed as an industry-wide recommenda…
  20. 20OpenAI Preparedness Framework and DeepMind FSFOpenAI Preparedness Framework v2 (April 2025) introduces Research Categories — Long-range Autonomy, Sandbagging, Autonomous Replication and Adaptation, Undermining Safeguards — …
  21. 21METR Time Horizons and External EvaluationMETR (ex-ARC Evals) is an independent 501(c)(3) since December 2023. Their Time Horizon 1.1 benchmark (January 2026) fits a logistic curve to task-success probability vs log(exp…
  22. 22CAIS, CAISI, and Societal-Scale RiskThe Center for AI Safety (CAIS, San Francisco, founded 2022 by Hendrycks and Zhang) publishes the four-risk framework — malicious use, AI races, organizational risks, rogue AIs …