Week 15: Autonomous AI Systems and Reasoning Advances

The AI Scientist-v2 autonomously generates workshop-accepted research manuscripts, removing human-authored code dependencies and integrating agentic tree-search and vision-language models.

• Eliminates reliance on human-crafted code templates for out-of-the-box deployment
• Agentic tree search refines hypotheses via branching exploration
• One manuscript accepted at an ICLR workshop, showcasing end-to-end AI-driven discovery

BrowseComp introduces a challenging benchmark with 1,266 questions requiring persistent web searches, testing AI agents' ability to locate entangled information.

• Only 29.2% of tasks solved by humans; Deep Research achieves 51.5% accuracy
• Reasoning outperforms browsing; OpenAI o1 beats GPT-4.5 with browsing
• Test-time scaling with 64 parallel samples boosts performance by 15–25%

OLMOTrace traces LLM-generated text to its verbatim sources in multi-trillion-token corpora, enabling fact-checking and creativity audits with sub-5-second latency.

• Uses infini-gram with suffix arrays for efficient text search
• Supports OLMo models with 4.6T-token datasets
• Average relevance score of 1.82/3 for top retrieved documents

This work uses a two-phase RL strategy to promote concise and accurate reasoning in LLMs, reducing token usage by over 50% without accuracy loss.

• Improves MMLU-STEM accuracy by 12.5% while halving response length
• Effective with just 4–8 training examples
• Robust at low sampling temperatures, outperforming baselines by 10–30%

This paper shows reflection emerges during pre-training, with adversarial datasets revealing self-correction capabilities as compute scales.

• Reflection rates on GSM8K-Platinum grow from 10% to over 60%
• Simple triggers like “Wait” induce reflection
• More pre-training compute reduces need for test-time reasoning

LightPROF enables small LLMs to reason over knowledge graphs with a retrieve-embed-reason pipeline, outperforming larger models like ChatGPT.

• Achieves 83.8% on WebQSP and 59.3% on CWQ
• Reduces token input by 98% and runtime by 30%
• Plug-and-play with no LLM fine-tuning

Compute Agent Arena benchmarks LLM and VLM agents on real-world computer tasks like coding and web navigation in a virtual desktop environment.

• OpenAI and Anthropic lead with modest success rates
• Platform supports crowdsourced tasks and open-source infrastructure
• Focuses on practical, real-world agent performance

KnowSelf enables LLM agents to dynamically reflect or seek knowledge using special tokens, achieving SOTA performance on ALFWorld and WebShop.

• Mimics human cognition with fast, slow, and knowledgeable thinking
• Reduces inference costs with minimal external knowledge
• Outperforms baselines in task-oriented environments

This work introduces TTT layers for single-shot one-minute video generation from storyboards, achieving 34 Elo points over baselines.

• Integrates TTT layers into pre-trained diffusion models
• Enables multi-scene video generation with self-supervised test-time updates
• Outperforms Mamba 2 and DeltaNet in human evaluations

NoProp introduces gradient-free learning where layers denoise targets independently, achieving competitive performance on MNIST and CIFAR.

• Avoids hierarchical representation learning
• Inspired by diffusion and flow matching
• Matches backpropagation efficiency on image classification