Post

More from @omarsar0

elvis

@omarsar0

Jun 5

Knowledge or Reasoning?

Evaluation matters, and even more so when using reasoning LLMs.

Look at final response accuracy, but also pay attention to thinking trajectories.

Lots of good findings on this one.

Here are my notes:

Summary

Introduces a fine-grained evaluation framework to dissect LLM thinking into two components: knowledge correctness and reasoning informativeness, measured via Knowledge Index (KI) and Information Gain (InfoGain), respectively.

The authors apply this framework to evaluate how reasoning transfers across domains, particularly medical and mathematical, using Qwen2.5-7B and its DeepSeek-R1-distilled variant trained via SFT and RL.

Read 7 tweets

elvis

@omarsar0

Jun 2

Reasoning Models Thinking Slow and Fast at Test Time

Another super cool work on improving reasoning efficiency in LLMs.

They show that slow-then-fast reasoning outperforms other strategies.

Here are my notes:

What's the high level?

Introduces a universal framework, AlphaOne (α1), for modulating the reasoning progress of large reasoning models (LRMs) during inference.

Rather than relying on rigid or automatic schedules, α1 explicitly controls when and how models engage in “slow thinking” using a tunable parameter α.

Token magic

The method dynamically inserts “wait” tokens to encourage deeper reasoning & then deterministically ends slow thinking with a “</think>” token to prompt efficient answer generation.

This yields better accuracy & efficiency than previous test-time scaling methods.

Read 7 tweets

elvis

@omarsar0

May 31

Open-Ended Evolution of Self-Improving Agents

Can AI systems endlessly improve themselves?

This work shows the potential of self-improving AI, inspired by biological evolution and open-ended exploration.

This is a must-read!

Here are my notes:

What's the high level?

This work presents the Darwin Gödel Machine (DGM), a system that advances the vision of self-improving AI by combining self-referential code modification with open-ended evolutionary search...

Unlike the original Gödel machine, which requires provable benefits for code changes (a practically intractable constraint), the DGM adopts an empirical approach: it modifies its own codebase and evaluates improvements on coding benchmarks.

Read 10 tweets

elvis

@omarsar0

May 30

Building Production-Grade Conversational Agents with Workflow Graphs

Uses DAG to design robust and complex agentic systems.

If you're building AI agents, this is worth a read.

Here are my notes:

Quick overview

This paper presents a pragmatic, production-ready framework for building LLM-powered conversational agents using workflow graphs, with a specific focus on e-commerce scenarios.

Instead of relying solely on end-to-end generation, the authors design agents using a directed acyclic graph (DAG), enabling flexible yet controllable interactions that adhere to strict business rules and format constraints.

Multi-State DAG Framework

Each node in the graph corresponds to a conversational state with its own system prompt, tool access, and execution rules.

This structure enables robust constraint handling (e.g., avoiding hallucinated responses or non-compliant suggestions) by localizing logic and formatting within specific graph nodes.

Read 5 tweets

elvis

@omarsar0

May 29

An Operating System for Memory-Augmented Generation in LLMs

Lots of great ideas on how to think about memory and better manage it in LLM-based agents.

Must read!

Here are my notes:

It introduces a unified operating system for managing memory in LLMs, addressing a key limitation in current architectures: their lack of structured, persistent, and governable memory...

While today's LLMs rely primarily on parametric memory (model weights) and limited short-term context, MemOS proposes a comprehensive memory lifecycle and management infrastructure designed to support continual learning, behavioral consistency, and knowledge evolution.

Read 8 tweets

elvis

@omarsar0

May 28

New Lens on RAG Systems

RAG systems are more brittle than you think, even when provided sufficient context.

Great work from Google and collaborators.

Good tips for devs included.

Here are my notes:

What is the paper about?

It introduces a new empirical framework for analyzing RAG systems through the lens of sufficient context, whether the retrieved content alone enables answering a query.

This notion helps decouple retrieval failures from generation errors in LLMs.

New definition and classifier for sufficient context

The authors formalize “sufficient context” as context that plausibly allows answering a query, without requiring ground truth.

They develop a high-accuracy LLM-based autorater (Gemini 1.5 Pro, 93% accuracy) to label instances as having sufficient or insufficient context, enabling large-scale evaluation without needing ground-truth answers.

Read 7 tweets

Share this page!

Enter URL or ID to Unroll

elvis

Try unrolling a thread yourself!

More from @omarsar0

elvis

elvis

elvis

elvis

elvis

elvis

Did Thread Reader help you today?

Don't want to be a Premium member but still want to support us?

Send Email!