Lex Fridman Podcast

#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

February 1, 2026

Key Takeaways Copied to clipboard!

  • The AI competition is characterized by fluid idea sharing, meaning technological breakthroughs are unlikely to remain proprietary to a single lab, making budget and hardware constraints the primary differentiating factors. 
  • The current landscape of leading consumer chatbots involves users frequently switching between models (like GPT-5, Gemini, and Claude Opus 4.5) based on the specific task, often sticking with the one that 'wins their heart' until it fails a critical query. 
  • Despite rapid advancement in AI capabilities, the fundamental autoregressive Transformer architecture derived from GPT-2 remains the state-of-the-art foundation for frontier models, with most innovation occurring in algorithmic stages (like post-training) and system optimizations (like quantization and hardware utilization). 
  • While the autoregressive transformer remains the state-of-the-art architecture, alternatives like Mamba and text diffusion models are emerging, and scaling laws are now considered across three axes: pre-training, reinforcement learning, and inference time. 
  • The primary driver for recent capability leaps, such as tool use, is the scaling of post-training via Reinforcement Learning with Verifiable Rewards (RLVR), which amplifies existing knowledge by enabling models to self-correct through iterative reasoning steps. 
  • The economics of AI development are shifting focus from the one-time cost of pre-training to the massive recurring costs of serving models, making inference-time scaling and model efficiency increasingly critical considerations for frontier labs. 
  • The research community faces a significant challenge in verifying LLM performance due to pervasive data contamination in benchmarks like math datasets and MMLU. 
  • Reinforcement Learning from Verifiable Rewards (RLVR) is seen as a more direct path to improving core model capabilities (like math) compared to RLHF, which primarily refines style and formatting. 
  • The high compute cost and memory-bound nature of RL training runs are approaching the scale of pre-training, contrasting with RLHF, which has diminishing returns on additional compute budget. 
  • Scaling context length is currently compute-bound, with expectations for increases up to 2-5 million tokens this year, but true breakthroughs beyond that require new architectural innovations. 
  • The path to AGI/ASI is debated, with some milestones like the 'superhuman coder' being concrete but potentially delayed due to the jagged nature of AI capabilities, especially in complex areas like distributed systems and tool use. 
  • The immediate economic impact of LLMs is currently felt through amplification and improved accessibility of knowledge (like a better search engine), rather than an immediate, massive GDP leap, which may require solving complex agentic tool use or specialized scientific breakthroughs. 
  • The trajectory of US open-source AI development, exemplified by initiatives like the Atom Project, is crucial for competing with China's rapidly advancing open-weight models and ensuring US leadership in AI research. 
  • The success and adoption of open-source models, despite initial backlash or internal corporate politics (like those potentially affecting OpenAI's strategy), are vital for fostering broader AI innovation, talent development, and preventing excessive centralization. 
  • Singular, highly focused leaders like Jensen Huang at NVIDIA have an outsized impact on technological revolutions (like the GPU acceleration of deep learning), potentially accelerating progress by decades, even if the underlying scientific breakthroughs might eventually occur through diffusion. 

Segments

Introduction and Sponsor Reads
Copied to clipboard!
(00:00:00)
  • Key Takeaway: The episode features machine learning researchers Nathan Lambert and Sebastian Raschka discussing the state of AI, covering technical breakthroughs and future predictions.
  • Summary: Lex Fridman introduces guests Nathan Lambert (Ai2, author of The RLHF Book) and Sebastian Raschka (author of Build a Large Language Model (From Scratch)). The introduction is followed by extensive sponsor acknowledgments for companies like Box, Quo, Uplift Desk, and Perplexity. Fridman notes he is running on fumes due to working insane hours.
China vs US AI Race
Copied to clipboard!
(00:16:29)
  • Key Takeaway: The AI competition is characterized by fluid idea sharing, meaning technological breakthroughs are unlikely to remain proprietary to a single lab, making budget and hardware constraints the primary differentiating factors.
  • Summary: The discussion begins by framing the current state around the ‘DeepSeek moment’ from January 2025. Sebastian Raschka argues that ideas flow freely due to researcher rotation, meaning budget and hardware will be the differentiator, not proprietary technology access. Nathan Lambert notes the intense hype surrounding Anthropic’s Claude Opus 4.5 model, contrasting it with the quieter impact of Google’s Gemini 3.
Model Performance and User Loyalty
Copied to clipboard!
(00:25:11)
  • Key Takeaway: Users often stick with a specific LLM until it breaks or fails a critical task, leading to a dynamic where models like ChatGPT maintain momentum despite competitors releasing superior performance in specific areas.
  • Summary: The guests analyze which consumer chatbot is ‘winning,’ noting that while Gemini showed momentum in 2025, OpenAI’s incumbent status is hard to bet against. Users often maintain multiple subscriptions, separating personal and work use cases for models. The preference for speed versus intelligence is highlighted, with users often defaulting to faster models for quick queries but reserving slower, more intelligent models for thorough checks.
Open Source Models and Chinese Influence
Copied to clipboard!
(00:43:02)
  • Key Takeaway: Chinese open-weight models are gaining traction due to unrestricted licenses and large MOE architectures, challenging US models which often have usage restrictions attached to their open releases.
  • Summary: The landscape of Open Language Models (OpenLMs) is dominated by Chinese players like DeepSeek, Kimi, and Minimax, who often release larger Mixture-of-Experts (MOE) models. Western open models (like Olmo, Gemma, and GPT-OSS) are often smaller but emphasize full transparency in training data and code. Chinese open models are popular because their licenses are often less restrictive than those from Meta or Google, appealing to users who want to run models locally or customize them without usage caps.
Transformer Architecture Evolution
Copied to clipboard!
(00:54:41)
  • Key Takeaway: The fundamental autoregressive Transformer architecture derived from GPT-2 remains the state-of-the-art foundation, with advancements primarily stemming from algorithmic tweaks (like MOE, GQA) and system optimizations (like FP8/FP4 training) rather than architectural overhauls.
  • Summary: The core architecture, based on the decoder-only Transformer from the ‘Attention Is All You Need’ paper, has seen few fundamental changes since GPT-2. Key advancements include the implementation of Mixture-of-Experts (MOE) layers to increase model size without proportional compute cost, and tweaks to the attention mechanism like Group Query Attention (GQA). The major gains in performance are attributed to faster experimentation cycles enabled by system-level optimizations like lower-precision training (FP8/FP4) and better data utilization.
Architecture Alternatives to Transformers
Copied to clipboard!
(01:01:17)
  • Key Takeaway: The autoregressive transformer architecture, derived from GPT-2, remains the state-of-the-art, despite the emergence of alternatives like text diffusion models and Mamba state-space models.
  • Summary: While optimizations like FP4 increase throughput for existing models, they do not introduce new capabilities. Text diffusion models and Mamba models represent completely different paradigms but have not yet replaced the autoregressive transformer for state-of-the-art performance. The current state-of-the-art architecture is fundamentally still derived from GPT-2.
Defining and Validating Scaling Laws
Copied to clipboard!
(01:02:45)
  • Key Takeaway: Scaling laws are power-law relationships between compute/data (x-axis) and prediction accuracy (y-axis), now recognized across three axes: pre-training, reinforcement learning, and inference time.
  • Summary: Traditional scaling laws focus on pre-training size and data set size, but OpenAI’s work introduced scaling for reinforcement learning and inference time compute. Inference time scaling is responsible for the recent step-function change in model abilities, enabling complex behaviors like tool use by allowing the model to generate hidden thoughts for extended periods. The low-hanging fruit has largely been taken in RL with verifiable rewards and inference time scaling.
Pre-training Scaling Economics and Viability
Copied to clipboard!
(01:06:33)
  • Key Takeaway: Pre-training scaling is becoming prohibitively expensive relative to serving costs, leading labs to prioritize efficiency and post-training techniques for immediate performance gains.
  • Summary: The cost of training large models (e.g., $1M to $5M for pre-training) is low compared to the billions in recurring compute costs for serving millions of users. While the fundamental scaling law for pre-training is unlikely to stop, current progress is often unlocked more attractively through inference scaling or RL techniques. Future large compute clusters, like those expected in 2026, will likely be utilized for slightly bigger models and higher subscription costs.
Training Phases: Pre, Mid, and Post
Copied to clipboard!
(01:18:45)
  • Key Takeaway: Pre-training focuses on knowledge acquisition, mid-training on specialized tasks like long context, and post-training (RLVR) on skill unlocking and refinement.
  • Summary: Pre-training involves next-token prediction on a vast corpus, increasingly refined with high-quality synthetic data to accelerate learning. Mid-training is a specialized phase focusing on specific data distributions, like long documents, to avoid catastrophic forgetting. Post-training, dominated by RLVR, focuses on skill learning—how to use the knowledge gained in pre-training—rather than teaching new knowledge.
RLVR Mechanics and Impact
Copied to clipboard!
(01:52:04)
  • Key Takeaway: RLVR, popularized by DeepSeek’s R1, leverages verifiable task accuracy (like math or code) as a reward signal, causing rapid accuracy improvements by amplifying the model’s inherent ability to self-correct via step-by-step reasoning.
  • Summary: RLVR shifts optimization away from learned human preference models (RLHF) to direct, verifiable rewards, enabling scaling in domains like math and code. The model generates step-by-step explanations during inference, which acts as a self-correction mechanism, leading to massive accuracy gains even in a few steps. This process unlocks knowledge already present in the pre-training data rather than teaching new fundamental knowledge.
LLM Data Contamination Debate
Copied to clipboard!
(01:59:36)
  • Key Takeaway: Data contamination, especially in math benchmarks, makes it difficult to trust reported LLM performance gains without new, unseen test sets.
  • Summary: Base models like Qwen can achieve high accuracy on word problems by recognizing near-identical test set examples, suggesting reliance on memorization rather than true reasoning without tools. This contamination issue casts doubt on the validity of results reported in many reinforcement learning papers. The only fair evaluation method requires using benchmarks created after the LLM’s data cutoff date.
Post-Training Recipe Explained
Copied to clipboard!
(02:01:55)
  • Key Takeaway: Effective LLM post-training involves careful data curation during mid-training, followed by RLVR on verifiable rewards for skill acquisition, and concluding with RLHF for usability polish.
  • Summary: Mid-training requires providing reasoning traces—intermediate steps in problem-solving—to enable subsequent learning via verifiable rewards. RLVR focuses on trial-and-error learning across increasingly difficult problems, as signal diminishes when models solve problems perfectly. RLHF serves as a finishing touch, optimizing organization, style, and tone to make the model more useful to the end-user.
RL Compute vs. RLHF Budget
Copied to clipboard!
(02:05:34)
  • Key Takeaway: RLVR scales effectively with compute, following scaling laws, whereas RLHF reaches a saturation point where further compute yields little benefit.
  • Summary: RL training is memory-bound due to long sequence generation and quadratic memory increase from attention mechanisms, unlike compute-bound pre-training. RLVR benefits from continuous scaling, evidenced by linear performance increases with logarithmic compute increases, unlike RLHF, which is limited by preference tuning saturation. Academic research is often constrained by the high cost of scaling RLVR experiments.
Learning AI Fundamentals
Copied to clipboard!
(02:12:43)
  • Key Takeaway: Newcomers should implement a simple LLM from scratch to understand fundamentals, then reverse-engineer complex production code like Hugging Face Transformers for deeper insight.
  • Summary: Building a small model from scratch provides a solid understanding of pre-training, SFT, and attention mechanisms without the complexity of large-scale infrastructure. The Hugging Face Transformers library, while canonical for loading models, is too complex for initial learning due to its need to support many architectures. Verifying one’s custom implementation against reference outputs acts as a crucial unit test for architectural correctness.
Career Paths and Research Taste
Copied to clipboard!
(02:17:28)
  • Key Takeaway: After mastering fundamentals, researchers should specialize in narrow areas where they can contribute novel insights, as the field moves too fast for generalists.
  • Summary: Impact with minimal compute can be achieved by focusing on evaluation, creating representative problems where frontier models fail, which can lead to career momentum if picked up by leading labs. Academic research faces high opportunity costs due to low PhD stipends compared to lucrative industry positions at frontier labs. Developing ‘school taste’ involves knowing which struggles are productive for long-term career growth versus which are merely homework hurdles.
Work Culture and Silicon Valley Bubble
Copied to clipboard!
(02:35:20)
  • Key Takeaway: The intense competitive dynamic among frontier AI labs fosters a ‘996’ (9 a.m. to 9 p.m., six days a week) work culture driven by fervor and the belief in imminent transformation.
  • Summary: The competitive leapfrogging between companies like Anthropic and OpenAI drives extreme dedication, often at the cost of employee well-being, mirroring historical examples of intense industrial effort. Silicon Valley functions as a productive, yet potentially dangerous, echo chamber or ‘reality distortion field’ that can lead to missing broader human experiences. Young professionals entering this environment must consciously seek external context by engaging with history and literature outside the tech sphere.
Text Diffusion Models Exploration
Copied to clipboard!
(02:43:19)
  • Key Takeaway: Text diffusion models offer a parallel generation alternative to autoregressive LLMs, potentially offering faster, cheaper output for tasks like generating long code diffs.
  • Summary: Diffusion models, known from image generation, iteratively refine text by filling in masked or noisy tokens, contrasting with the sequential, one-token-at-a-time nature of GPT-style models. While they promise efficiency, they currently struggle with sequential tasks requiring external tool use or reasoning steps. They are likely to find niches for quick, high-volume tasks rather than replacing general-purpose autoregressive LLMs.
Tool Use and Recursive LLMs
Copied to clipboard!
(02:49:01)
  • Key Takeaway: Tool use significantly reduces hallucination by outsourcing factual tasks to external systems, but the LLM must still learn when and how to invoke the correct tool.
  • Summary: Recursive language models break complex tasks into sub-tasks, recursively calling the LLM or tools for each step, which is powerful for large QA tasks involving web searches. Open models must be designed to interface with multiple tools flexibly, whereas closed models deeply integrate specific proprietary tools. Granting LLMs tool access, especially for sensitive tasks like email management, requires significant trust due to potential security risks.
Continual Learning vs. Context Learning
Copied to clipboard!
(02:53:17)
  • Key Takeaway: The immediate solution for adapting models to new information is In-Context Learning via large context windows, as weight updates (Continual Learning) are too expensive for personalized, rapid adaptation.
  • Summary: Continual learning involves updating model weights rapidly to mimic human on-the-job learning, but this is prohibitively expensive at scale for individual users. In-context learning leverages massive context windows to provide necessary information dynamically, giving the appearance of fast learning without weight modification. Personalized memory solutions like LoRA adapters offer a middle ground but are subject to the ’no free lunch’ theorem regarding learning capacity versus forgetting.
Continual Learning Trade-offs
Copied to clipboard!
(02:58:16)
  • Key Takeaway: Continual learning involves a ’no free lunch’ trade-off where learning more requires more weights, leading to increased expense and potential forgetting.
  • Summary: LoRA research suggests that learning more requires using more weights, which increases computational cost. Simultaneously, increased learning can lead to increased forgetting, forcing models to find a ‘Goldilocks zone’ balance. This principle applies across different learning strategies.
Innovations in Context Length
Copied to clipboard!
(02:58:37)
  • Key Takeaway: Scaling context length beyond current limits (like 1 million tokens) requires architectural breakthroughs, as current methods like hybrid attention models remain expensive due to data scarcity for very long sequences.
  • Summary: Context length improvement is seen as a compute and data problem, sometimes involving attention variants like hybrid state space models that reduce compute for distant tokens. The world lacks sufficient training sequences of 100,000+ tokens to easily scale context to extreme lengths like 100 million tokens. Agentic models may learn to manage context compaction as an action to optimize evaluation scores against minimal history length.
World Models and Simulation
Copied to clipboard!
(03:05:30)
  • Key Takeaway: World models, which simulate the world to verify intermediate steps beyond simple next-token prediction, are expected to add value to LLMs, potentially by modeling code environments more rigorously.
  • Summary: World models involve the AI running a simulation of the world to unlock capabilities the LLM is not explicitly aware of. Meta’s Coda research applied this to LLMs by checking the correctness of intermediate variables, not just the final output. This sophistication, while expensive, mirrors successes in fields like protein folding (AlphaFold) where physical interactions were explicitly modeled.
Robotics Challenges and Investment
Copied to clipboard!
(03:08:32)
  • Key Takeaway: Robotics development is being supercharged by LLM infrastructure and investment, but deployment in consumer homes faces severe safety hurdles and the difficulty of on-the-fly customization for unique real-world environments.
  • Summary: Locomotion in robotics is more solved than manipulation, but end-to-end learning for complex tasks like human-hand manipulation is extremely difficult. The ecosystem is improving due to better transformer tooling and increased compute, but safety constraints mean robots are almost never allowed to fail in consumer settings. Industrial automation designed specifically for robots (like in Amazon centers) is a more reasonable near-term path than general in-home robots.
Defining and Timing AGI Milestones
Copied to clipboard!
(03:14:04)
  • Key Takeaway: A common grounding point for AGI is an AI capable of reproducing most digital economic work (the ‘remote worker’), but concrete milestones like the ‘superhuman coder’ are now predicted to occur around 2031, followed quickly by AI researchers.
  • Summary: Definitions for AGI vary, but the AI2027 report focused on concrete milestones like the superhuman coder, which they pushed back to a mean prediction of 2031. One speaker believes AI capabilities will remain ‘jagged,’ excelling at traditional ML tasks but struggling with complex distributed systems, leading to a continued dance between human enablement and AI assistance rather than full automation of research.
Software Automation and Interface Difficulty
Copied to clipboard!
(03:20:17)
  • Key Takeaway: Full automation of programming is approaching for common tasks like website creation, but the primary bottleneck for advanced agentic systems is the difficulty of human specification and the complexity of integrating into existing, large-scale production codebases.
  • Summary: The industrialization of software suggests that humans will shift roles toward system design and goal setting as AI handles implementation details, potentially leading to agents implementing features in complex systems like Slack within low single-digit years. The challenge lies in specification; asking an AI to book a complex trip requires guiding it through steps, which is much harder than clarifying a text-based request. Labs like Anthropic are experimenting with rapid model weight updates based on real-world feedback, approaching continuous learning.
Economic Impact and Monetization
Copied to clipboard!
(03:39:06)
  • Key Takeaway: The immediate impact of current LLMs is best described as a ’nice helper’ or ‘Clippy on steroids,’ providing amplification across many domains, while the dream of a single, general model handling the entire digital life remains challenged by tool use difficulty and interface complexity.
  • Summary: The true, long-term impact of LLMs may be the quiet permeation of human knowledge accessibility worldwide, enabling innovation across fields like science and space travel. However, the current reality is that models are being used as powerful assistants for tasks like coding and personalized planning, rather than achieving full autonomy in complex areas like computer use. Companies are currently hesitant to introduce ads into chat interfaces due to reputation risk, despite the massive potential revenue flywheel.
Business Consolidation and Future Landscape
Copied to clipboard!
(03:51:02)
  • Key Takeaway: The AI sector is seeing consolidation through high-value licensing deals (like Grok/NVIDIA) rather than traditional acquisitions, which negatively impacts the broader startup ecosystem by bypassing equity payouts for rank-and-file employees.
  • Summary: Consolidation pressure is mounting, evidenced by multi-billion dollar startup valuations, but many deals are structured as licensing agreements to avoid antitrust scrutiny and benefit fewer top individuals. Frontier model providers like OpenAI and Anthropic may not be winner-take-all; if LLM services commoditize, these companies will likely pivot to specialized niches, similar to how Anthropic focused on code after initial generalist efforts. Public markets are being avoided by major US startups, contrasting with Chinese firms like MiniMax and Z.ai which are filing IPO paperwork.
Open Source Model Release Failures
Copied to clipboard!
(03:59:47)
  • Key Takeaway: Over-excitement about frontier benchmarks led to a failure to release smaller, usable open models, causing internal organizational collapse due to misaligned incentives.
  • Summary: The goal of having usable, modifiable, and understandable models was lost due to an over-focus on benchmark performance, leading to overfitting on metrics. This internal political fighting and misaligned incentives caused the effort to crash out, neglecting the release of smaller, accessible models. The focus on headline-grabbing frontier performance overshadowed practical utility.
Meta’s Open Source Stance
Copied to clipboard!
(04:00:58)
  • Key Takeaway: Mark Zuckerberg’s consistent advocacy for open source is a critical factor that could lead to superior open models like a hypothetical Llama 5, despite internal debates with leaders more skeptical of open releases.
  • Summary: Mark Zuckerberg’s commitment to open source is credited as a major positive force, potentially leading to a future ‘GPT-OSS’ providing an excellent library of open models. There is an internal debate, notably with Alexander Wong, regarding the openness of models, which could influence future releases. Negative backlash from the open-source community toward initial releases may have caused companies to re-evaluate their commitment to openness.
US Open Model Initiative (Atom Project)
Copied to clipboard!
(04:03:57)
  • Key Takeaway: The Atom Project (American Truly Open Models) is a US-based initiative designed to build and host high-quality open-weight models to counter the growing influence of Chinese open-source AI ecosystems.
  • Summary: The project aims to ensure the US remains the home of leading AI research by building models competitive with Chinese offerings like DeepSeek, which are gaining traction in US enterprises. Building models competitive with the cutting edge costs around $100 million, necessitating a centralizing force for investment. Support for this initiative exists across policy circles, including the administration, and organizations like AI2 have received major NSF grants to pursue this goal.
Future of Open Source Dominance
Copied to clipboard!
(04:12:18)
  • Key Takeaway: If AI progress saturates within a few years, optimized and cheaper open models will likely win out over closed offerings due to the massive community effort dedicated to optimizing serving common architectures.
  • Summary: The trajectory of progress determines if open models will dominate; if saturation occurs soon, optimization efforts will make open models significantly cheaper to run. This optimization effort, driven by many more contributors than closed labs, will establish common architectures as standards. Chinese companies releasing powerful open models are seen as inadvertently pushing US labs to release better open models sooner.
NVIDIA’s Moat and Singular Leaders
Copied to clipboard!
(04:14:43)
  • Key Takeaway: NVIDIA’s sustained dominance relies less on the GPU chip itself and more on the deeply entrenched CUDA ecosystem, though the pace of AI progress dictates how quickly competitors can develop bespoke hardware.
  • Summary: NVIDIA’s competitive advantage is rooted in the two-decade evolution of the CUDA ecosystem, making adoption of risky, new hardware difficult for large-scale clients. Singular figures, like Jensen Huang, are crucial for placing focused, high-stakes bets that accelerate technological revolutions, potentially preventing decades-long delays in AI progress. The initial success of deep learning relied on the lucky coincidence that GPUs, developed for gaming, were perfectly suited for the linear algebra required by neural networks.
Historical Breakthroughs and Future Interface
Copied to clipboard!
(04:22:46)
  • Key Takeaway: Future historians will likely emphasize generalized computing power and connectivity as the fundamental breakthroughs leading to the singularity, rather than specific algorithms like the Transformer.
  • Summary: The core breakthrough leading to future states will likely be broadly defined as increased compute (Moore’s Law) and the connectivity provided by the internet, which enables distributed intelligence. Deep learning and the concept of the Transformer may be remembered as specific, important algorithms, analogous to the steam engine during the Industrial Revolution. The future interface will likely move beyond smartphones toward brain-computer interfaces, though a private, physical device may persist for personal data.
Societal Impact and Human Agency
Copied to clipboard!
(04:30:03)
  • Key Takeaway: Despite potential mass job loss and the proliferation of AI-generated ‘slop,’ human agency, community, and the premium placed on in-person, physical experiences will persist and potentially increase in value.
  • Summary: The suffering caused by individual job losses due to automation must not be forgotten, necessitating robust social support systems like UBI to manage the transition. The overwhelming volume of AI-generated content (‘slop’) is expected to increase the value and appreciation for authentic, physical human experiences, art, and in-person interaction. Humans retain agency because current AI requires explicit direction, meaning the user remains in charge, making humans worth fighting for even in hypothetical conflicts.