Lenny's Podcast: Product | Career | Growth

How to measure AI developer productivity in 2025 | Nicole Forsgren

October 19, 2025

Key Takeaways Copied to clipboard!

  • Most traditional productivity metrics, like lines of code, are easily gamed by AI tools and are insufficient for measuring true engineering value in the AI era. 
  • Developer Experience (DevEx) is crucial because poor DevEx (high friction, high cognitive load) negates the potential speed gains from AI, emphasizing the need to focus on flow state, cognitive load, and feedback loops. 
  • When measuring the impact of AI tools, focus on metrics that align with leadership's primary concerns (e.g., speed/velocity, cost/margin) rather than just developer-centric metrics like raw output, and be prepared to attribute gains jointly with DevEx improvements. 
  • Satisfaction surveys are more useful than happiness surveys for measuring developer experience because satisfaction can be tied directly to specific tools and work aspects, unlike the all-encompassing nature of happiness. 
  • Popular AI tools currently seeing success among developers include GitHub Copilot, Cursor, Gemini, and Claude Code, with Claude Code noted as being particularly underrated for non-coding use cases. 
  • DevEx improvements should adopt a product mindset, involving identifying user problems, running MVP experiments, establishing clear strategies, and continuously iterating or sunsetting metrics/tools as needed, especially given the rapid changes brought by AI. 

Segments

Productivity Metrics Are Lies
Copied to clipboard!
(00:00:00)
  • Key Takeaway: Metrics like lines of code are easily gamed by AI prompting, leading to inflated, meaningless output.
  • Summary: Most productivity metrics are inherently flawed, especially when AI is involved, as systems can be easily gamed to maximize output proxies. If the goal is more lines of code, an LLM can generate excessive verbosity, introducing technical debt. This necessitates a shift away from simple output counts toward evaluating code quality and downstream impact.
AI’s Impact on Flow State
Copied to clipboard!
(00:08:38)
  • Key Takeaway: AI interaction interrupts traditional deep work flow, but advanced users are creating parallel agent workflows to maintain flow at a higher, goal-oriented level.
  • Summary: The nature of AI coding—prompting, reviewing, integrating—can interrupt the deep flow state previously achieved through sustained coding. Highly skilled engineers are setting up workflows where AI agents work in parallel on sub-tasks after initial planning, allowing the human to focus on high-level architecture and review, thus keeping them in a productive flow state.
DevEx and Business Value
Copied to clipboard!
(00:21:32)
  • Key Takeaway: Improving DevEx is critical because it directly enables the rapid experimentation and software creation necessary to achieve business value like market share and customer retention.
  • Summary: Developer Experience enables all software creation, which in turn drives business needs such as market share and customer acquisition. Good DevEx allows for extremely rapid experimentation, enabling teams to move from idea to customer feedback in hours instead of months. The best immediate action for PMs is to start by listening to developers about their friction points, rather than immediately jumping to tool automation.
Signs of Slow Engineering Teams
Copied to clipboard!
(00:26:53)
  • Key Takeaway: Teams moving slower than their potential often exhibit systemic friction indicated by constantly breaking builds, flaky tests, and high switching costs between projects.
  • Summary: Most engineering teams can move faster than they currently do, but speed must be balanced with strategy to avoid shipping ’trash’ quickly. Key indicators of underlying friction include unreliable systems (breaking builds, flaky tests) and high cognitive load associated with context switching between tasks or projects. Strategy and smart decision-making on what to ship are necessary complements to speed.
Frictionless Book Framework
Copied to clipboard!
(00:36:39)
  • Key Takeaway: The Frictionless seven-step process guides organizations from initial listening tours and quick wins to data-driven strategy, selling the plan, and scaling change.
  • Summary: The book, co-authored with Abhi Noda, provides a structured approach for improving DevEx, starting with Step 1: beginning the journey through listening tours and workflow visualization. Subsequent steps involve securing quick wins, using data (like surveys) to optimize work, prioritizing strategy, selling that strategy, driving change at scale, and finally, evaluating progress to loop back. The framework emphasizes treating DevEx as a product, requiring PM skills for success.
Measuring DevEx Impact
Copied to clipboard!
(00:45:46)
  • Key Takeaway: Measuring DevEx impact requires tailoring metrics to the audience: developers care about time saved and reduced toil, while leadership focuses on revenue acceleration, time-to-value, and cost savings.
  • Summary: To communicate value, DevEx teams must speak the language of their audience; developers value reduced toil and focus time, whereas leadership prioritizes business outcomes like revenue speed and cost reduction. When measuring AI impact, if leadership emphasizes speed, focus on time from idea to experiment; if they emphasize margin, quantify savings from reduced cloud spend or vendor costs. Starting measurement should involve subjective surveys to quickly baseline satisfaction and identify the top three recurring barriers.
Happiness vs Satisfaction Surveys
Copied to clipboard!
(00:56:43)
  • Key Takeaway: Satisfaction surveys are preferred over happiness surveys for developer experience measurement due to their specificity.
  • Summary: Happiness surveys are avoided because happiness is influenced by too many external factors like family and hobbies. Satisfaction surveys allow for targeted questioning about specific tools or work aspects. Greater satisfaction contributes to happiness, but directly influencing happiness is too challenging and all-encompassing for utility.
Popular AI Tools Endorsements
Copied to clipboard!
(00:57:59)
  • Key Takeaway: Claude Code is highlighted as an underrated AI tool capable of powerful non-engineering tasks.
  • Summary: Commonly successful tools mentioned include Copilot, Cursor, and Gemini. Nicole Forsgren specifically praises Claude Code for its utility beyond coding, citing its ability to perform local system tasks like cleaning up storage. Dan Shipper previously noted Claude Code’s underrated capabilities in a prior podcast appearance.
Product Mindset for DevEx
Copied to clipboard!
(00:59:22)
  • Key Takeaway: DevEx improvements require applying a product mindset, including defining users, strategy, and sunsetting obsolete metrics.
  • Summary: Applying a product mindset means identifying a problem for a set of users and iterating rapidly through experiments. This approach requires defining an addressable market, knowing what success looks like, and establishing a go-to-market function for DevEx initiatives. It is crucial now to reassess legacy metrics to ensure they still drive necessary actions in the rapidly changing AI landscape.
AI Corner: Home Design Visualization
Copied to clipboard!
(01:00:40)
  • Key Takeaway: ChatGPT and Gemini excel at visualizing home design changes using floor plans and existing photos.
  • Summary: Nicole uses ChatGPT and Gemini to render pictures for home redecorating projects by providing floor plans and existing room photos. This allows her to quickly visualize changes to walls or furniture layouts, helping her determine what she likes. The AI sometimes incorporates elements based on inferred knowledge, such as suggesting a dog bed based on the user owning dogs.
Lightning Round Recommendations
Copied to clipboard!
(01:02:36)
  • Key Takeaway: Recommended books include ‘Outlive,’ ‘Back Mechanic,’ ‘How Big Things Get Done,’ and ‘The Undoing Project.’
  • Summary: For health, ‘Outlive’ by Peter Atia and ‘Back Mechanic’ by Stuart McGill (for lower back issues) are recommended. ‘How Big Things Get Done’ offers insights into large project failures, relevant for the current AI transformation. ‘The Undoing Project’ by Michael Lewis is also highly recommended for its impact.
Product Discoveries and Life Motto
Copied to clipboard!
(01:04:14)
  • Key Takeaway: The Ninja Creami and Jura coffee maker are highly recommended products for convenient indulgence.
  • Summary: The Ninja Creami is praised for turning frozen protein shakes into ice cream, offering a healthy treat option. The Jura coffee maker is valued for providing high-quality, customized coffee drinks at the push of a button. The guiding life motto emphasizes giving grace, acknowledging that decisions are made based on the information available at the time, even if hindsight reveals errors.
Nicole’s New Role at Google
Copied to clipboard!
(01:05:44)
  • Key Takeaway: Nicole is the Senior Director of Developer Intelligence in Google Core Developer, focusing on improving DevEx and measurement.
  • Summary: Her role involves improving developer experience, productivity, and velocity across Google’s properties and underlying infrastructure. A key focus is determining how measurement, feedback loops, and experience can be improved to drive meaningful change faster. Listeners can find her book at developerexperiencebook.com and connect with her at nicolefv.com or LinkedIn.