Guest:

Ellen Brandenburger – Product leader and coach; former head of product at Chegg Skills and Stack Overflow’s data licensing team.

What we cover in this episode:

• How Ellen joined Stack Overflow just two weeks before ChatGPT launched, reshaping the company’s future overnight


• The creation of Overflow AI: a team tasked with exploring “what’s just now possible” for developers


• Four iterations of conversational search:


• V1: a chat UI on top of keyword search


• V2: semantic search to handle natural questions


• V3: fallback to GPT-4 for gaps in Stack Overflow’s corpus


• V4: adding RAG for attribution and transparency


• Why attribution and transparency were critical for developer trust


• How the team used simple spreadsheets and subject-matter experts to evaluate answer accuracy, relevance, and completeness


• Why Stack decided to sunset conversational search despite heavy investment—what they learned and why it wasn’t wasted


• The pivot to data licensing: how Stack Overflow leveraged its 14M+ Q&A corpus to power LLM training and benchmarks


• Building industry benchmarks with subject-matter experts to prove Stack data improved LLM accuracy and relevance

Key lessons:

• Take one bite of the apple at a time—prototype, learn, iterate


• Product in the AI era means managing probabilities, not certainties

Links & References:

• Ellen Brandenburger on LinkedIn


• The Changing State of the Internet and Related Business Models


• ProLLM: LLM benchmarks for real-world use-cases