Discover
The Experimentation Edge
The Experimentation Edge
Author: Growthbook
Subscribed: 0Played: 0Subscribe
Share
© Ashley Stirrup
Description
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
62 Episodes
Reverse
SummaryHow do you measure the quality of an AI-generated ad when designers, legal teams, and advertisers may all define “good” differently? Nishanth Kadiyala, Product Leader at HelloFresh and formerly an ad tech product leader at Walmart Connect, joins Ashley Stirrup on The Experimentation Edge to explain how the team shifted from subjective creative preferences to measurable acceptability, compliance, and performance.Nishanth walks through the realities of building creative automation for smaller advertisers: segmenting users, validating AI output against a large evaluation dataset, running A/B tests to earn stakeholder buy-in, and letting real advertiser needs shape the product. He describes moving creative production from weeks to minutes as more than 250 advertisers tried the product, then shares how those lessons inform creative velocity and diversity in his current work at HelloFresh.Chapters00:00 Meet Nishanth Kadiyala01:16 Inside Walmart Connect’s retail media stack05:30 Why B2B experimentation requires a different playbook07:30 Moving from subjective quality to acceptability09:45 Helping smaller advertisers create better ads12:21 Iterating with user research and phased launches14:47 Testing AI creative against human-designed ads15:57 Building a golden dataset for AI evaluation18:05 Scaling across products, advertisers, and categories22:18 Defining baseline quality and performance24:50 Unexpected lessons from advertiser feedback26:35 Choosing the right level of AI control28:33 Results: 250+ advertisers and minutes to create29:44 From manual reviews to AI evaluation pipelines31:35 Creative velocity and diversity at HelloFresh34:57 AI’s role in the next generation of testing36:40 Closing reflectionsTakeaways-An ad that looks better is not necessarily an ad that performs better; A/B testing turns aesthetic debates into learning.-AI creative must first clear acceptability guardrails for product fidelity, context, and brand compliance, even while models keep improving.-Start with the advertisers and product categories that current technology can support, then expand with evidence from real users.-A representative golden dataset and ongoing evaluations are as important as the generative model itself.-Nishanth says the Walmart Connect effort reached more than 250 advertisers and reduced creative production from multiple weeks to minutes.-Creative velocity, diversity, and excellence are now central to Nishanth’s paid-social work at HelloFresh.Connect with the GuestNishanth Kadiyala on LinkedIn: linkedin.com/in/nishkadiFor more of Nishanth’s perspective on Walmart Connect and AI-powered ads, listen to his Rethink Retail conversation.SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
SummaryDelina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams.Delina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and combined offers. She also shares how to diagnose a losing test, her plan for meta-learning and uplift modeling, and how AI makes it cheaper to test ideas before investing in a full build. For product, data and growth leaders who want experiments that measure the thing they are trying to drive. Chapters00:00 Wealthsimple, 4 million clients and a never-ending experimentation world02:46 Two years of formalizing experimentation03:42 What Misplay taught her about when you need a test06:00 Rules of thumb: when to test and when to just ship08:21 Ronny Kohavi, Bing and the value of a millisecond09:26 How Wealthsimple's 3 data science teams run experiments12:15 Engagement features, lagging metrics and early signals14:49 Launching margin and finding an education gap16:57 Why surveys need experiments19:10 Sharing results and building systemic learning21:56 The margin promotion that taught a messaging lesson26:23 Experiments as a way to learn about people27:39 Messaging, targeting or experience: diagnosing a losing test30:35 Meta-learning and uplift modeling33:42 Using AI to test ideas faster and cheaper Takeaways- Skip the test when you are removing a clear blocker for clients, and test when the change is optional or the outcome is uncertain- Deposits and revenue can lag 60 to 90 days in financial services, so pick early metrics that are proven to lead to them- Combining 2 incentives in one message hides what you are trying to learn; decouple offers so each test isolates one outcome- When a feature you believe in loses, check whether it is a messaging, targeting or experience problem, and design variants to find out- Pooling every past experiment into one view turns tests into a mega learning system for pricing, messaging and personalization Connect with the GuestDelina Ivanova LinkedIn: https://www.linkedin.com/in/delina-ivanova/Company Website: https://www.wealthsimple.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
SummaryAndy Denning, Product Leader, joins host Ashley Stirrup to talk about the judgment that comes before any A/B test. Drawing on his experience with a two-sided jobs marketplace serving job seekers and recruiters across the UK and US, Andy explains why underpowered, badly chosen tests can create false confidence, how experimentation brings organizational honesty, and why his teams start with the riskiest assumption instead of the most comfortable one. He breaks down a US branding test that barely moved applications, how traffic and market shape test design, and why AI makes judgment, not test volume, the real bottleneck. For product and growth leaders who want fewer wasted tests and more learning from the ones they run. Chapters00:00 Test the riskiest assumption, not the comfortable one00:55 A two-sided jobs marketplace in the UK and US01:53 Andy's product role and AI-powered job matching02:52 Deciding what's worth testing at a smaller company04:21 Underpowered tests and false confidence04:45 Experimentation as organizational honesty05:49 From prototyping software to Optimizely06:58 Cross-functional teams that own experiments end to end08:13 Sharing learnings and popcorn sessions10:31 Guardrail metrics in regulated industries11:53 Experimenting across the UK and US markets13:38 The branding test that didn't move applications17:04 The judgment layer: is the test worth running?19:03 Experiment design starts with the riskiest assumption20:39 What to do when a test will take 6 weeks21:27 B2C vs B2B: what your traffic can support22:27 Pre-mortems, non-goals and trade-offs23:49 AI and the flood of low-conviction tests27:36 Where to spend human judgment28:26 The AI apply tsunami in hiring Takeaways- Test the riskiest assumption, not the most comfortable one; it is the one the bet rests on- Underpowered or badly chosen tests can manufacture false confidence that is worse than not testing- Before you test, ask whether the outcome is uncertain and whether you would ship it anyway- Run the numbers up front; if a test needs 6 weeks, find a faster or cheaper way to learn- AI makes variants cheap, so the bottleneck is now the quality of the hypothesis Connect with the GuestAndy Denning LinkedIn: https://www.linkedin.com/in/andy-denning-547a0862/SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
SummaryTom Gores, Director, Product - Digital Experience & E-commerce at GE HealthCare, joins host Ashley Stirrup to share what changes when you take an experimentation program from B2C retail to B2B healthcare. Tom built his first A/B testing program at Kohl's 12 years ago and now runs a centralized program at the center of GE HealthCare's digital ecosystem, where buying cycles can take a year and each qualified lead carries much higher stakes. Tom explains why a Contact Us test that lifted clicks also lifted junk leads, how his team turns one-off page ideas into bigger, broader tests across templates, and why planning for failure before launch keeps experimentation framed as learning. He also covers the 3 areas where he sees experimentation growing, people and culture, data, and technology, and the metrics he watches, from CSAT and conversion rate to visibility in AI search. For product, digital and growth leaders running experiments in long, high-stakes B2B journeys. Chapters00:00 Welcome Tom Gores of GE HealthCare00:51 From Kohl's to GE HealthCare: 12 years of A/B testing02:01 GE HealthCare's digital ecosystem02:51 B2B vs B2C: longer buying cycles and the long game03:49 A centralized program with lower traffic and higher stakes05:52 The Contact Us test: more clicks, junkier leads07:26 Funneling the right buyers to the right experience08:47 Designing experiments for the most learning10:33 Planning for failure and picking leading indicators13:07 Where experimentation grows next: people, data and technology15:22 CSAT, conversion rate and AI visibility17:34 How GE HealthCare measures CSAT Takeaways- In B2B healthcare, buying cycles can run months to a year, so experiments need leading indicators and long-term tracking rather than instant conversions- Pushing a metric in front of people can lift clicks while lowering lead quality; check results with the sales teams who work the leads- Turn a single-page idea into a bigger, broader test by applying it to a whole template, so thousands of visitors are affected instead of hundreds- Map out what you will do if a test fails before launch, so results stay framed as learning and avoid wasted builds- Track CSAT, conversion rate and AI visibility together to measure an ecosystem that serves many audiences and journey stages Connect with the GuestTom Gores LinkedIn: https://www.linkedin.com/in/tom-gores-6925a81aCompany Website - GE HealthCare: https://www.gehealthcare.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io
SummaryJeremy Echols, Director of Digital Analytics at Glassdoor, part of Indeed, joins host Ashley Stirrup to talk about building an experimentation program around people, not sessions. Jeremy came to analytics from an unusual path, as a former pastor with a master of divinity, and he sees math and statistics as a way to do applied psychology. Jeremy explains how Glassdoor went from a slow homegrown system and a handful of tests a year to dozens of tests a quarter, with a center of excellence, office hours, Amplitude templates and a shared data language. He breaks down an Indeed account-linking test that exposed variant jumpers across devices, why every test needs a hypothesis and a curious set of secondary metrics, and how some friction is worth keeping. For analytics, product and growth leaders who want faster iterations and deeper learning from every test. Chapters00:00 Sessions don't buy anything, people do00:42 Glassdoor's mission: workplace transparency01:27 Glassdoor Recruiter: a recruiter that works for the job seeker02:50 From pastor to analytics: statistics as applied psychology03:46 Why synthetic users can't replace real people04:14 Centralizing experimentation at Glassdoor05:24 Templates, guardrails and an experimentation community06:49 The account-linking test that exposed variant jumpers08:10 Why users browse without logging in09:48 Guest checkout hides the customer journey11:01 Costco and the case for personalization11:40 What to learn when a new feature fails12:11 Write the hypothesis, then ask what else might happen14:00 Tying experiments to your analytics funnels14:51 Pre-mortems and testing the opposite15:56 When removing friction hurts the funnel17:04 Turning experimentation marathons into sprints18:19 One data language for every team19:02 Glassdoor Recruiter and SuperMatch20:35 Hiring managers and passive candidates Takeaways- Measure people, not sessions; users move across devices, so incentivize login to see the whole journey- Bake best-practice settings and guardrail metrics into test templates so no one has to remember them- Write a hypothesis for every test, then add the secondary metrics that could teach you something unexpected- Some friction is good; more leads can mean fewer qualified leads and a weaker downstream funnel- One shared data language across marketing, email and product makes experiments faster and easier to compare Connect with the GuestJeremy Echols LinkedIn: https://www.linkedin.com/in/jeremyechols/Company Website: https://www.glassdoor.com SponsorGrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide. Go to growthbook.io








