The GAIN Conversion Podcast

What does experimentation look like when you’re optimising one of the biggest retail sites in the world?In this episode, we talk with Kevin Washington (former Director of Product Management over Experimentation at Best Buy) about how large e-commerce teams think about testing when revenue comes from multiple directions: purchase conversion, retail media, and marketplace growth.We cover:The “job of the page” mindset (homepage vs. PLP vs. PDP)Choosing primary metrics vs. guardrails (and why “purchase conversion” can mislead)How ad placements can cannibalise core retail performanceWhy short-term revenue wins can quietly damage long-term trust and LTVNon-endemic ads, emergency testing, and what “passes the smell test”Note: This episode includes a screen-share walkthrough — watch the video version on YouTube for the full experience.Bonus: Kevin will be speaking at UNITE Summit.

In this episode of the Conversion Podcast, Matt is joined by Colin, Vice President of North America at Kameleoon, to explore prompt-based experimentation (PBX) and how AI is transforming the way experiments are built and launched.Colin breaks down what PBX is, how it connects AI-driven “vibe coding” with experimentation, and why the build phase has long been the biggest bottleneck for experimentation teams. They discuss how PBX allows teams to create variants, tests, and prototypes directly on live websites using natural-language prompts, cutting build time from weeks or days down to minutes.The episode also covers where PBX fits into the experimentation lifecycle, the importance of keeping humans in the loop, current limitations of the approach, and what adoption looks like for teams just getting started. They close with a discussion of the Prompt-a-thon, an open challenge designed to explore how far prompt-based experimentation can be pushed.What is prompt-based experimentation (PBX) isHow PBX differs from AI coding and vibe-coding toolsWhy building experiments is the biggest friction pointHuman-in-the-loop experimentation with AIAdoption tips and limitationsThe Prompt-a-thon

Setting the scene for this one and diving right in. Tim talks about successful experimentation being cultivated, not designed or manufactured and why building adoption for experimentation (or any organisational change) comes down to a simple question: are you on the soapbox or are you driving the bus?Tim breaks down why correcting people (“you’re doing it wrong”) creates defensiveness, why the “soapbox” approach is the shortcut, and how the “bus ride” approach builds sustainable adoption. He shares an example from Minecraft, walking through a product lifecycle approach, observation, theory, test, bringing teams along for research, insights, solutioning tied to OKRs, prioritisation, experimentation, analysis, and implementation.They then shift into AI and experimentation: where Tim sees real value today (speed from large datasets to insights), where he’s sceptical (AI-generated high-fidelity designs and experiment variations), and where opportunities may be strongest (analytics, projections, predictive modelling, multi-armed bandits, variance reduction, and qualifying/disqualifying users).They close with Tim sharing why he opened time slots for people to chat about experimentation, why he was encouraged by his manager, and why LinkedIn is the best way to connect.What’s covered (from the conversation):“Soapbox vs bus ride” and sustainable adoptionDefensiveness, reactance, and avoiding “shaming” in change effortsMinecraft example: bringing teams through the product lifecycle with experimentationSeparating observation → theory → testTying solutioning to OKRs and prioritising experimentsAI in discovery: summarising large qualitative (and quantitative) datasets quicklySpot-checking + context pitfalls (e.g., “creeper,” profanity interpreted as negative)Prompt templates vs AI agents with guardrailsScepticism on AI for full-fidelity designs and experiment variationsAI/ML in analytics: sample size, test duration, forecasting, bandits, variance reductionCommunity learning + LinkedIn as the main contact channel

In this episode, Matt sits down with Tim Mehta, Experimentation PM at Minecraft, to discuss how one of the world's biggest games approaches growth and testing. Tim shares his journey from agency life to big tech (Microsoft, Lucid) and reveals the inner workings of Minecraft's experimentation program.We dive deep into:• The "Spillover Effect": Unique challenges of A/B testing in a multiplayer game environment and how to solve them.• Delivery vs. Growth Mindset: Why simply shipping features isn't enough and how to shift focus to measurable outcomes.• The Ladder of Evidence: Why tests backed by user research have a 50% success rate compared to the industry average.• 3 Pillars of Experimentation: Trustworthiness, Velocity, and Democratization—and why trust must always come first.Whether you're a Product Manager or Growth Marketer, Tim's insights on building a culture of evidence-based decision-making are invaluable.

Join host Matt Wright with Brent Kostak and David Arbour from Adobe as they reveal how the new Experimentation Accelerator is transforming digital experimentation through AI automation, causal insights, and agentic workflows.Learn how Adobe is helping brands scale testing velocity, uncover deeper experiment insights, and unify teams across marketing, product, and analytics. Featuring early beta results, real-world use cases, and practical advice for adopting AI-driven experimentation successfully.

Are your experiments too safe? In this coffee break episode of The Conversion Podcast, Matt and Steven reveal how to build an experimentation “risk profile” that balances safe, incremental wins with bold bets your competitors are too nervous to run.You’ll learn why treating your test backlog like an investment portfolio unlocks competitive advantage, how A/B testing acts as your safety net, and real-world examples where high-risk experiments drove massive gains, from Domino’s delivery fees to SaaS free trial removals.Whether you’re running CRO programs, growth experiments, or digital product tests, this episode shows you how to reduce fear, maximise ROI, and scale with confidence.

When your paid ads promise one thing and your landing page says another, conversions suffer. In this episode of The Conversion Podcast, Matt and James dig into a classic - yet often overlooked - challenge in digital marketing: maintaining a consistent “scent trail” from ad to landing page.The case study centres on a SaaS brand running heavily paid search campaigns, both branded and non-branded. Their ads emphasised a free trial, but the landing page buried this message at the very bottom. The result? Mismatched expectations and flat conversions, despite expanding the page with more product information and CTAs.By aligning the landing page headline and CTA with the ad promise, “Start your free trial today”, and reinforcing key points like 14-day trial length, easy setup, and cancellation flexibility, the team achieved a 7% lift in leads.This episode is a must-listen for performance marketers, growth teams, and product leaders who want to maximise ROI from paid campaigns and ensure messaging alignment drives results.#paidmedia #saasmarketing #conversionoptimization #digitalmarketing #landingpages #growthmarketing #abtesting #customerjourney #ppc #marketingstrategy #onlinemarketing #performanceadvertising #cx

Sometimes, the “no-brainer” ideas turn out to be… not so smart.In this episode of The Conversion Podcast, Matt and James break down a fascinating experiment run on KitchenAid’s iconic stand mixers — one that proves why you should never skip testing, even when you’re certain the idea will win. The team merchandised a 90-day money-back guarantee on product and listing pages, expecting to reduce buyer anxiety and boost conversions. Instead? Orders dropped by double digits.We explore why this happened, the psychology behind it, and how context changes everything when it comes to persuasion tactics. You’ll learn how guarantees can sometimes introduce doubt instead of removing it, why category leaders need different messaging strategies, and how to structure clean, holistic experiments for reliable insights.If you’ve ever launched a “sure thing” without testing — or almost did — this episode will make you think twice.#conversionrateoptimization #abtesting #digitalmarketing #experimentdesign #productmanagement #growthmarketing #ecommerce #uxdesign #onlinemarketing #conversionoptimization #customerpsychology #contentstrategy #productpage #marketingstrategy #businessgrowth

In this bonus episode of the Conversion Podcast, James shares a fascinating case study about using video to improve product comprehension for a tech product, a cell phone signal booster. We discuss the results of running experiments that tested video placement on landing pages and product pages, revealing unexpected insights about video’s effect on conversions. By iterating and testing in different ways, the team discovered that adjusting video positioning led to a 35% increase in sales despite a drop in video views. Tune in to hear how this experimentation approach helped boost product comprehension and sales, and why understanding user behavior is crucial for conversion optimization. Perfect for marketers, product managers, and business leaders looking for data-driven strategies.

HP’s Instant Ink has revolutionized the ink cartridge industry, offering consumers up to 50% savings and the convenience of never running out of ink. Launched in 2013, it now boasts over 10 million subscribers across 38 countries. In this episode, Sovrin Tolia, HP's Head of Product, and Stephane Bardin, Worldwide Product Manager, dive deep into the secrets of Instant Ink’s rapid growth and success. From overcoming initial consumer behavior challenges to utilizing experimentation and innovative business models, they share key insights into scaling a successful subscription-based service. Learn how HP transformed the printing experience and continues to evolve with the help of data-driven decisions and continuous testing.#HP #InstantInk #SubscriptionBusiness #Innovation #GrowthStrategies #CustomerExperience #BusinessModelInnovation #DataDriven #ABTesting #MarketingStrategy #ProductLedGrowth #ConsumerBehavior #TechGrowth #DigitalTransformation #SubscriptionServices

From CRO to Decision-Making: How to Truly Unlock Value from ExperimentationIn this episode of The Conversion Podcast, we explore how experimentation can go beyond simple conversion rate optimization (CRO) to become a powerful decision-making and risk management tool.We dive into:Why experimentation isn’t just for optimizing pages — it’s for making big, bold business decisionsHow to frame experiments as risk management instead of just validationBalancing safe bets vs. high-risk experiments in your testing portfolioReal-world examples of using experiments to guide pricing, product, and strategic movesIf you’re a marketer, product manager, or leader looking to make smarter, bolder decisions with data, this one’s for you.

Prioritization That Works: How to Turn Your Experiment Backlog into Real WinsIf you could wave a magic wand and design the perfect experiment prioritization system, what would it look like? In this Coffee Break episode of The Conversion Podcast, we explore exactly that.Stephen and Matt dive deep into why prioritization is one of the most overlooked levers for driving conversion program success. They unpack:Why traditional impact/ease models failHow to make prioritization objective (and bias-proof)The power of meta-analysis and the Lever FrameworkHow AI-driven prioritization tools like Confidence AI are reshaping the gameWhether you’re running 20 tests a year or 200, this episode gives you a roadmap for testing smarter, learning faster, and driving more impact.Grab your coffee, this one might just change how you think about your entire experimentation program.Links we mentioned in the episode:LeversAI Confidence AI NewsletterLevers Framework

How mature is your experimentation program, really?In this episode of the Conversion Podcast – Coffee Break Edition, Matt is joined by Steph, Senior Consultant at Convergent, to discuss how companies can better evaluate and grow their A/B testing and digital experimentation programs using a new, evolved maturity framework.They break down:The 5 stages of experimentation maturity (Reactive to Optimized)Why the classic “Packet” model needed an updateKey dimensions of maturity: goals, delivery, culture, toolsHow to benchmark your team and set actionable growth goalsCommon pitfalls, including over-engineering and under-trackingWhether you're just starting out or looking to scale your experimentation practice, this episode offers practical insights to improve velocity, volume, and value—and turn your testing program into a true growth engine. 

Using Friction to Drive ConversionsA Surprising Experiment That Increased Leads by 9%In this bonus episode of The Conversion Podcast, James shows us one of his top five most fascinating experiments, and it’s all about friction.Instead of streamlining the user experience, James intentionally added a step in a lead generation funnel using behavioral psychology: commitment bias and the foot-in-the-door technique. The result? A 9% increase in email signups.You’ll learn:Why not all friction is bad in customer journeysHow a simple yes/no question changed conversion behaviorWhat happens when people say "no" and still convert anywayHow to apply psychological principles to your marketing funnelPerfect for marketers, CRO specialists, UX designers, and anyone obsessed with optimizing the user journey.

You can find the full transcript for this episode and video here.Is perfect testing slowing down your business? In this episode, James Flory breaks down the difference between pragmatic and purist approaches to experimentation and why focusing on speed, context, and decision-making matters more than statistical perfection.Discover how to select the optimal testing method, foster organizational buy-in, and identify your “killer test” that drives tangible impact.Topics covered:Pragmatic vs. Purist experimentationHow to test with limited dataReal-world business examplesCommunicating results to stakeholdersBuilding a data-driven cultureFeaturing: James FloryHost: Matt WrightHead to conversion.com to find out more!#Experimentation #ConversionOptimization #ABTesting #DigitalStrategy #UXResearch #DataDriven