Experiment Nation: The Podcast

In this episode of Experiment Nation, host Jason sits down with Namata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.Together, they explore:- Ethical testing: how to balance user needs with business goals.- Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.- The role of design in CRO and experimentation.- Frameworks, guardrails, and collaboration strategies for building an experimentation culture.Whether you’re a CRO specialist, product manager, or experimentation nerd, this episode is packed with actionable insights you can bring to your own team.👉 Don’t forget to like, comment, and subscribe to stay updated on the latest in experimentation and growth!📖 Check out Prove It or Lose It: The (Mostly) No-Nonsense Guide to Surviving Experimentation Program Drama .⏱️ Chapters with Timestamps0:00 – Why design must be involved early0:22 – Welcome & intro to Jason0:43 – Guest intro: Namata Agarwal0:55 – Topics: Ethical testing & Leaks vs. Puddles1:18 – What is ethical testing?2:22 – Balancing business needs vs. user needs2:54 – Handling stakeholder pushback4:01 – Misaligned metrics and real solutions5:17 – The Leaks vs. Puddles analogy explained6:32 – Why better hypothesis building matters7:48 – Using Airtable & frameworks for test prioritization8:40 – Why experimentation must be collaborative9:24 – Reframing experiments around the user journey10:00 – Unpredictable user journeys & design challenges11:12 – Involving guest collaborators in experimentation11:50 – Mapping user journeys with Miro & qualitative research12:50 – Jobs To Be Done & other qualitative frameworks14:44 – Turning qualitative insights into hypotheses15:32 – How product design elevates CRO strategies16:11 – The importance of forms in experimentation17:36 – Why design should be included from day one18:20 – Design is not just visuals—it’s about impact19:11 – Governance, responsibility, and inclusivity in testing19:46 – Leak vs. Puddle framework deep dive22:58 – Running collaborative workshops25:45 – The “Stinky Fish” workshop format27:21 – Micro vs. macro frameworks for problem-solving28:05 – Balancing bold ideas vs. safe bets29:19 – Planning tests like an investment portfolio30:13 – Testing across 13 sites at scale30:26 – What Namata would A/B test in daily life31:57 – Art, painting, and creative balance32:42 – Martial arts & keeping calm32:56 – Wrap up & final thoughts33:14 – How to connect with Namata This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of the Experiment Nation Podcast, host Ward Vanestra sits down with Ryan Lux, one of the first team members at Eppo and now Experimentation Evangelist at Datadog.They dive into:What Eppo is and how it helps companies experiment at scaleWhy data warehouses are critical for trustworthy resultsThe recent Datadog acquisition of Eppo and what it means for the futureAdvanced methods like CUPED (noise-cancelling for experiments) and contextual banditsHow to scale experimentation culture and get promoted by driving adoption, efficiency, and trustworthinessWhether you’re a CRO, growth leader, or experimentation enthusiast, this episode is packed with insights from someone who’s been shaping experimentation for over a decade.📘 Check out Experiment Nation’s new book: Prove It or Lose It: The Mostly No-Nonsense Guide to Surviving Experimentation Program Drama → experimentnation.com⏱️ Chapters0:00 – Why win rates don’t really increase0:20 – Introduction to Ryan Lucht & Eppo1:20 – What Eppo does and who it’s for2:18 – Experimentation at smaller companies3:18 – Datadog’s acquisition of Eppo3:59 – The data warehouse advantage6:03 – CUPED explained: noise-cancelling for experiments9:13 – Segments, personalization & pitfalls11:09 – Heterogeneous treatment effects (HTEs)12:06 – Personalization with contextual bandits14:00 – Why Datadog acquired Eppo16:22 – Data observability & product analytics17:52 – Experimentation and site performance19:46 – Content, culture, and experimentation career paths22:10 – Every experiment is valuable (win, lose, or flat)24:17 – OKRs for experimentation leaders25:47 – Three key OKRs: adoption, efficiency, trustworthiness28:14 – Knowledge building and learning plans29:34 – The vision: experiments as the ultimate insight system30:24 – Ryan’s Substack “Everything is an Experiment”31:02 – Closing thoughts & where to connect This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode of the Experiment Nation Podcast, host Richard Joe welcomes Manuel Da Costa, founder of Effective Experiments and now Efestra, to unpack a critical issue in the experimentation industry: the trust gap between practitioners and business decision-makers.🎯 Manuel shares how running more experiments isn’t the goal—making better decisions is. He introduces the Learning Loop, explains why most experimentation programs stall, and gives practical strategies for connecting experimentation insights to executive decisions.🔍 Whether you’re in CRO, product experimentation, or experimentation governance, this is a must-watch for moving beyond vanity metrics and toward real impact.📌 Connect with Manuel:📍 LinkedIn: https://www.linkedin.com/in/manueldacosta/#CRO #Experimentation #ABTesting #EffectiveExperiments #ProductManagement #DigitalMarketingChapter00:00 – Experiments don't matter unless they drive decisions 00:30 – Meet Manuel Da Costa, founder of Effective Experiments 01:45 – Manuel’s journey: from web design to CRO and Lean Startup 03:00 – How experimentation became his product 05:20 – The limitations of spreadsheets in experimentation 06:45 – Building Effective Experiments to streamline the workflow 09:10 – What is the "Trust Gap" in experimentation? 11:30 – Why 88% of experimentation programs aren’t transformative 13:00 – The survival mindset vs. strategic experimentation 16:00 – The echo chamber of experimentation advice 17:40 – Why velocity and win rate are not the real success metrics 19:00 – The problem with how learnings are shared 21:00 – How to communicate insights in one powerful slide 23:00 – Building organizational learning with the “Learning Loop” 25:00 – Step-by-step breakdown of the Learning Loop framework 27:15 – Why experimentation needs executive alignment 29:00 – How to reset a tactical experimentation program 30:40 – Introducing Efestra: a governance platform for experimentation 32:00 – Final thoughts and how to get in touch with Manuel 33:14 – About the book: *Prove It or Lose It* This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this episode, we dive into Speero’s shift from CRO to Experiment-Led Growth, how they structure their testing systems, where to start with a new client, and what makes a good growth bet. Plus, we discuss the power of insights, the rise of EXL meetups and the future of Speero.Chapters----00:00 - Introduction 00:18 - Meet the Hosts & Guest 01:11 - From CRO to Growth Experimentation 02:55 - Building a System for Experimentation 05:00 - Organizational Buy-in and Culture Shift 06:02 - Client Reactions and Collaboration 08:00 - Applying Experimentation Across the Business 10:00 - Strategy vs. Tactics 11:30 - Why Create a Strategy? 13:02 - Making Better Bets with Experimentation 15:05 - Breaking Down Big Initiatives 16:41 - Landing Pages, Messaging & Paid Spend 18:20 - Irreversible Risks and Thought Experiments 20:15 - Research Frameworks: Qual & Quant 21:47 - Copy Testing and Messaging Insights 23:48 - Combining Qualitative and Quantitative Testing 25:49 - Using Ads to Accelerate Learning 26:33 - EXL Meetups: Community and Format 28:00 - How to Host an EXL Meetup 30:04 - Upcoming Global Meetups 31:35 - How to Connect with Paul Randall 32:40 - Outro#cro #experimentnation #abtesting This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

This episode features Achille Vaulot, a CRO expert, discussing the evolution of conversion rate optimization (CRO) and the steps to building a mature program. He highlights the shift from basic A/B testing to a data-driven, strategic approach encompassing research, thorough analysis (including post-test analysis of both primary and secondary KPIs), and effective client communication. Achille emphasizes the importance of planning for both successful and unsuccessful tests, managing client expectations, and fostering a culture of experimentation within the client’s organization. He stresses the value of qualitative data alongside quantitative results and the importance of aligning CRO strategies with broader business goals. The conversation underscores the crucial role of education and collaboration in scaling a successful CRO program. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Ekaterina (Shpadareva) Gamsriegler, a seasoned growth and marketing expert, shared her insights on mobile app experimentation. Here are 5 key takeaways:Mobile experimentation is more complex: Release cycles are longer, and dependencies on app versions can slow down learning.Think beyond the app: Experimentation should encompass ads, store listing pages, and the app itself.Borrow insights from other areas: Use data from paid advertising to inform store listing optimizations.Interconnectivity is key: Changes in one area can impact others. Strong stakeholder management is crucial.Document everything: Detailed documentation helps track results and avoid surprises.Check out the full conversation on Experiment Nation’s YouTube channel!LinkedIn: https://www.linkedin.com/in/ekaterina-shpadareva-gamsriegler/#mobilemarketing #appmarketing #experimentation #growthmarketing #experimentnation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Tracy Laranjo shares learnings from years of growing her career with Suchita Srinivasan, here are some key takeaways:1. CRO vs Experimentation: CRO focuses on driving performance and results, while experimentation centers around decision-making and validating ideas.2. Agency vs In-House: Tracy’s journey highlights the benefits of agency experience for rapid skill development and exposure to diverse challenges.3. Building Your Experimental Brand: Don’t be afraid to be yourself online, even if it’s not the typical thought leader persona.4. Adapting to Constant Change: Focus on fundamentals and building your skills; stay curious and open to new developments, but don’t fear the future.5. CRO is for Everyone: While AB testing requires scale, CRO encompasses qualitative research and user interviews, making it accessible for businesses of all sizes. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

This episode of Experiment Nation features an interview with Dr. Else van der Berg, a product management consultant for B2B SaaS startups and scale-ups. Dr. Else van der Berg discusses her process for helping companies become more outcome-driven, focusing on understanding the ideal customer profile (ICP) and its associated value. She also shares insights on common challenges faced during the testing phase, including participant recruitment, data analysis, and differentiating signal from noise.Highlights🎯 Dr. Else van der Berg emphasizes the importance of defining a narrow and well-defined ICP, emphasizing the need to focus on a specific customer segment.🔎 She believes in a layered approach to testing, using a variety of methods, including interviews, landing page tests, and moderated user tests.🗣️ Empathy plays a crucial role in interpreting user feedback, recognizing that data interpretation can be subjective and require a “gut feeling” alongside evidence.📈 Dr. Else van der Berg stresses the importance of finding a mentor within the field and learning on the job to master product management techniques.👩‍💻 She encourages aspiring product managers to reach out to her via LinkedIn to learn more and potentially connect for career advice. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

We recently had the pleasure of chatting with Rishi, a Shopify product expert. Here are 5 key takeaways for optimizing your product pages:1) Product pages are the Grand Central Station of conversions. They’re where buyers make the decision to buy or leave.2) Pop-ups aren’t as effective as you think. Focus on valuable content, not annoying interruptions.3) Content is king (but don’t be afraid to go long!). Detailed product descriptions are more effective than short ones.4) Build trust above the fold. Establish your company’s credibility before highlighting product features.5) Understand the filters buyers use. Address common concerns like “too good to be true” to build confidence.Want to learn more? Check out Rishi’s website, frictionlesscommerce.com!#ecommerce #productpages #conversionrateoptimization #marketing #experimentation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Black Friday is coming, and the pressure is on. But don’t worry, Ilan Hurwitz has some expert advice to help you navigate the chaos.Here are five key takeaways from his chat with Tracy Laranjo:1) Start now! Don’t wait until the last minute to get buy-in and start testing.2) Speak the language of profit. Showcase the ROI of CRO to your stakeholders, focusing on increasing conversions and profits.3) Don’t be afraid to experiment. Test offers, product placements, and even email campaigns outside of standard website A/B testing.4) Focus on the customer journey. Don’t just look at your website. Analyze customer service tickets, reviews, and LTV to identify areas for improvement.5) Think beyond discounting. Explore creative offers like gift with purchase, cross-sells, and subscription incentives to boost sales.Get ready to conquer Black Friday!#BlackFriday #CRO #ecommerce #marketing #experimentation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Experimentation is a journey, not a destination. Don’t expect to build a culture of experimentation overnight.Cross-functional teams are critical. Breaking down silos and bringing together engineers, product managers, and designers will create a unified vision and drive faster iteration.Leadership buy-in is essential. A top-down approach to embedding experimentation ensures everyone is on board and resources are allocated strategically.Don’t underestimate the importance of organizational rhythms. Establish clear processes and structures to ensure smooth collaboration and avoid friction.Embrace the growing pains. Change is rarely easy. Be prepared for challenges and celebrate milestones along the way. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

In this podcast episode of Experiment Nation, hosted by Charlotte Bomford, guest Iqbal Ali, an experienced web designer, researcher, and comic book writer, discusses the concept of solution bias and the importance of identifying underlying problems before diving into solutions. Ali emphasizes using AI to help understand problem landscapes, avoiding hastily jumping to solutions, and improving experimentation processes. The conversation covers techniques like premortems and frameworks to minimize potential issues and ensure experiments address real user problems. Ali highlights the collaborative potential of AI tools in maintaining the connection between problem statements and solutions, advocating for a more thoughtful approach to A/B testing and problem-solving in product development. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Ruben de Boer, a lead consultant at Online Dialogue and the founder of Conversion Ideas, shares his approach to conversion rate optimization (CRO) and experimentation. With 15 years of experience, Ruben emphasizes the importance of data-driven decision-making and understanding human behavior. He highlights the shift from focusing solely on digital products to also optimizing organizational processes. Ruben provides insights into the challenges of implementing experimentation in organizations, the need for higher management support, and the significance of aligning with colleagues' goals and motivations. He advocates for a research-driven approach, continuous feedback, and fostering an experimentation culture within teams to drive better results. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Tymur Donets, a seasoned Growth PM and experimentation consultant, shared some powerful insights during his Experiment Nation talk. Here are five key takeaways:Growth PMs are holistic: Don’t be siloed! Growth PMs work across the entire customer journey, ensuring marketing and product teams are aligned.Commercial awareness is key: Both Growth PMs and CROs must understand the business impact of their work and how it contributes to the bottom line.Empathy is vital: CROs need to understand the pressures their PM colleagues face and proactively offer support.Leverage your expertise: If you’re a CRO, consider specializing in Growth PM to expand your impact and skillset.Transparency is key: Clear communication between teams helps everyone understand their roles and collaborate effectively.Find Tymur on LinkedIn: https://www.linkedin.com/in/tim-donets-2358364aChapters:0:00 - Introduction and The Role of a Growth PM0:58 - Growth PM vs. Product Manager2:40 - Tymur’s Journey and How He Became a Growth PM5:23 - Importance of Business Savvy and Data Literacy8:30 - Bridging the Gap Between Marketing and Product Teams12:42 - Experimenters as Facilitators16:38 - The Complexity of a Product Manager’s Role19:50 - CRO Responsibilities and Building Relationships with PMs24:48 - Transitioning from CRO to PM29:00 - Finding Growth PM Opportunities and Wrap-Up#growthmarketing #experimentation #cro #productmanagement #collaboration This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Kelly Wortham dropped some serious knowledge bombs in our recent Experiment Nation chat. Here’s what stood out:- CRO isn’t just for marketers: Lawyers, project managers, even developers can thrive in this field. It’s all about a love of learning and adaptability.- Don’t underestimate the power of mentorship: Kelly has THREE mentors, each focusing on different aspects of her career. Find your own advisors to help you grow!- Certifications are nice, but experience trumps all: While CXL is great, hands-on testing platform experience is the real deal.- UX psychology is CRUCIAL: Understand your users’ motivations and behaviors to create truly effective tests. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Simar, CEO and Founder of OptiPhoenix, shared invaluable insights on A/B testing and CRO at Experiment Nation. Here are 5 key takeaways:Don’t rely solely on macro goals: Micro goals reveal user behavior, driving better testing strategies and informing future hypotheses.Iterative testing trumps scrapping: Learn from failed tests by adjusting hypotheses and running new experiments, boosting your win rate.QA is critical, but often neglected: Invest time in thorough QA, ensuring your test is accurate and integrated correctly.Outsourcing AB test development can be beneficial: Consider outsourcing for faster scale-up, access to expert resources, and reduced development burden.Prioritize experimentation within your workflow: Make A/B testing an integral part of your product development process, leading to better decisions and faster results. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Avishek Basu Mallick, a Senior PM at CheapOair, talks about:- Experimentation at scale: Even with high traffic, rigorous experimentation is crucial.- Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas.- Communicating progress: Regular updates with leadership build trust and ensure alignment.- Personalization is nuanced: Distinguish between customization, segmentation, and true personalization.- Generative AI’s potential: Explore its use for simplifying the travel planning experience. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Michael St Laurent, a seasoned CRO expert, shared his insights on large-scale testing Here are 5 key takeaways for marketers:Prioritization is key: Large-scale testing requires prioritizing tests based on potential impact and aligning with overall business goals.AI is the future, but humans guide it: Mike emphasizes the power of AI in automating processes and analyzing data, but highlights the importance of human expertise and guidance.Data quality is crucial: AI models rely heavily on the quality of data. Invest in building a robust data infrastructure to ensure accurate results.People are the biggest challenge: Navigating internal politics and managing stakeholder expectations is often the biggest hurdle for large-scale testing.Building a testing culture is a win: The ultimate goal of large-scale testing is to create a data-driven culture that fosters continuous improvement and better user experiences. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Ellie Hughes, a top CRO expert and Experimentor of the Year, shared some incredible insights in her recent Experiment Nation interview. Here are 5 takeaways to boost your experimentation game:- Start Small and Stay Curious: Ellie emphasizes that experimentation is a journey, not a destination. Start small, stay curious, and never stop learning!- Prioritize Value, Not Tasks: Focus your time on activities that truly drive client value, like uncovering insights and ideation, rather than spending hours on tedious tasks.- Set Clear Expectations: Clearly define reporting timelines and communicate about peeking, managing client expectations, and setting the stage for success.- Embrace Iteration: Experimentation is an iterative process. Be open to changing your process, adapting to new learnings, and continuously improving.- Maximize Impact Through Big Ideas: Larger changes can be easier to detect, resulting in faster results and more impactful growth. Encourage clients to think big!#experimentation #CRO #conversionrateoptimization #marketing #digitalmarketing #experimentnation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com

Join us for a deep dive into conversion rate optimization on low-traffic sites with these actionable insights Chris MercerMeasure before optimizing: Understand the current user journey and behaviors to identify areas for improvement.Test sequentially: Iterate faster on low-traffic websites by testing changes sequentially instead of using AB tests.Focus on the right metrics: Prioritize actions that drive meaningful outcomes, such as lead generation or revenue.Listen to the market: Gather user feedback to inform your optimization strategy and meet their needs.Automate measurement: Leverage Google Analytics 4 to collect and analyze data effectively to guide decision-making.#CRO #Experimentation #ConversionOptimization This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit experimentnation.substack.com