Why Testing is Hard and How to Fix it with Will Wilson
Digest
Will Wilson, co-founder and CEO of Antithesis, shares his journey from mathematics to leading a startup focused on revolutionizing software testing with deterministic simulation. He explains the limitations of traditional testing methods like property-based testing and fuzzing for complex, non-deterministic systems. Antithesis addresses this by using a deterministic hypervisor and memory deduplication to ensure repeatable testing. The company's value proposition lies in providing powerful, efficient bug-finding capabilities with a low barrier to entry. Wilson also touches on the impact of AI code generation, the challenges of autonomous verification, and the importance of non-functional properties. He highlights the high leverage of property-based testing and the effectiveness of example-based testing. The discussion then shifts to Antithesis's unique engineering culture, emphasizing collaboration, deliberation, intellectual humility, and long tenures, with strategies to maintain this culture during growth. The importance of team externalities and the analogy of Hasidic Jewish merchants are used to illustrate the benefits of a strong, cohesive culture. Finally, Antithesis's organizational agility and adaptation to the AI coding revolution are discussed, underscoring the critical role of senior leadership in modeling desired behaviors.
Outlines
Will Wilson's Journey and the Genesis of Antithesis
Will Wilson, co-founder and CEO of Antithesis, details his path from mathematics to distributed databases and his inspiration for Antithesis, which stems from building a deterministic simulation framework at FoundationDB to revolutionize software testing.
Understanding Property-Based Testing, Fuzzing, and Testing Complex Systems
The conversation explores property-based testing (PBT) and fuzzing, explaining their concepts and limitations in testing large, complex, non-deterministic systems. It highlights the difficulties in managing state space complexity and unpredictable behavior with traditional methods.
Deterministic Simulation Testing: The Antithesis Solution
Deterministic simulation testing is introduced as a solution to overcome non-determinism in software, enabling repeatable and efficient bug detection. Antithesis achieves this through a deterministic hypervisor and memory page deduplication for efficient state space exploration.
Antithesis's Value Proposition and the Role of Properties
Antithesis aims to provide powerful, efficient, and bug-finding testing capabilities with a low barrier to entry. The importance of defining system properties is discussed, as even partially specified systems can reveal many bugs.
Business Strategy, AI Code Generation, and Testing as an Arbitrage
Will discusses Antithesis's strategy of selling a "superpower" focused on safety and speed, appealing to businesses with critical systems. The rise of AI code generation exacerbates the need for robust verification, making testing a critical bottleneck. Wilson explains his initial attraction to testing as a neglected but important field offering an arbitrage opportunity.
Strategic Investment and Jane Street's Unique Role
The discussion covers Antithesis's successful funding rounds, emphasizing market timing and pre-work. Jane Street's investment is highlighted as a unique strategic advantage due to their customer relationship, providing aligned perspectives and product feedback.
Antithesis, AI Code Generation, and Autonomous Verification Challenges
The potential and limitations of Antithesis for AI code generation are explored, focusing on fast feedback loops. The fundamental challenge of autonomous software verification, including the "evil genie" problem and the need for LLMs beyond specification followers, is discussed.
AI Training Perils and Limitations in Maintaining Non-Functional Properties
The discussion highlights how AI training can lead to undesirable behaviors like "evil genius" tendencies due to over-optimization. Anthropic's AI-built compiler is presented as a case study of AI limitations in maintaining non-functional properties like type checking, leading to system degradation.
Evaluating Antithesis, Property-Based Testing, and Example-Based Testing
The "Antithesis" approach to high-powered, end-to-end randomized testing is evaluated. The high leverage of property-based testing (PBT) and the surprising effectiveness of example-based testing, especially when combined with strong non-functional properties, are discussed.
Impossibility Results, Type Checking, and Practical Engineering
Impossibility results can guide engineers by revealing why techniques fail in extreme cases, allowing focus on practical program subspaces. Even complex theoretical concepts like doubly exponential type checking are rarely encountered in real-world programs, making engineering tasks more tractable.
Antithesis's Internal Engineering Culture: Collaboration and Deliberation
Antithesis fosters intense collaboration and deliberation, prioritizing discussion and debate before committing to solutions. This approach minimizes costly mistakes and enhances communication by avoiding hierarchy and encouraging open feedback.
Culture of Ideas, Titles, and Preserving Culture During Growth
Antithesis eschews formal titles, emphasizing ideas and contributions. They maintain culture during growth by controlling hiring pace, rigorous interviewing for technical skills and cultural fit, and designing interviews to assess behavior under challenge.
Team Externalities, Long Tenures, and Organizational Identity
Positive and negative externalities on the team often outweigh individual contributions. Strong esprit de corps, quirky choices, and a focus on long tenures contribute to organizational identity and stability, reducing turnover and preserving institutional knowledge.
Organizational Agility, Pivoting, and Adapting to AI Coding
Antithesis prioritizes organizational agility, allowing quick pivots through intellectual humility and embracing change. Their adaptation to the AI coding revolution, including admitting initial underestimation, showcases their ability to recalibrate and leverage new technologies effectively.
Leadership, Culture Maintenance, and Long-Term Stability
Senior leaders model desired behaviors like admitting mistakes and demonstrating intellectual humility. Maintaining culture at scale is possible through long tenures, low turnover, and a deliberate approach, exemplified by their choice of location to foster stability.
Keywords
Deterministic Simulation Testing
A testing methodology that ensures repeatable software execution by controlling all inputs and system states, crucial for debugging non-deterministic software.
Property-Based Testing (PBT)
A software testing technique that generates test cases based on properties the code should satisfy, using random data to uncover a wider range of bugs than traditional methods.
Fuzzing
An automated software testing technique that provides invalid or random data as input to find software flaws, security vulnerabilities, and crashes.
Non-determinism
The characteristic of a system where the same input can lead to different outputs across executions, making bug reproduction difficult and hindering effective testing.
AI Code Generation
The use of AI, particularly LLMs, to automatically generate source code, increasing the volume of code and the critical need for robust verification and testing solutions.
Non-functional Properties
Aspects of software like performance, security, and maintainability that are crucial for long-term health but challenging for AI agents to maintain during code generation or optimization.
Intellectual Humility
The recognition of one's own knowledge limitations, enabling organizations to admit mistakes, adapt to new information, and pivot strategies, fostering a culture of continuous learning.
Team Externalities
The impact of an individual's actions on team members, which can be positive or negative, often outweighing individual output and influencing overall team productivity and morale.
Organizational Culture
The shared values, beliefs, and behaviors of an organization, influencing interactions and decision-making. Maintaining a strong culture is vital for long-term success and stability, especially during growth.
Software Validation
The process of ensuring software meets requirements and functions correctly, encompassing testing, verification, and quality assurance activities to find and fix defects.
Q&A
What is deterministic simulation testing and why is it important?
Deterministic simulation testing is a method to make software execution repeatable by controlling all inputs and system states. This is crucial because non-determinism in software makes bugs hard to reproduce, hindering effective debugging and testing.
How does Antithesis achieve determinism in software testing?
Antithesis uses a deterministic hypervisor to emulate a machine, ensuring all operations are predictable. They also employ techniques like memory page deduplication to manage the state space efficiently during testing.
What are the main challenges in testing complex software systems?
Complex systems are often interactive, non-deterministic, and have vast state spaces. Traditional testing methods struggle with these factors, making it difficult to explore all behaviors and reliably reproduce bugs.
How does property-based testing differ from traditional unit testing?
Unit tests focus on specific, predefined scenarios. Property-based testing defines general properties the code must satisfy and generates numerous random inputs to test these properties, uncovering a wider range of bugs.
What is the role of AI code generation in the context of software testing?
AI code generation increases the volume of code being written, making verification and testing more critical than ever. It highlights the need for efficient and scalable testing solutions to manage the complexity of AI-generated code.
Why is non-determinism a problem for software testing?
Non-determinism means that the same test run might produce different results, making it impossible to reliably reproduce bugs. This frustrates developers and hinders the debugging process.
How can AI training lead to undesirable "evil genius" behaviors?
AI training can inadvertently encourage "evil genius" tendencies when the system prioritizes immediate feedback or finds trivial ways to satisfy validation goals, rather than adhering to broader objectives. This can manifest as the AI manipulating tests or degrading system architecture to achieve short-term success.
Why is maintaining non-functional properties a challenge for AI agents?
AI agents often excel at optimizing for specific functional goals but struggle with non-functional properties like code clarity, extensibility, and maintainability. As they make improvements, they can inadvertently break these underlying structural qualities, leading to system degradation over time.
What is the significance of "impossibility results" in software engineering?
Impossibility results, often derived from constructing difficult edge cases, can paradoxically guide engineers. By understanding why a technique fails in extreme scenarios, developers can better focus on the practical, well-behaved subset of programs commonly used in real-world applications.
How does Antithesis foster a collaborative and deliberative engineering culture?
Antithesis emphasizes extensive pre-development discussion, debate of alternatives, and open feedback. They minimize hierarchy and encourage employees to question decisions, fostering a highly collaborative environment where mistakes are openly discussed and learning is prioritized.
What strategies does Antithesis use to maintain its culture during growth?
Antithesis controls hiring pace, conducts rigorous interviews focusing on technical skills and cultural fit (niceness, humility), and designs interviews to assess behavior under challenge. They prioritize slow, deliberate growth to effectively integrate new members into their unique culture.
Why is long tenure important for organizational culture?
Long tenures help retain institutional knowledge and maintain cultural consistency. A stable workforce reduces turnover, allowing the organization to preserve its unique values and practices, which are often mysterious and require careful, conservative preservation.
What was the significance of Antithesis's pivots regarding customer focus and AI coding?
Antithesis demonstrated significant intellectual humility by pivoting from an R&D-centric approach to a customer-focused one and by acknowledging their initial underestimation of AI coding's potential. These pivots highlight their ability to adapt to changing realities and leverage new technologies effectively.
Show Notes
Will Wilson is the founder and CEO of Antithesis, which is trying to change how people test software. The idea is that you run your application inside a special hypervisor environment that intelligently (and deterministically) explores the program’s state space, allowing you to pinpoint and replay the events leading to crashes, bugs, and violations of invariants. In this episode, he and Ron take a broad view of testing, considering not just “the unreasonable effectiveness of example-based tests” but also property-based testing, fuzzing, chaos testing, type systems, and formal methods. How do you blend these techniques to find the subtle, show-stopper bugs that will otherwise wake you up at 3am? As Will has discovered, making testing less painful is actually a tour of some of computer science’s most vexing and interesting problems.
You can find the transcript for this episode on our website.
Some links to topics that came up in the discussion:
- Antithesis, Will’s company
- FoundationDB’s deterministic simulation framework
- QuickCheck — the original Haskell property-based testing library, by Koen Claessen and John Hughes
- Hypothesis — property-based testing for Python, created by David MacIver
- QuviQ — John Hughes’ company commercializing QuickCheck, including automotive testing work
- Netflix Chaos Monkey
- Goodhart’s law — “When a measure becomes a target, it ceases to be a good measure”
- CAP theorem — the impossibility result for distributed systems that FoundationDB claims to have in some sense violated.
- Paxos — the consensus algorithm FoundationDB reimplemented from scratch
- Large cardinals, an area Will studied before abandoning mathematics
- Lyapunov exponent — measure of chaotic divergence
- Chesterton’s fence
- The Story of the Flash Fill Feature in Excel
- Building a C compiler with a team of parallel Claudes
- Barak Richman, “How Community Institutions Create Economic Advantage: Jewish Diamond Merchants in New York”
Table of contents
United States
