Rust in Production

What's even cooler than writing your own text editor or your own operating system? Building your own hardware from scratch with all the software written in Rust -- including firmware, the scheduler, and the hypervisor. Oxide Computer Company is one of the most admired companies in the Rust community. They are building "servers as they should be" with a focus on security and performance to serve the needs of modern on-premise data centers.In this episode, I talk to Steve Klabnik, a software engineer at Oxide and renowned Rustacean, about the advantages of building hardware and software in tandem, the benefits of using Rust for systems programming, and the state of the Rust ecosystem.

Picture this: Your organization's data infrastructure resembles a busy kitchen with too many cooks. You're juggling Kafka for messaging, Flink for processing, Spark for analytics, Airflow for orchestration, and various Lambda functions scattered about. Each tool excellent at its job, but together they've created a complex feast of integration challenges. Your data teams are spending more time managing tools than extracting value from data. InfinyOn reimagines this chaos with a radically simple approach: a unified system for data streaming that runs everywhere. Unlike traditional solutions that struggle at the edge, InfinyOn gracefully handles data streams from IoT devices to cloud servers. And instead of cobbling together different tools, developers can build complete data pipelines using their preferred languages - be it Rust, Python, or SQL - with built-in state management. At the heart of InfinyOn is Fluvio, a Rust-based data streaming platform that's fast, reliable, and easy to use.

Next to writing their own operating system, another dream shared by many developers is building their own text editor. Conrad Irwin, a software engineer at Zed, is doing just that. Zed is a fully extensible, open-source text editor written entirely in Rust. It's fast, lightweight, and comes with excellent language support out of the box.In the first episode of the third season, I sit down with Conrad to discuss Zed's mission to build a next-generation text editor and why it was necessary to rebuild the very foundation of text editing software from scratch to achieve their goals.

As we approach the finale of our second season, it's time for another recap.Could we shed some light on the current state of Rust's usage in the industry? What has changed in our perception of Rust in production since our last season?While more companies started to embrace Rust, some of the magic of Rust's early days is gone. I expect more ripple effects as the community clashes with the industry's demands.This episode takes on a more somber tone, as we peer into the massive tech debt we have accumulated as an industry. And in the dark: a faint glow, a narrow golden path that points us towards a more hopeful future.

Many devs dream of one day writing their own operating system. Ideally in their favorite language: Rust. For many of us, this dream remains just that: a dream.Jeremy Soller from System76, however, didn't just contribute kernel code for Pop!_OS, but also started his own operating system, RedoxOS, which is completely written in Rust. One might get the impression that he likes to tinker with low-level code!In this episode of Rust in Production, Jeremy talks about his journey. From getting hired as a kernel developer at Denver-based company System76 after looking at the job ad for 1 month and finally applying, to being the maintainer of not one but two operating systems, additional system tools, and the Rust-based Cosmic desktop. We'll talk about why it's hard to write correct C code even for exceptional developers like Jeremy and why Rust is so great for refactoring and sharing code across different levels of abstraction.

Rust can run everywhere, and by everywhere, we don't just mean on all operating systems, but also in all kinds of harsh environments: from the depths of the ocean to the vastness of space. Today we talk to a company that is using Rust to conquer the air. Fusion Engineering is building drone control systems for the next generation of drones.Jakub Valtar walks us through how Fusion Engineering came to use Rust as the foundation of their company. He explains why Rust is the safest choice for building drone control systems and what it takes to get into drone development.

It has become a trope by now: "Cars are computers on wheels." In modern cars, not only the infotainment system but also the engine, brakes, and steering wheel are controlled by software. Better make sure that software is safe. Alexandru Radovici is a Software Engineer at OxidOS, a company that builds a secure, open-source operating system for cars built on Rust and Tock. We talk about the challenges of certifying Rust code for the automotive industry and the new possibilities with Rust-based car software.

The idea of smart robots automating away boring household chores sounds enticing, yet these devices rarely work as advertised: they get stuck, they break down, or are security nightmares. And so it's refreshing to see a company like Matic taking a different approach by attempting to build truly smart, reliable, and privacy-respecting robots. They use Rust for 95% of their codebase, and use camera vision to navigate, vacuum, and mop floors.I sit down with Eric Seppanen, Software Engineer at Matic, to learn about vertical integration in robotics, on-device sensor processing, large Rust codebases, and why Rust is a great language for the problem space.

There are probably only a handful of open-source projects that had a bigger impact on the world than Mozilla Thunderbird. The email client has been around for over two decades and has been a staple for many users (me included). Dealing with a legacy codebase that servers millions of users is no easy feat. The team at MZLA, a subsidiary of Mozilla, has been working hard to modernize the core of Thunderbird by writing new parts in Rust. In this episode, I talk to Brendan Abolivier, a software engineer at MZLA, about the challenges of working on a legacy codebase, the new Rust-based Exchange protocol support, which is the first new protocol in Thunderbird in over a decade, and the future of Thunderbird.

Imagine you're faced with the challenge to build a system that can handle billions of recyclable items per year with the goal of being better than a human at identifying what can be recycled. Material classification is a complex problem that requires a lot of data and a lot of processing power and it is a cutting-edge field of research. Carters Schultz and his colleagues at AMP chose Rust to build the core of this system -- and it worked "shockingly well". In this interview, Carter, charismatic and witty, shares his experience of getting your hands dirty with Rust, and building a cutting-edge production-ready system, which can be now replicated across the world.

In the season premier we talk to none other than Daniel Stenberg! We focus on integrating Rust modules in curl, their benefits, ways in which Rust and Rust crates helped improve curl, but also how curl helped those crates, and where curl is used in the official Rust toolchain. Along the way we also learn about the early history of curl and Rust, which section of your car's owner's-manual you should "re"-read, some weird HTTP edge-cases, and Daniel's experience in open-source maintainership.And don't forget: have fun!

In this episode of Rust in Production, we reflect on the inaugural season, sharing insights from companies pioneering with Rust to shape the future of infrastructure. Starting with Rust's early days before the stable compiler, we chart the language's evolution and its growing adoption in production by leading companies. The episode explores why companies choose Rust, highlighting its unparalleled safety features, concurrency management, and performance that not only drives down costs but significantly enhances user experience. The narrative then shifts to the practicalities of adopting Rust, including learning resources, the importance of community support, and integrating Rust into existing codebases. We discuss the surprisingly positive experience of hiring Rust developers, thanks to the language's appealing prospects. Moreover, the episode delves into Rust's explicit approach to handling complexity, offering a steeper yet rewarding learning curve. We also address the misconception of Rust's limited adoption, revealing its widespread use across various industries and the consensus around certain Rust patterns and libraries. Despite the challenges in showcasing Rust's adoption due to the reticence of some companies to share their stories, the positive feedback from the podcast's audience underscores a keen interest in Rust's real-world applications. Concluding the season, we extend a heartfelt thanks to all guests and listeners, emphasizing the value of sharing Rust experiences and the potential for further growth in the Rust ecosystem. We share a wishlist for the Rust community, including expanded learning resources, improved tooling for production environments, and more public discussions of Rust's benefits. As the show gears up (see what we did there?) for its second season, listeners are encouraged to engage and suggest topics or guests, which could broaden our common understanding of Rust in production.

In an ever-expanding world of microservices, APIs, and devices, maintaining an overview of application states and the myriad errors that can occur is challenging. For years, Sentry has been a go-to choice for developers to monitor their applications and receive notifications about issues within their code.Traditionally, Sentry was predominantly a Python shop, but they became one of the early adopters of Rust in their technology stack. They have been utilizing Rust for a few years now (since at least 2017), starting with sentry-cli, a command-line utility to work with Sentry, and continuing with their source map parsing pipeline, which reduced processing times from 20 seconds to less than 500 milliseconds.More recently, they have been developing two new projects in Rust: relay and symbolicator. Relay acts as a proxy for sending events to Sentry, while symbolicator is a service for handling the symbolication of stack traces. Both projects are open source and available on GitHub.Arpad Borsos (swatinem), Senior Native Platform Engineer at Sentry, sat down with me to discuss their journey with Rust and how it has enabled them to build a cutting-edge monitoring platform for large-scale applications.Our conversation covered topics such as 100x speedups, the Rust-Python interface, and the comparison between actor-based and task-based concurrency.

Have you ever wondered how computers stay in sync with the time? That is theresponsibility of the Network Time Protocol (NTP). Around since 1985, NTP is oneof the oldest protocols still in use on the internet and its referenceimplementation, ntpd, written in C, is still widely used today.That's a problem. C is a language that is not memory safe, and ntpd has had itsshare of security vulnerabilities. Here is a list of CVEs.As part of Project Pendulum, Folkert de Vries and his colleagues from TweedeGolf have been working on a Rust implementation of NTP. I sit down with Folkertto talk about the project, the challenges they faced, and the benefits of usingRust for this kind of project.Along the way, we learn about funding open source projects, the challenges ofestablishing a new implementation of a protocol, and all sorts of otherinteresting things that might or might not be related to NTP.

In this episode, we have Micah Wylde from Arroyo as our guest. Micah introduces us to Arroyo, a real-time data processing engine that simplifies stream processing for data engineers using Rust. They explain how Arroyo enables users to write SQL queries with Rust user-defined functions on top of streaming data, highlighting the advantages of real-time data processing and discussing the challenges posed by competitors like Apache Flink. Moving on, we dive into the use of Rust in Arroyo and its benefits in terms of performance and memory safety. We explore the complementarity of workflow engines and stream processors and examine Arroyo's approach to real-time SQL and its compatibility with Postgres. Micah delves into memory and lifetime concerns and elaborates on how Arroyo manages them in its storage layer. Shifting gears, we explore the use of the Tokyo framework in the Arroyo system and how it has enhanced speed and efficiency. Micah shares insights into the challenges and advantages of utilizing Rust, drawing from their experiences with Arroyo projects. Looking ahead, we discuss the future of the Rust ecosystem, addressing the current state of the Rust core and standard library, as well as the challenges of interacting with other languages using FFI or dynamically loading code. We touch upon Rust's limitations regarding a stable ABI and explore potential solutions like WebAssembly. We also touch upon industry perceptions of Rust, investor perspectives, and the hiring process for Rust engineers. The conversation takes us through the crates used in the Arroyo system, our wishlist for Rust ecosystem improvements, and the cost-conscious nature of companies that make Rust an attractive choice in the current macroeconomic environment. As we wrap up, we discuss the challenges Rust faces in competing with slower Java systems and ponder the potential for new languages to disrupt the trend in the future. We touch upon efficiency challenges in application software and the potential for a new language to emerge in this space. We delve into the increasing interest in using Rust in data science and the promising prospects of combining Rust with higher-level languages. Finally, we discuss the importance of fostering a welcoming and drama-free Rust community. I would like to thank Micah for joining us today and sharing their insights. To find more resources related to today's discussion, please refer to the show notes. Stay tuned for our next episode, and thank you for listening!

In this episode, Nicolas, a staff software engineer at Apollo GraphQL, discusses the company's use of GraphQL API technologies. Apollo GraphQL specializes in open-source libraries for both client and server-side applications, with a focus on integrating Rust into their main offerings: the Apollo router and GraphOS cloud. Nicolas explains how the Apollo router consolidates multiple microservices into a single API, efficiently routing requests to appropriate services.He delves into GraphQL's role as an effective query language for APIs, highlighting its ability to provide a comprehensive description of API data and its compatibility with existing data systems. The shift from the JavaScript-based Apollo Gateway to the Rust-based Apollo Router is a key topic, with an emphasis on the performance and safety improvements Rust brings to the table.The conversation covers the use of Rust for the router and GraphQL parser, alongside Kotlin for the management plane and GraphQL for the API. Challenges in stability and reliability are discussed, as well as Rust's advantages in safety and type system consistency. Nicolas shares insights on Async Rust, particularly its impact on productivity and application in CLI tools like Rover.The episode also addresses learning Rust in stages, from basic language concepts to advanced internal mechanisms. It touches on functional patterns in Rust and strategies for effective dependency management. Closing the discussion, Nicolas highlights the inclusive and supportive nature of the Rust community.

In this episode, we are joined by Steven, the CTO of PubNub, a company that has developed an edge net messaging network with over a billion connected devices. Steven explains that while message buses like Kafka or RabbitMQ are suitable for smaller scales, PubNub focuses on the challenges of connecting mobile devices and laptops at a web scale. They aim to provide instant signal delivery at a massive scale, prioritizing low latency for a seamless user experience. To achieve this, PubNub has architected their system to be globally distributed, running on AWS with Kubernetes clusters spread across all of Amazon's zones. They utilize GeoDNS to ensure users connect to the closest region for the lowest latency possible. Steven goes on to discuss the challenges they faced in building their system, particularly in terms of memory management and cleanup. They had to deal with issues such as segmentation faults and memory leaks, which caused runtime problems, outages, and potential data loss. PubNub had to invest in additional memory to compensate for these leaks and spend time finding and fixing the problems. While C was efficient, it came with significant engineering costs. As a solution, PubNub started adopting Rust, which helped alleviate some of these challenges. When they replaced a service with Rust, they observed a 5x improvement in memory and performance. Steven also talks about choosing programming languages for their platform and the difficulties in finding and retaining C experts. They didn't consider Java due to its perceived academic nature, and Go didn't make the list of options at the time. However, they now have services in production written in Go, though rewriting part of their PubSub bus in Go performed poorly compared to their existing C system. Despite this, they are favoring Rust as their language of choice for new services, citing its popularity and impressive results. The conversation delves into performance considerations with Python and the use of PyPy as a just-in-time compiler for optimization. While PyPy improved performance, it also required a lot of memory, which could be expensive. On the other hand, Rust provided a significant boost in both memory and performance, making it a favorable choice for PubNub. They also discuss provisioning, taking into account budget and aiming to be as close to what they need as possible. Kubernetes and auto scaling with HPAs (Horizontal Pod Autoscaling) are used to dynamically adjust resources based on usage. Integrating new services into PubNub's infrastructure involves both API-based communication and event-driven approaches. They use frameworks like Axiom for API-based communication and leverage Kafka with Protobuf for event sourcing. JSON is also utilized in some cases. Steven explains that they chose Protobuf for high-traffic topics and where stability is crucial. While the primary API for customers is JSON-based, PubNub recognizes the superior performance of Protobuf and utilizes it for certain cases, especially for shrinking down large character strings like booleans. They also discuss the advantages of compression enabled with Protobuf. The team reflects on the philosophy behind exploring Rust's potential for profit and its use in infrastructure and devices like IoT. Rust's optimization for smaller binaries is highlighted, and PubNub sees it as their top choice for reliability and performance. They mention developing a Rust SDK for customers using IoT devices. The open-source nature of Rust and its ability to integrate into projects and develop open standards are also praised. While acknowledging downsides like potential instabilities and longer compilation time, they remain impressed with Rust's capabilities. The conversation covers stability and safety in Rust, with the speaker expressing confidence in the compiler's ability to handle alpha software and packages. Relying on native primitives for concurrency in Rust adds to the speaker's confidence in the compiler's safety. The Rust ecosystem is seen as providing adequate coverage, although packages like libRDKafka, which are pre-1.0, can be challenging to set up or deploy. The speaker emphasizes simplicity in code and avoiding excessive abstractions, although they acknowledge the benefits of features like generics and traits in Rust. They suggest resources like a book by David McCloyd that focuses on learning Rust without overwhelming complexity. Expanding on knowledge sharing within the team, Stephen discusses how Rust advocates within the team have encouraged its use and the possibilities it holds for AI infrastructure platforms. They believe Rust could improve performance and reduce latency, particularly for CPU tasks in AI. They mention the adoption of Rust in the data science field, such as its use in the Parquet data format. The importance of tooling improvements, setting strict standards, and eliminating unsafe code is highlighted. The speaker expresses the desire for a linter that enforces a simplified version of Rust to enhance code readability, maintainability, and testability. They discuss the balance between functional and object-oriented programming in Rust, suggesting object-oriented programming for larger-scale code structure and functional paradigms within functions. Onboarding Rust engineers is also addressed, considering whether to prioritize candidates with prior Rust experience or train individuals skilled in another language on the job. Recognizing the shortage of Rust engineers, Stephen encourages those interested in Rust to pursue a career at PubNub, pointing to resources like their website and LinkedIn page for tutorials and videos. They emphasize the importance of latency in their edge messaging technology and invite users to try it out.

For our very first episode, we welcome a special guest, Paul Dix, the CTO of InfluxData.He starts by giving us an overview of InfluxDB, an open source time series database used by developers to track server and application data. He takes us back to the early days of InfluxDB and explains how it came into existence, starting with the challenges they faced with their initial SaaS application and how they made the decision to repurpose their infrastructure and create this open source database. Paul also sheds light on the popularity of the programming language Go, which had a significant influence on their decision to use it for their project.He takes us through the journey of InfluxDB's development and the improvements that have been made over the years. He emphasizes the enhancements made in versions 0.11 and 1.0 to improve performance and query capabilities. Moreover, he shares their decision to explore using Rust for certain parts of the project and the positive impact it has had. Moving forward, the conversation delves into the challenges of managing high volumes of data in time series databases.Paul talks about the solutions they implemented, such as using BoltDB and developing the time-structured merge tree storage engine. We then dive into the decision to rewrite InfluxDB in Rust and the benefits it offers. He explains the improved performance, concurrency, and error handling that Rust brings to the table. Paul goes on to discuss the development process and how the engineering team has embraced Rust across their projects.As the conversation progresses, we touch on the performance improvements in InfluxDB 3 and the future plans for the database. Paul shares their vision of incorporating additional features and integrating with other tools and languages. He also mentions InfluxDB's involvement in open-source projects like Apache Aero Rust and Data Fusion, highlighting their ambition to extend beyond metric data. Paul concludes the conversation by discussing the standards and libraries in analytics, the role of Apache Iceberg, and the collaboration among data and analytics companies. He provides advice for getting started with Rust and InfluxDB, urging listeners to engage in hands-on projects and learn from books and online documentation.Thank you, Paul, for sharing your insights and expertise.

This is Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. We follow their journey in pursuit of more reliable and efficient software as they solve some of the most challenging technical problems in the world.I'm your host, Matthias Endler, and I'm a software engineer at corrode, a consultancy that helps companies make the most of Rust. I've been using Rust since 2015, have been a member of the Rust Cologne meetup since Rust 1.0 and ran a YouTube channel called "Hello Rust".There are plenty of great podcasts about Rust, but I felt that there was a missing piece. I wanted to hear more about how companies who use Rust in production. What are the challenges they face? How do they overcome them? What are the benefits of using Rust? How does the company find and hire Rust developers? And what advice would they give to other companies who want to use Rust.I sit down with decision-makers from companies that bet big on Rust and ask them in-depth questions about what they learned along the way. New episodes air every two weeks on Thursdays at 4pm UTC. If you don't want to miss out, please subscribe on Spotify, Apple Podcasts, or wherever you listen to podcasts. This helps other people find the show and supports our work.If you want to learn more about the show, please visit corrode.dev/podcast. Stay tuned for the first episode, where I talk to Paul Dix from InfluxData about how they use Rust in the latest version of InfluxDB.