How to use Python with Apache Kafka
Description
Can you use Apache Kafka® and Python together? What’s the current state of Python support? And what are the best options to get started? In this episode, Dave Klein joins Kris to talk about all things Kafka and Python: the libraries, the tools, and the pros & cons. He also talks about the new course he just launched to support Python programmers entering the event-streaming world.
Dave has been an active member of the Kafka community for many years and noticed that there were a lot of Kafka resources for Java but few for Python. So he decided to create a course to help people get started using Python and Kafka together.
Historically, Java has had the most documentation, and people have often missed how good the Python support is for Kafka users. Python and Kafka are an ideal fit for machine learning applications and data engineering in general. Yet there are a lot of use cases for building, streaming, and machine learning pipelines. In fact, someone conducted a survey to find out what languages were most popular in the Kafka community and Python came in second after Java. That’s how Dave got the idea to create a course for newbies.
In this course, Dave combines video lectures with code-heavy exercises to give developers a taste of what the code looks like, how to structure it, a preview of the shape of the code, and the structure of the classes and the functions so you can get hands-on practice using the library. He also covers building a producer and a consumer and using the admin client. And, of course, there is a module that covers working with the schemas supported by the Kafka library.
Dave explains that Python opens up a world of opportunity and is ripe for expansion. So if you are ready to dive in, head over to developer.confluent.io to learn more about Dave’s course.
EPISODE LINKS
- Blog: Getting Started with Python for Apache Kafka
- Course: Introduction to Apache Kafka for Python Developers
- Step-by-step guide: Building a Python client application for Kafka
- Coding in Motion
- Building and Designing Events and Event Streams with Apache Kafka
- Watch the video version of this podcast
- Kris Jenkins’ Twitter
- Streaming Audio Playlist
- Join the Confluent Community
- Learn more with Kafka tutorials, resources, and guides at Confluent Developer
- Live demo: Intro to Event-Driven Microservices with Confluent
- Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)
United States
Remember those childhood dreams where you’d build the most fantastic machine out of LEGOs, only to find out that real-life engineering is much harder? Double Cloud is here to make those dreams come true—but with data, not LEGOs. Imagine constructing an entire data pipeline in just one day. That’s right—one day. Check out https://double.cloud/services/managed-kafka/. Double Cloud’s platform is like a giant box of digital LEGOs, offering managed services like ClickHouse and Kafka, all while making sure everything fits together perfectly. Their no-code ELT tool is the magic brick that snaps everything into place, letting you sync data in real time without breaking a sweat. And maintenance? It’s like having a team of LEGO master builders on call, handling sharding, backups, and scaling, so you can focus on the fun part—innovating. Worried about security? Double Cloud has that covered too, with encryption and compliance so tight, it’s like locking your LEGOs in a vault. Curious about their