DiscoverStreaming Audio: A Confluent podcast about Apache Kafka
Streaming Audio: A Confluent podcast about Apache Kafka
Claim Ownership

Streaming Audio: A Confluent podcast about Apache Kafka

Author: Confluent, original creators of Apache Kafka®

Subscribed: 261Played: 10,622
Share

Description

Streaming Audio is a podcast from Confluent, the team that built Apache Kafka®️. Host Tim Berglund (Senior Director of Developer Experience, Confluent) and guests unpack a variety of topics surrounding Apache Kafka, event stream processing and real-time data. The show also features the segment "Ask Confluent," in which Gwen Shapira (Engineering Manager, Confluent) and guests respond to a handful of questions and comments about the Confluent and Kafka ecosystems—from Kafka connectors, to distributed systems, data integration, Kafka deployment, and managed Apache Kafka as a service—on Twitter, YouTube, and elsewhere. Apache®️, Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
130 Episodes
Reverse
Jason Bell (Apache Kafka® DevOps Engineer, digitalis.io, and Author of “Machine Learning: Hands-On for Developers and Technical Professionals” ) delves into his 32-year journey as a DevOps engineer and how he discovered Apache Kafka. He began his voyage in hardware technology before switching over to software development. From there, he got involved in event streaming in the early 2000s where his love for Kafka started. His first Kafka project involved monitoring Kafka clusters for flight search data, and he's been making magic ever since!Jason first learned about the power of the event streaming during Michael Noll’s talk on the streaming API in 2015. It turned out that Michael had written off 80% of Jason’s streaming API jobs with a single talk. As a Kafka DevOps engineer today, Jason works with on-prem clusters and faces challenges like instant replicas going down and bringing other developers who are new to Kafka up to speed so that they can eventually adopt it and begin building out APIs for Kafka. He shares some tips that have helped him overcome these challenges and bring success to the team.EPISODE LINKSMachine Learning: Hands-On for Developers and Technical Professionals by Jason Bell Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Multi-tenancy has been quite the topic within the Apache Kafka® community. Anna Povzner, an engineer on the Confluent team, spends most of her time working on multi-tenancy in Kafka in Confluent Cloud.Anna kicks off the conversation with Tim Berglund (Senior Director of Developer Experience, Confluent) by explaining what multi-tenancy is, why it is worthy to be desired, and advantages over single-tenant architecture. By putting more applications and use cases on the same Kafka cluster instead of having a separate Kafka cluster for each individual application and use case, multi-tenancy helps minimize the costs of physical machines and also maintenance.She then switches gears to discuss quotas in Kafka. Quotas are essentially limits—you must set quotas for every tenant (or set up defaults) in Kafka. Anna says it’s always best to start with bandwidth quotas because they’re better understood.Stick around until the end as Anna gives us a sneak peek on what’s ahead for multi-tenant Kafka, including KIP-612, the addition of the connection rate quota, which will help protect brokers.EPISODE LINKSSharing is Caring: Toward Creating Self-Tuning Multi-Tenant Kafka (Anna Povzner, Confluent)Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Roger Hoover, one of the first engineers to work on Confluent Cloud, joins Tim Berglund (Senior Director of Developer Experience, Confluent) to chat about the evolution of Confluent Cloud, all the stages that it’s been through, and the lessons he’s learned on the way. He talks through the days before Confluent Platform was created, and how he contributed to Apache Kafka® to run it on OpenStack (the feature used to separate advertised hostnames from the internal hostnames).The Confluent Cloud control plane is now run in over 40 regions. Under the covers, Roger and his team are managing tens of thousands of resources at the cloud provider layer. This means creating VPCs, VMs, volumes, and DNS records, to manage software artifacts, like what version of Kafka is running and user management. Confluent Cloud is a complex application and distributed system spread across the entire world, but Roger reveals how it's done.EPISODE LINKSBuilding Confluent Cloud – Here’s What We’ve Learned Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Do you ever feel like you’re short on time? Well, good news! Confluent Software Engineer Matthias J. Sax is back to discuss how event streaming has changed the game, making time management more simple yet efficient. Matthias explains what watermarking is, the reasons behind why Kafka Streams doesn’t use them, and an alternative approach to watermarking informally called the “slack time approach.” Later, Matthias discusses how you can compare “stream time,” which is the maximum timestamp observed, to the watermark approach as a high-time watermark. Stick around for the end of the episode, where Matthias reveals other new approaches in the pipeline. Learn how to get the most out of your time on today’s episode of Streaming Audio!EPISODE LINKSKafka Summit talk: The Flux Capacitor of Kafka Streams and ksqlDBWatermarks, Tables, Event Time, and the Dataflow ModelKafka Streams’ Take on Watermarks and TriggersJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
What's it like being a distributed systems engineer? Apurva Mehta (Engineering Leader, Confluent) explains what attracted him to Apache Kafka®, the challenges and uniqueness of distributed systems, and how to excel in this industry. He dives into the complex math behind the temporal logic of actions (TLA) and shares about his experiences working at Yahoo and Linkedin, which have prepared him to be where he is today.Apurva also shares what he looks for when hiring someone to join his team. When you're working on a system like Kafka and Kafka Streams, really understanding what your machine is doing, where the bottlenecks are, and how to design improvements to address inefficiencies is critical. EPISODE LINKSJason Gufstason discusses TLA validation (and distributed systems engineering in general) MIT Courseware on Distributed Systems Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
It’s Halloween again, which means Anna McDonald (Staff Technical Account Manager, Confluent) is back for another spooktacular episode of Streaming Audio.In this episode, Anna shares six of the most spine-chilling, hair-raising  Apache Kafka® JIRAs from the past year. Her job is to help hunt down problems like these and dig up skeletons like: Early death causes epoch time travelAttack of the clonesMissing snapshot file leads to madnessShrink inWriteLock time to avoid maiming cluster performanceOlder groups are forced to flatlineGhost segment haunts for eternity If JIRAs are undead monsters, Anna is practically a zombie slayer. Get a haunting taste of the horrors that she's battled with as she shares about each of these Kafka updates. Keep calm and scream on in today’s special episode of Streaming Audio!EPISODE LINKSKafka: A Modern Distributed SystemFrom Eager to Smarter in Apache Kafka Consumer Rebalances by Sophie Blee-GoldmanThe Magical Rebalance Protocol of Apache Kafka (Strange Loop)Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
It’s the first work-from-home episode of Ask Confluent, where Gwen Shapira (Core Kafka Engineering Leader, Confluent) virtually sits down with Apache Kafka® expert Anna McDonald (Staff Technical Account Manager, Confluent) to answer questions from Twitter. Find out Anna’s favorite Kafka Improvement Proposal (KIP), which  will start to use racially neutral terms in the Kafka community and in our code base, as well as answers to the following questions: If you could pick any one KIP from the backlog that hasn't yet been implemented and have it immediately available, which one would you pick?Are we able to arrive at any formula for identifying the consumer/producer throughput rate in Kafka with the given hardware specifications (CPU, RAM, network, and disk)? Does incremental cooperative rebalancing also work for general Kafka consumers in addition to Kafka Connect rebalancing?They also answer how to determine throughput and achieve your desired SLA by using partitions. EPISODE LINKSWatch Ask Confluent #18: The Toughest Questions ft. Anna McDonaldFrom Eager to Smarter in Apache Kafka Consumer RebalancesStreaming Heterogeneous Databases with Kafka Connect – The Easy WayKeynote: Tim Berglund, Confluent | Closing Keynote Presentation | Kafka Summit 2020Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Wouldn’t it be awesome if there was a language as elegant as Spring Boot is as a framework? In this episode of Streaming Audio, Tim Berglund talks with Josh Long, Spring developer advocate at VMware about Kotlin, about the productivity-focused language from our friends at JetBrains, and how it works with Spring Boot to make the experience leaner, cleaner, and easy to use.Josh shares how the Spring and Kotlin teams have worked hard to make sure that Kotlin and Spring Boot are a first-class experience for all developers trying to get to production faster and safer. They also talk about the issues that arise when wrapping one set of APIs with another, as often arises in the Spring Framework: when APIs should leak, when they should not, and how not to try to be a better Kafka Streams when the original is working well enough. EPISODE LINKSJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Neil Buesing, an Apache Kafka® community stalwart at Object Partners, spends his days building things out of Kafka and helping others do the same. Today, he discusses the concept of a CoE (center of excellence), and how a CoE is integral to attain and sustain world-class performance, business value, and success in a business. Neil talks us through how to make a CoE successful, the importance of event streaming, how to better understand streaming technologies, and how to best utilize CoE for your needs. This includes evangelizing Kafka, building a Proof of Value (PoV) with team members, defining deliverables as part of that CoE, and understanding how to implement Kafka into your organization. EPISODE LINKSEoS in Kafka: Listen up, I will only say this once! by Jason Gustafson The Magical Rebalance Protocol of Apache Kafka by Gwen Shapira Chair-throwing meme that was discussed at end of episode Apache Kafka and Confluent Platform Reference ArchitectureBenchmark Your Dedicated Apache Kafka Cluster on Confluent CloudOptimizing Your Apache Kafka DeploymentCluster sizingJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Ever wonder what it's like to intern at a place like Confluent? How about working with Kafka Streams and creating your own KIP? Well, that's exactly what we discuss on today's episode with Leah Thomas. Leah Thomas, who first interned as a recruiter for Confluent, quickly realized that she was enamored with the problem solving the engineering team was doing, especially with Kafka Streams. The next time she joined Confluent's intern program, she worked on the Streams team and helped bring KIP-450 to life. With KIP-450, Leah started learning Apache Kafka® from the inside out and how to better address the user experience. She discusses her experience with getting a KIP approved with the Apache Software Foundation and how she dove into solving the problem of hopping windows with sliding windows instead.EPISODE LINKSRange: How Generalists Triumph in a Specialized WorldConfluent CareersJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
The feature-rich release of Confluent Platform 6.0, based on Apache Kafka® 2.6, introduces Tiered Storage, Self-Balancing Clusters, ksqlDB 0.10, Admin REST APIs, and Cluster Linking in preview. These features enhance the platform with greater elasticity, improved cost effectiveness, infinite data retention, and global availability so that you can simplify management operations, reduce the cost of adopting Kafka, and focus on building event streaming applications.EPISODE LINKSConfluent Platform 6.0 Release NotesIntroducing Confluent Platform 6.0Download Confluent Platform 6.0Watch the video version of this podcastJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Bobby Calderwood (Founder, Evident Systems) discusses event streaming, event modeling, and event-driven architecture. He describes the emerging visual language and process, how to effectively understand and teach what events are, and some of Bobby's own use cases in the field with oNote, Evident System’s new SaaS platform for event modeling. Finally, Bobby emphasizes the power of empowering and informing the community on how best to integrate event streaming with the outside world.EPISODE LINKSBuilding Information Systems Using Event Modeling Real-Time Payments with Clojure and Apache Kafka ft. Bobby CalderwoodEvent modeling leaders Adam Dymitruk and Greg YoungGood Enough Software is by Definition Good Enough written by Greg YoungoNoteEvent modelingJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Did you know that a team of 900 developers at Wix is using Apache Kafka® to maintain 1,500 microservices? Tim Berglund sits down with Natan Silnitsky (Backend Infrastructure Engineer, Wix) to talk all about how Wix benefits from using an event streaming platform. Wix (the website that’s made for building websites) is designing a platform that gives people the freedom to create, manage, and develop their web presence exactly the way they want as they look to move from synchronous to asynchronous messaging. In this episode, Natan and Tim talk through some of the vital lessons learned at Wix through their use of Kafka, including common infrastructure, at-least-once processing, message queuing, and monitoring. Finally, Natan gives Tim a brief overview of the open source project Greyhound and how it's being used at Wix. EPISODE LINKSgithub.com/wix/greyhoundJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
This year, Confluent turns six! In honor of this milestone, we are taking a very special moment to celebrate with Gwen Shapira by highlighting the top six things everyone should know about Apache Kafka®:Clients have metricsBug fix releases/Kafka Improvement Proposals (KIPs)Idempotent producers and how they workKafka Connect is part of Kafka and Single Message Transforms (SMTs) are worth not missing out onCooperative rebalancing Generating sequence numbers and how Kafka changes the way you thinkListen as Tim and Gwen talk through the importance of Kafka Connect, cooperative rebalancing protocols, and the promise (and warning) that your data architecture will never be the same. As Gwen puts it, “Kafka gives you the options, but it's up to you how you use it.”EPISODE LINKSKIP-415: Incremental Cooperative Rebalancing in Kafka ConnectWhy Kafka Connect? ft. Robin Moffatt Confluent Hub Creativity IncFifth Discipline Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
With the explosion of real-time data, Apache Kafka and event stream processing (ESP) have grown in proliferation, with event streaming technology becoming the de facto technology transforming businesses across numerous verticals. Gwen Shapira (Engineering Leader, Confluent), Ben Stopford (Senior Director, OCTO, Confluent), and Michael Noll (Principal Technologist, Confluent) meet up to talk all about their last five years at Confluent and the changes they’ve seen in event streaming. They discuss what they were doing with Apache Kafka® before they arrived at Confluent, challenges in event streaming challenges that have arisen, and their favorite use cases. They then talk through what they think the Kafka community is undervaluing and where they think event streaming will be in the next five years. EPISODE LINKSTim’s Budapest Drone Footage Rolling Kafka Upgrades and Confluent Cloud ft. Gwen ShapiraDistributed Systems Engineering with Apache Kafka ft. Gwen ShapiraImproving Fairness Through Connection Throttling in the Cloud with KIP-402 ft. Gwen ShapiraAsk ConfluentApache Kafka Fundamentals: The Concept of Streams and Tables ft. Michael NollBen Stopford on Microservices and Event StreamingThe Portable Wonder Synthesizer Children's Hospital of Atlanta: Helping Healthcare with Apache Kafka and KSQL ft. Ramesh SringeriJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Jay Smith helps Google Cloud users modernize their applications with serverless eventing. This helps them focus on their code instead of managing infrastructure, as well as ultra-fast deployments and reduced server costs. On today’s show, he discusses the definition of serverless, serverless eventing, data-driven vs. event-driven architecture, sources and sinks, and hybrid cloud with on-prem components. Finally, Jay shares how he sees application architecture changing in the future and where Apache Kafka® fits in.EPISODE LINKSQuine ProgramsGet Started with QwiklabsKubernetes PodcastsJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Multi-Region Clusters improve high availability in Apache Kafka®, ensure cluster replication across multiple zones, and help with disaster recovery. Making sure users are successful in every area of their Kafka deployment, be it operations or application development for specific use cases, is what Anna McDonald (Team Lead Customer Success Technical Architect) and Mitch Henderson (Principal Customer Success Technical Architect) are passionate about here at Confluent.In this episode, they share common challenges that users often run into with Multi-Region Clusters, uses cases for them, and what to keep in mind when considering replication. Anna and Mitch also discuss consuming from followers, auto client failover, and offset issues to be aware of.EPISODE LINKSKafka Screams: The Scariest JIRAs and How to Survive Them ft. Anna McDonaldDeploying Confluent Platform, from Zero to Hero ft. Mitch HendersonJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
All Confluent developer advocates...assemble! COVID-19 has changed the face of meetings and events, halting all in-person gatherings and forcing companies to adapt on the fly. In today's episode of Streaming Audio, the developer advocates come together to discuss how their jobs have changed during the worldwide pandemic. Less than a year ago, this group was constantly on the road or in a plane on their way to present something new about Apache Kafka and event streaming, so how has the current climate affected their work? The group talks about Zoom fatigue, online presenting, online conferences/meetups, and of course, Kafka Summit 2020. EPISODE LINKSGrowing the Event Streaming Community During COVID-19 ft. Ale MurrayRegister for Kafka Summit 2020Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Apache Kafka® 2.6 is out! This release includes progress toward removing ZooKeeper dependency, adding client quota APIs to the admin client, and exposing disk read and write metrics, and support for Java 14. In addition, there are improvements to Kafka Connect, such as allowing source connectors to set topic-specific settings for new topics and expanding Connect worker internal topic settings. Kafka 2.6 also augments metrics for Kafka Streams and adds emit-on-change support for Kafka Streams, as well as other updates. EPISODE LINKSWatch the video version of this podcastRead about what's new in Apache Kafka 2.6Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Viktor Gamov (Developer Advocate, Confluent) returns to Streaming Audio to explain the magic of ksqlDB, ideal testing environments for ksqlDB, and the ksqlDB test runner. For those who are just starting to explore the interface, Viktor provides some tips and best practices for what to look out for too. He also talks about the future of ksqlDB, the future of integration testing, and his favorite new feature among recent upgrades.EPISODE LINKSStreaming Audio episodes on ksqlDBWatch #LiveStreams with Viktor Gamov KsqlServerContainerTest.javaI Don't Always Test My StreamsJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
loading
Comments 
Download from Google Play
Download from App Store