DiscoverStreaming Audio: A Confluent podcast about Apache Kafka
Streaming Audio: A Confluent podcast about Apache Kafka
Claim Ownership

Streaming Audio: A Confluent podcast about Apache Kafka

Author: Confluent, original creators of Apache Kafka®

Subscribed: 241Played: 9,007
Share

Description

Streaming Audio is a podcast from Confluent, the team that built Apache Kafka®️. Host Tim Berglund (Senior Director of Developer Experience, Confluent) and guests unpack a variety of topics surrounding Apache Kafka, event stream processing and real-time data. The show also features the segment "Ask Confluent," in which Gwen Shapira (Engineering Manager, Confluent) and guests respond to a handful of questions and comments about the Confluent and Kafka ecosystems—from Kafka connectors, to distributed systems, data integration, Kafka deployment, and managed Apache Kafka as a service—on Twitter, YouTube, and elsewhere. Apache®️, Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
112 Episodes
Reverse
Apache Kafka® 2.6 is out! This release includes progress toward removing ZooKeeper dependency, adding client quota APIs to the admin client, and exposing disk read and write metrics, and support for Java 14. In addition, there are improvements to Kafka Connect, such as allowing source connectors to set topic-specific settings for new topics and expanding Connect worker internal topic settings. Kafka 2.6 also augments metrics for Kafka Streams and adds emit-on-change support for Kafka Streams, as well as other updates. EPISODE LINKSWatch the video version of this podcastRead about what's new in Apache Kafka 2.6Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Viktor Gamov (Developer Advocate, Confluent) returns to Streaming Audio to explain the magic of ksqlDB, ideal testing environments for ksqlDB, and the ksqlDB test runner. For those who are just starting to explore the interface, Viktor provides some tips and best practices for what to look out for too. He also talks about the future of ksqlDB, the future of integration testing, and his favorite new feature among recent upgrades.EPISODE LINKSStreaming Audio episodes on ksqlDBWatch #LiveStreams with Viktor Gamov KsqlServerContainerTest.javaI Don't Always Test My StreamsJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
As developers, we are good at envisioning the future state of any given system we want to build, but are we as good at telling the business how those changes positively impact the bottom line? Lyndon Hedderly (Team Lead, Business Value Consulting, Confluent) describes his approach to business value, how to justify a new technology that you’re introducing to your company, and tips on adopting new technologies and processes effectively. As Lyndon walks through each part of the business value framework: (1) baseline, (2) target state, (3) quantified benefits, (4) unquantified benefits, and (5) proof points, you’ll learn about cost effectiveness with Confluent Cloud, how to measure ROI vs. TCO, and a retail example from a customer that details their implementation of an event streaming platform.EPISODE LINKSMeasuring the Cost Effectiveness of Confluent Cloud Measuring TCO: Apache Kafka vs. Confluent Cloud’s Managed Service Get a Free TCO AssessmentJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
Inventory management systems are crucial for reducing real-time inventory data drift, improving customer experience, and minimizing out-of-stock events. Apache Kafka®’s real-time data technology provides seamless inventory tracking at scale, saving billions of dollars in the supply chain, making modernized data architectures more important to retailers now more than ever.  In this episode, we’ll discuss how Apache Kafka allows the implementation of stateful event streaming architectures on a cloud-native platform for application and architecture modernization. Sina Sojoodi (Global CTO, Data and Architecture, VMware) and Rohit Kelapure (Principal Advisor, VMware) will discuss data modeling, as well as the architecture design needed to achieve data consistency and correctness while handling the scale and resilience needs of a major retailer in near real time. The implemented solution utilizes Spring Boot, Kafka Streams, and Apache Cassandra, and they explain the process of using several services to write to Cassandra instead of trying to use Kafka as a distributed log for enforcing consistency.  EPISODE LINKSHow to Run Kafka Streams on Kubernetes ft. Viktor GamovMachine Learning with Kafka Streams, Kafka Connect, and ksqlDB ft. Kai WaehnerUnderstand What’s Flying Above You with Kafka Streams ft. Neil BuesingJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
Apache Kafka® Committer and PMC member Matthias J. Sax explains fault tolerance, high-availability stream processing, and how it’s done in Kafka Streams. He discusses the differences between changelogging vs. checkpointing and the complexities checkpointing introduces. From there, Matthias explains what hot standbys are and how they are used in Kafka Streams, why Kafka Streams doesn’t do watermarking, and finally, why Kafka Streams is a library and not infrastructure. EPISODE LINKSAsk Confluent #7: Kafka Consumers and Streams Failover Explained ft. Matthias SaxAsk Confluent #8: Guozhang Wang on Kafka Streams Standby TasksHow to Run Kafka Streams on Kubernetes ft. Viktor GamovKafka Streams Interactive Queries Go Prime TimeHighly Available, Fault-Tolerant Pull Queries in ksqlDBKIP-535: Allow state stores to serve stale reads during rebalanceKIP-562: Allow fetching a key from a single partition rather than iterating over all the stores on an instanceKIP-441: Smooth Scaling Out for Kafka Streams Skip to end of metadataJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
Real-time stock trades, GPS location, and website click tracking are just a few industries that heavily rely on Apache Kafka®'s real-time messaging and data delivery functions. As such, Kafka's latency is incredibly important.Anna Povzner (Software Engineer, Confluent) gives you the breakdown and everything you need to know when it comes to measuring latency. The five components of latency are produce time, publish time, commit time, catch-up time, and fetch time. When consumer pulling adds to latency, Anna shares some best practices to keep in mind for how to think about partitioning in conjunction with latency. She also discusses client configuration in the cloud, interesting problems she's helped solve for customers, and her top two tips for debugging latency. EPISODE LINKS99th Percentile Latency at Scale with Apache KafkaBenchmark Your Dedicated Apache Kafka Cluster on Confluent CloudDistributed Systems Engineering with Apache Kafka ft. Gwen ShapiraJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
What started out as a consulting company, Camunda eventually turned into a developer-friendly, open source vendor that now focuses on workflow automation. Bernd Ruecker, a co-founder and the chief technologist at Camunda, talks through the company's journey, how he ended up in open source, and all things automation, including how it differs from business process management and the issue of diagrams. Bernd also dives into dead letter topics in Apache Kafka®, software interacting with software, orchestration tension, and best practices for approaching challenges that pop up along the way. This episode will take you through a thorough introduction of Camunda Cloud, a cloud-native workflow engine, as well as Camunda’s Kafka connector. EPISODE LINKSJay Kreps, Confluent | Kafka Summit SF 2019 Keynote ft. Dev Tagare, Lyftzeebe.ioJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
We've all been affected by COVID-19 in one way or another, resulting in big changes in workplace functionality, productivity, and even our relationships within the Apache Kafka® and Confluent communities as meetings and events have needed to turn virtual. Ale Murray (Global Community Manager, Confluent) shares interesting trends, changes in community metrics, and what we’ve done to adapt as a response. Ale also explains what makes a comprehensive community program and the value of community meetups in light of the pandemic. Despite how much we miss in-person interactions, by digitizing events and focusing on the community, we saw great growth in attendance and engagement across our Slack community, online hackathons, MVP program, and online meetups over the last couple of months, proving that nothing can stop this amazing community from thriving.EPISODE LINKSGet involved with the Confluent CommunityJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage*
Author Sam Newman catches up with Tim Berglund (Senior Director of Developer Advocacy, Confluent) in the virtual studio on what microservices are, how they work, the drawbacks of microservices, what splitting the monolith looks like, and patterns to look for. The pair talk through Sam's book “Monolith to Microservices” chapter by chapter, looking at key components of microservices in more detail. Sam also walks through database decomposition, integrating with new technology, and performing joins in event streaming architecture. Lastly, Sam shares what he’s excited for in the future, which includes “Monolith to Microservices Volume II.”EPISODE LINKSMonolith to MicroservicesJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperUse 60PDCAST to get an additional $60 of free Confluent Cloud usage* 
Tim Baldridge (Senior Software Engineer, Cisco) joins us on Streaming Audio to talk about event streaming, stream processing use cases, and µKanren. First, Tim shares about his work at Cisco related to intaking viruses, the backend, and finding new ways to process data. Later, Tim talks about interesting bank and airline use cases, as well as his time at Walmart, taking a closer look at specific retail use cases and the product that Walmart used to process data streams. If you’re curious about what µKanren is, how it relates to relational programming, the complex math that goes into the workflow of µKanren, and how Apache Kafka® holds up to all other event streaming platforms, Tim also dives into that too. EPISODE LINKSµKanren: A Minimal Functional Core for Relational ProgrammingIt's Actors All The Way Down  Den of Clojure Build Your Own Logic EngineJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent Developer
Confluent Platform 5.5 introduces long-awaited JSON Schema and Protobuf support in Confluent Schema Registry and across other platform components. Support for Protobuf and JSON Schema in Schema Registry provides the same assurances of data compatibility and consistency we already had with Avro, while opening up Kafka to more businesses, applications, and use cases that are built upon those data serialization formats. Tushar Thole (Engineering Leader, Confluent) and David Araujo (Product Manager, Confluent) share about these new improvements to Confluent Schema Registry, the differences between Apache Avro™, Protobuf, and JSON Schemas, how to treat optional fields, some of the arguments between Avro and Protobuf, and why it took some time for Schema Registry to support JSON Schemas and Protobuf.Later, they talk about custom plugins, adding another layer of safety in Confluent Platform 5.5, and their vision for data governance.EPISODE LINKSIntroducing Confluent Platform 5.5Confluent Platform Now Supports Protobuf, JSON Schema, and Custom FormatsDownload Confluent PlatformGetting Started with Protobuf in Confluent CloudRead articles by Robert Yokota Schema Validation with Confluent Platform 5.4 Playing Chess with Confluent Schema RegistryJSON Schema specsSend feedback to datagovernance@confluent.ioFully managed Apache Kafka as a service! Try free.Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent Developer
Apache Kafka® is a powerful toolset for microservice architectures. In this podcast, we’ll cover how Boden, an online retail company that specializes in high-end fashion linked to the royal family, used streaming microservices to modernize their business. Matt Simpson (Solutions Architect, Boden) shares a real life use case showing how Kafka has helped Boden digitize their business, transitioning from catalogs to online sales, tracking stock, and identifying buying patterns. Matt also shares about what he's learned through using Kafka as well as the challenges of being a product master. And lastly, what is Matt excited for for the future of Boden? Find out in this episode!EPISODE LINKSCheck out BodenETL and Event Streaming Explained ft. Stewart BrysonConnecting Snowflake and Apache Kafka ft. Isaac KunenInstagram for Kensington PalaceJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent Developer
Isaac Kunen (Senior Product Manager, Snowflake) and Tim Berglund (Senior Director of Developer Advocacy, Confluent) practice social distancing by meeting up in the virtual studio to discuss all things Apache Kafka® and Kafka Connect at Snowflake. Isaac shares what Snowflake is, what it accomplishes, and his experience with developing connectors. The pair discuss the Snowflake Kafka Connector and some of the unique challenges and adaptations it has had to undergo, as well as the interesting history behind the connector. In addition, Isaac talks about how they’re taking on event streaming at Snowflake by implementing the Kafka connector and what he hopes to see in the future with Kafka releases. EPISODE LINKSDownload the Snowflake Kafka ConnectorPaving a Data Highway with Kafka Connect ft. Liz BennettMaking Apache Kafka Connectors for the Cloud ft. Magesh NandakumarMachine Learning with Kafka Streams, Kafka Connect, and ksqlDB ft. Kai WaehnerConnecting to Apache Kafka with Neo4jContributing to Open Source with the Kafka Connect MongoDB Sink ft. Hans-Peter GrahslConnecting Apache Cassandra to Apache Kafka with Jeff Carpenter from DataStaxWhy Kafka Connect? ft. Robin MoffattJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent Developer
Happy 100th episode of Streaming Audio! Thank you to everyone who has listened, subscribed, left a review, and mostly, for sharing our passion for event streaming. We can't wait for the next 100! To celebrate, Ben Stopford (Senior Director of the Office of the CTO, Confluent) hosts an AMA (ask me anything) with Tim, covering 62 questions in total—from his career, his time at Confluent, Marvel vs. DC, and what he looks for in a new hire, to how to nail your next conference talk. We hope you enjoy this special 100th episode of Streaming Audio: a podcast about Apache Kafka®, Confluent, and the cloud.EPISODE LINKSThe Song of the Strange AsceticAvoiding Lock-InBlogs by Ben Stopford Join the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent Developer
Kelsey Hightower was already an advocate, just like all other developers, long before joining Google officially as a developer advocate and Kubernetes expert. Gaining trust in your product, process, and the way you develop code requires the ability to explain those things well. Kelsey reflects on the journey that brought him to where he is today and how Kubernetes has evolved over the years too, including what makes Kubernetes so successful. But Tim is not the only one with questions. Kelsey asks a few of his own: does Apache Kafka® want to be a database? Does Kafka want to be a system of record? Is there overlap between Kubernetes and Kafka? Can you run Kafka on Kubernetes?EPISODE LINKSKubernetes the Hard WayJoin the Confluent Community SlackLearn about Kafka at Confluent Developer
If you’ve ever wondered what Apache Kafka® is, what it’s used for, or wanted to learn about Kafka architecture and all its components, buckle up! In today’s episode, Michael Noll (Principal Technologist, Confluent) and Tim Berglund (Senior Director of Developer Advocacy, Confluent) discuss a series of fundamental questions: What is Kafka? What is an event? How do we organize and store events? And what is Kafka Streams? Over the course of this episode, Michael covers an in-depth look into Kafka technology and core concepts: the process of reading from a topic, differences between tables and streams, mutability, and what ksqlDB is and what its event streaming database features accomplish. If you've ever wanted to get a better grasp on how Kafka works, this episode is for you!EPISODE LINKSStreams and Tables in Apache Kafka: A PrimerStreams and Tables in Apache Kafka: Topics, Partitions, and Storage FundamentalsStreams and Tables in Apache Kafka: Processing Fundamentals with Kafka Streams and ksqlDBStreams and Tables in Apache Kafka: Elasticity, Fault Tolerance, and other Advanced ConceptsBrowse the Confluent HubJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at Confluent Developer
There are two primary industries within the Internet of Things (IoT): industrial IoT (IIoT) and consumer IoT (CIoT), both of which can benefit from the Apache Kafka® ecosystem, including Kafka Streams and Kafka Connect. Kai Waehner, who works in the advanced tech group at Confluent with customers, defining their needs, use cases, and architecture, shares example use cases where he’s seen IoT integration in action. He specifically focuses on Walmart and its real-time customer integration using the Walmart app. Kafka Streams helps fine-tune the Walmart app, optimizing the user experience, offering a seamless omni-channel experience, and contributing to business success. Other topics discussed in today’s episode include integration from various legacy and modern IoT data sources, latency sensitivity, machine learning for quality control and predictive maintenance, and when event streaming can be more useful than traditional databases or data lakes.EPISODE LINKSApache Kafka 2.5 – Overview of Latest Features, Updates, and KIPsMachine Learning with Kafka Streams, Kafka Connect, and ksqlDB ft. Kai WaehnerBlog posts by Kai WaehnerProcessing IoT Data from End to End with MQTT and Apache Kafka®End-to-End Integration: IoT Edge to Confluent CloudApache Kafka is the New Black at the Edge in Industrial IoT, Logistics, and RetailingApache Kafka, KSQL, and Apache PLC4X for IIoT Data Integration and ProcessingStreaming Machine Learning at Scale from 100,000 IoT Devices with HiveMQ, Apache Kafka, and TensorFlowEvent-Model Serving: Stream Processing vs. RPC with Kafka and TensorFlowJoin the Confluent Community SlackLearn about Kafka at Confluent Developer
Confluent Platform 5.5 is out, and Tim Berglund (Senior Director of Developer Advocacy, Confluent) is here to give you the latest updates! The first is improved schema management and Confluent Schema Registry support for Protobuf and JSON, making these components pluggable. The second is better support for languages other than Java within the sphere of librdkafka. And finally, this release includes an upgrade to ksqlDB, which expands its functionality, supports more data types, increases availability for pull queries, and adds a new aggregate function.EPISODE LINKSConfluent Platform 5.5 Release NotesIntroducing Confluent Platform 5.5Watch the video version of this podcastJoin the Confluent Community SlackLearn about Kafka at Confluent Developer
During his time at Twitter, Sam Ritchie (Staff Research Engineer, Google) led the development of Summingbird, a project that helped Twitter ingest and process massive amounts of data. It relieved some key pain points, saving developers at Twitter from doing work twice, as was a natural consequence of the then-current Lambda Architecture. In this episode, Sam dives teaches us some abstract algebra and explains how it has informed his attempts to make stream processing programs easy to write in a more general way.EPISODE LINKSCheck out SummingbirdJoin the Confluent Community SlackLearn about Kafka at Confluent Developer
Apache Kafka® 2.5 is here, and we’ve got some Kafka Improvement Proposals (KIPs) to discuss! Tim Berglund (Senior Director of Developer Advocacy, Confluent) shares improvements and changes to over 10 KIPs all within the realm of Core Kafka, Kafka Connect, and Kafka Streams, including foundational improvements to exactly once semantics, the ability to track a connector’s active topics, and adding a new co-group operator to the Streams DSL.EPISODE LINKSCheck out the Apache Kafka 2.5 release notesRead about what’s new in Apache Kafka 2.5Watch the video version of this podcastJoin the Confluent Community SlackLearn about Kafka at Confluent Developer
loading
Comments 
Download from Google Play
Download from App Store