Scaling CI horizontally with Buildkite, Kubernetes, and multiple pipelines, with Ben Poland
Description
Ben Poland walks through Faire's complete CI transformation, from a single Jenkins instance struggling with thousands of lines of Groovy to a distributed Buildkite system running across multiple Kubernetes clusters.
He details the technical challenges of running CI workloads at scale, including API rate limiting, etcd pressure points, and the trade-offs of splitting monolithic pipelines into service-scoped ones.
You will learn:
How to architect CI systems that match team ownership and eliminate shared failure points across services
Kubernetes scaling patterns for CI workloads, including multi-cluster strategies, predictive node provisioning, and handling API throttling
Performance optimization techniques like Git mirroring, node-level caching, and spot instance management for variable CI demands
Migration strategies and lessons learned from moving away from monolithic CI, including proof-of-concept approaches and avoiding the sunk cost fallacy
Sponsor
This episode is brought to you by Testkube—where teams run millions of performance tests in real Kubernetes infrastructure. From air-gapped environments to massive scale deployments, orchestrate every testing tool in one platform. Check it out at testkube.io
More info
Find all the links and info for this episode here: https://ku.bz/klBmzMY5-
Interested in sponsoring an episode? Learn more.
United States
