DiscoverThe Subsurface PodcastEpisode 1: JD Long - The Big Join: Testing pipelines to join 30 billion rows of data… quickly
Episode 1: JD Long - The Big Join: Testing pipelines to join 30 billion rows of data… quickly

Episode 1: JD Long - The Big Join: Testing pipelines to join 30 billion rows of data… quickly

Update: 2021-09-16
Share

Description

JD Long is a veteran Quantitative Risk Analyst. He builds stochastic models to predict losses during catastrophic events like hurricanes, earthquakes, or droughts. He shares his data engineering team's painful experience standing up tooling pipelines to load 10s of billions of rows for imbalanced queries into multiple distributed systems. He’s the perfect first guest because he covers multiple tools and techniques and is not shy to share his team's mistakes. He calls it “learning out loud” and I enjoyed every minute of it.

Comments 
00:00
00:00
x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Episode 1: JD Long - The Big Join: Testing pipelines to join 30 billion rows of data… quickly

Episode 1: JD Long - The Big Join: Testing pipelines to join 30 billion rows of data… quickly

Dremio