DiscoverHow AI Is BuiltData Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16
Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Update: 2024-07-12
Share

Description

This episode of "How AI Is Built" is all about data processing for AI. Abhishek Choudhary and Nicolay discuss Spark and alternatives to process data so it is AI-ready.

Spark is a distributed system that allows for fast data processing by utilizing memory. It uses a dataframe representation "RDD" to simplify data processing.

When should you use Spark to process your data for your AI Systems?

→ Use Spark when:

  • Your data exceeds terabytes in volume
  • You expect unpredictable data growth
  • Your pipeline involves multiple complex operations
  • You already have a Spark cluster (e.g., Databricks)
  • Your team has strong Spark expertise
  • You need distributed computing for performance
  • Budget allows for Spark infrastructure costs

→ Consider alternatives when:

  • Dealing with datasets under 1TB
  • In early stages of AI development
  • Budget constraints limit infrastructure spending
  • Simpler tools like Pandas or DuckDB suffice

Spark isn't always necessary. Evaluate your specific needs and resources before committing to a Spark-based solution for AI data processing.

In today’s episode of How AI Is Built, Abhishek and I discuss data processing:

  • When to use Spark vs. alternatives for data processing
  • Key components of Spark: RDDs, DataFrames, and SQL
  • Integrating AI into data pipelines
  • Challenges with LLM latency and consistency
  • Data storage strategies for AI workloads
  • Orchestration tools for data pipelines
  • Tips for making LLMs more reliable in production

Abhishek Choudhary:

Nicolay Gerold:

Comments 
loading
Top Podcasts
The Best New Comedy Podcast Right Now – June 2024The Best News Podcast Right Now – June 2024The Best New Business Podcast Right Now – June 2024The Best New Sports Podcast Right Now – June 2024The Best New True Crime Podcast Right Now – June 2024The Best New Joe Rogan Experience Podcast Right Now – June 20The Best New Dan Bongino Show Podcast Right Now – June 20The Best New Mark Levin Podcast – June 2024
In Channel
Vector Search at Scale: Why One Size Doesn't Fit All | S2 E13

Vector Search at Scale: Why One Size Doesn't Fit All | S2 E13

2024-11-0736:26

Search Systems at Scale: Avoiding Local Maxima and Other Engineering Lessons | S2 E12

Search Systems at Scale: Avoiding Local Maxima and Other Engineering Lessons | S2 E12

2024-10-3154:47

Training Multi-Modal AI: Inside the Jina CLIP Embedding Model | S2 E11

Training Multi-Modal AI: Inside the Jina CLIP Embedding Model | S2 E11

2024-10-2549:22

Building the database for AI, Multi-modal AI, Multi-modal Storage | S2 E10

Building the database for AI, Multi-modal AI, Multi-modal Storage | S2 E10

2024-10-2344:54

Numbers, categories, locations, images, text. How to embed the world? | S2 E9

Numbers, categories, locations, images, text. How to embed the world? | S2 E9

2024-10-1046:44

Building Taxonomies: Data Models to Remove Ambiguity from AI and Search | S2 E8

Building Taxonomies: Data Models to Remove Ambiguity from AI and Search | S2 E8

2024-10-0458:40

From PDFs to Pixels: How ColPali is Changing Information Retrieval | S2 E7

From PDFs to Pixels: How ColPali is Changing Information Retrieval | S2 E7

2024-09-2754:57

Beyond Embeddings: The Power of Rerankers in Modern Search | S2 E6

Beyond Embeddings: The Power of Rerankers in Modern Search | S2 E6

2024-09-2642:29

Limits of Embeddings: Out-of-Domain Data, Long Context, Finetuning (and How We're Fixing It) | S2 E5

Limits of Embeddings: Out-of-Domain Data, Long Context, Finetuning (and How We're Fixing It) | S2 E5

2024-09-1946:06

RAG at Scale: The problems you will encounter and how to prevent (or fix) them | S2 E4

RAG at Scale: The problems you will encounter and how to prevent (or fix) them | S2 E4

2024-09-1250:09

From Keywords to AI (to GAR): The Evolution of Search, Finding Search Signals | S2 E3

From Keywords to AI (to GAR): The Evolution of Search, Finding Search Signals | S2 E3

2024-09-0552:16

Data-driven Search Optimization, Analysing Relevance | S2 E2

Data-driven Search Optimization, Analysing Relevance | S2 E2

2024-08-3051:14

Query Understanding: Doing The Work Before The Query Hits The Database | S2 E1

Query Understanding: Doing The Work Before The Query Hits The Database | S2 E1

2024-08-1553:02

Season 2 Trailer: Mastering Search

Season 2 Trailer: Mastering Search

2024-08-0804:16

Unlocking Value from Unstructured Data, Real-World Applications of Generative AI | ep 17

Unlocking Value from Unstructured Data, Real-World Applications of Generative AI | ep 17

2024-07-1636:28

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

2024-07-1246:26

Building AI Agents for the Enterprise: Realistic Use Cases, Cost Controls, Seamless UX | ep 15

Building AI Agents for the Enterprise: Realistic Use Cases, Cost Controls, Seamless UX | ep 15

2024-07-0435:12

Building Predictable Agents: Prompting, Compression, and Memory Strategies | ep 14

Building Predictable Agents: Prompting, Compression, and Memory Strategies | ep 14

2024-06-2732:14

Data Integration and Ingestion for AI & LLMs, Architecting Data Flows | changelog 3

Data Integration and Ingestion for AI & LLMs, Architecting Data Flows | changelog 3

2024-06-2514:53

ETL for LLMs, Integrating and Normalizing Unstructured Data | ep 13

ETL for LLMs, Integrating and Normalizing Unstructured Data | ep 13

2024-06-1936:48

loading
Google Play
Download from Google Play
Castbox
Download from App Store
usUnited States
00:00
00:00
1.0x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Data Processing for AI, Integrating AI into Data Pipelines, Spark | ep 16

Nicolay Gerold