Small Language Models with Luca Antiga
Description
In this episode, Luca Antiga, CTO of Lightning AI and ODSC East 2024 speaker, joins us to discuss Small Language Models (SLMs).
Lightning AI, well regarded for providing tools and platforms that streamline workflows for machine learning engineers and AI researchers, was founded by the creators of the very popular open-source framework PyTorch Lightning.
Luca himself was an early contributor to the PyTorch core library and co-authored the book “Deep Learning with PyTorch”.
In this interview, we’ll explore the differences between SLMs and Large Language Models (LLMs), examine the importance of data quality in training SLMs, and talk about key SLM frameworks such as TinyLama.
We’ll also touch on the process of combining SLMs with Retrieval Augmented Generation (RAG), address hallucinations within SLMs in particular, and how to deploy SLMs into production.
SHOWNOTE TOPICS:
1- Guest Introduction
2- Explanation of Small Language Models (SLMs) is and how they differ from Large Language Models (LLMs)?
3- How SLMs are trained and the importance of data quality
4- Continued pre-training vs fine-tuning
5- Incontext learning for LLMs
6- Benchmarking LLMS with MMLU
7- Small Language Model (SLM) frameworks such as MiniCPM, TinyLlama, and LitGPT
8- Combining SLMs with Retrieval Augmented Generation (RAG) and fine-tuning
9- Why SLMs are more prone to hallucinations than LLMs and how to solve
10- LLMs and SLMs in production - minimizing risks and maximizing value
11- The problem of over optimizing on AI solutions
12- Practical applications of SLMs,
13- Small Language Model (SLM) training challenges
14- Overview of Lightning studio
15- AI Agents and challenges
16- Compute access from single to multi-GPUs
17- Using lightning studio to easily switch between compute setups
18- Quick fire questions:
- Will more foundational LLMs emerge?
- Open source vs closed-scouce LLM
- Artificial General Intelligence?
- In this fast paced environment what should practitioners focus on?
- What are companies getting wrong with AI now?
- What are companies getting right with AI?
SHOW NOTES:
Learn more about Luca Antiga, CTO at Lightning AI:
https://www.linkedin.com/in/lantiga/?originalSubdomain=it
Learn more about Lightning AI and PyTorch Lightning:
https://lightning.ai/docs/pytorch/stable/
https://github.com/Lightning-AI
Learn more about Small Language Models
https://opendatascience.com/the-growth-of-small-language-models/
https://opendatascience.com/6-small-language-models-to-get-the-job-done-with-ease/
MMLU (Multi-task Language Understanding) Benchmark:
https://arxiv.org/pdf/2009.03300
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
https://ar5iv.labs.arxiv.org/html/2404.06395
Small Language Model Frameworks:
https://github.com/jzhang38/TinyLlama
https://github.com/Lightning-AI/litgpt
Why Larger Language Models Do In-context Learning Differently?
https://arxiv.org/abs/2405.19592
Learn more about RAG:
Learn more about dottxt:
Learn more about structured text generation:
https://github.com/outlines-dev/outlines
Learn more about HyperTalk:
https://en.wikipedia.org/wiki/HyperTalk
Learn more about crewAI:
https://github.com/joaomdmoura/crewAI
Learn more about Lightning AI Studios:
Learn more about Concurrent Versions System: