I'm excited to share this episode with Don Rosenthal. Don is a seasoned product leader with extensive experience in AI and large language models. He has led product teams at Google AI research, Facebook Applied AI, and Uber's Self-Driving Technology division. During this conversation, Don shared his insights on the anatomy of an LLM, ways to incorporate LLMs into products, risk mitigation strategies, and taking on LLM-powered projects.

Links

Don on LinkedIn

Attention is all you need

The Illustrated Transformer

Transcript

[00:00:00 ] Don Rosenthal: please, please, please do go out and do come up with unique and exciting, uh, important new applications, build stuff that solves important problems we couldn't even try to address previously. I just want you to be sure that, uh, you're going into this with your eyes open and that you've prepared your stakeholders properly.

[00:00:21 ] Don Rosenthal: There, um, there are a lot of successful applications that have been built with these LLMs and a lot of the pioneers have discovered all the pitfalls and where all the dragons hide so that we can We can avoid them. 

[00:00:35 ] Himakara Pieris: I'm, Himakara Pieris. You're listening to smart products. A show where we, recognize, celebrate and learn from industry leaders who are solving real world problems. Using AI.

[00:00:46 ]

[00:00:47 ] Himakara Pieris: , today we are going to talk about large language models. And I can't think of a better person to have this conversation with than Don Rosenthal. So Don has [00:01:00 ] spent most of his career in AI.

[00:01:02 ] Himakara Pieris: He started out as a developer. building ground support systems for the Hubble telescope, um, including being part of the team that built the first air ground system ever deployed for a NASA mission. He then went on to build and manage NASA's first AI applications group, where his team flew in, flew the first two AI systems in space.

[00:01:22 ] Himakara Pieris: And he worked on prototype architectures for autonomous Mars rovers, done then commercialized. Uh, the air technology from Hubble Telescope in two of his air companies that he founded. He was the group product manager for autonomy at Uber 80 G. Uber's autonomous vehicle spin off in Pittsburgh. He was the PM for face recognition at Facebook.

[00:01:43 ] Himakara Pieris: And most recently, Don was the group product manager for conversational at a I research 

[00:01:50 ] Himakara Pieris: done. Welcome to the smart production.

[00:01:53 ] Don Rosenthal: Thank you very much. I'm really, really excited to be here. You might. Thank you for inviting me.[00:02:00 ]

[00:02:01 ] Himakara Pieris: So let's start with the basics. What is an LLM? 

[00:02:05 ] Don Rosenthal: Good place to start. Um, let me start out by saying that, uh, LLMs have finally solved, and I don't think that's really an exaggeration.

[00:02:14 ] Don Rosenthal: They finally solved one of the longstanding foundational problems of natural language understanding. Understanding the user's intent. Um. What do I mean by that? Um, uh, any one of us who's used the recommender system for movies, TV, music, which pretty much all of us, um, we know how frustrating it can be to try to get the system to understand what we're, we're looking for.

[00:02:40 ] Don Rosenthal: These systems have all trained us to dumb down our queries. Uh, in order to have any chance of a successful retrieval, you can't talk to in the way you would to a friend or or to any other person. You can't, for example, say, Hey, um, I like all kinds of music. Uh, the genre is not [00:03:00 ] important, jazz, pop, classical, rock, even opera, as long as it's got a strong goosebump factor, put together a playlist for me with that kind of vibe for the next 30 minutes while I do chores.

[00:03:13 ] Don Rosenthal: But you can, in fact, say that to, uh, something that's got a large language model in it, like chat gbt. And go ahead and try it. When I did, I even asked it if it understood what I meant by goosebump factor, assuming I'd have to explain it, but it said, Sure, I know what it is and gave me a perfectly reasonable explanation and definition of it.

[00:03:36 ] Don Rosenthal: So... Why and how is it able to do that? Um, we can get into the technology a little bit later, but for the 3, 000 foot level to start with, the point is that through an absolutely enormous amount of training, um, these systems have internally created a highly nuanced model of language. Which they can [00:04:00 ] then use for the semantic understanding of language that is input to it, as well as to craft highly nuanced and natural sounding language responses.

[00:04:09 ] Don Rosenthal: Um, and it's important to, to, uh, to underscore that these are the two things that large language models do really well. Um, semantic understanding of language and its input to it, and Uh, highly nuanced and natural sounding land, which responses and yes, they hallucinate and they make up stuff out of thin air.

[00:04:30 ] Don Rosenthal: But the interesting thing is that they always seem to hallucinate within the correct context of your query. So, you know, if you ask them about strawberries, it might make stuff up about strawberries, but it's not going to make stuff up about fire engines. And as for the highly nuanced Natural sounding responses.

[00:04:53 ] Don Rosenthal: Um, just, uh, remember, for example, the response to the, uh, the query of generating instructions for [00:05:00 ] removing a peanut butter sandwich from a VCR written in the style of the ST James Bible, which kind of broke the Internet last November. 

[00:05:10 ] Himakara Pieris: Take us inside an LLM. Um, what makes this technology so transformative, if you will?

[00:05:17 ] Don Rosenthal: Um, I'm not going to go into the, the technical details of how they work, but, um, it'd be great to be able to cover. Why they're so important and what has enabled them to become the agent of change in LLP to become so transformative. Um, and if you are interested in, in more details, the original paper from 2017 is attention is all you need.

[00:05:42 ] Don Rosenthal: It's all over the internet. You can find it easily. I'd also recommend, um, the Illustrated Transformer by Jay Alamar, A L A M m a R, who is well known for his, uh, incredible capability of helping you to easily understand complicated, [00:06:00 ] uh, concepts. And if you'd rather watch a video than, than read an explanation to check out his video.

[00:06:06 ] Don Rosenthal: The narrated transformer anyway, six transformers were able to. Help us leapfrog into the current generation of NLP tools. It's kind of important to first explain the state of the art just prior to their introduction, if that's okay. So, at that time, uh, NLP, the NLP world was using a set of technologies which were, uh, grouped together under the subfield of recurrent neural networks.

[00:06:34 ] Don Rosenthal: Um, not a very descriptive name, um, But the TLDR is that these technologies took the input sequence, any type of sequence, but let's say with language, so sequence of words in the sentence, um, and, uh, the RRN took the sequence of words, fed them in, in order, one at a time, the, quick, brown, fox, etc. [00:07:00 ] But they included a really novel component, which enabled feedback connections that allowed them to inject information from previous time time steps.

[00:07:09 ] Don Rosenthal: And this is what enabled them to capture contextual dependencies between words in a sentence instead of just having a look at one particular word. But so when quick was input, you get some feedback from the When brown was input, some feedback from the quick problem with this was, I mean, it was, it worked well for the time, but the problem was that the farther along in the sentence you got, the weaker the feedback was from the early previous steps.

[00:07:39 ] Don Rosenthal: So by the time you got to the end of the input sequence, um, the system may have been left with so little signal from the initial inputs. that they had very little effect on the evaluation of the sequence. So, uh, put that all together. Words that were closer to each other affected each other more than words that were farther apart in [00:08:00 ] trying to understand what the sentence meant.

[00:08:02 ] Don Rosenthal: And obviously that's a problem because language isn't constructed that way. Um, it also meant that sequences could only be evaluated. Sequentially, one at a time, and that made RNN processing really low. So the two stripes against RNNs, although they were really valuable for the time, was that they focused more on words that happened to be closer together in a sentence, and that they only processed sequentially, uh, one word at a time.

[00:08:31 ] Don Rosenthal: Um, so then along came transformers with a new idea, which was, let's present all of the words in the sequence to the transformer at once, all at the same time. Thank you. And this lets the system evaluate the connections between each word and every other word, um, regardless of where they show up in the sentence, um, and it can do this to figure out which words shou