In episode 178 of the Content Strategy Experts podcast, Sarah O’Keefe and Christine Cuellar perform a pulse check on the state of AI as of December 2024. They discuss unresolved complex content problems and share key considerations for entering 2025 and beyond.

The truth that we’re finding our way towards appears to be that you can use AI as a tool and it is very, very good at patterns and synthesis and condensing content. And it is very, very bad at creating useful, accurate, net new content. That appears to be the bottom line as we exit 2024.

— Sarah O’Keefe

Related links:

• Pulse check on AI: May, 2024 (podcast)


• AI in the content lifecycle (white paper)


• The future of AI: structured content is key (webinar)


• Savor the season with Scriptorium: Our favorite holiday recipes

LinkedIn:

• Sarah O’Keefe


• Christine Cuellar

Transcript:

Disclaimer: This is a machine-generated transcript with edits.

Christine Cuellar: Welcome to the Content Strategy Experts Podcast brought to you by Scriptorium. Since 1997, Scriptorium has helped companies manage, structure, organize, and distribute content in an efficient way. In this episode, it’s time for another pulse check on AI. So our last check-in was in May, which in AI terms is ancient history, so today, Sarah O’Keefe and I are gonna be talking about what’s changed and how it can affect your content operations. Sarah, welcome to the show.

Sarah O’Keefe: Hey Christine, thanks.

CC: Yeah. So 2024, as we’re currently recording this 2024 is winding down. People are preparing for 2025. Throughout this year, we went to a lot of different conferences and events. Of course, everybody’s talking about AI. So Sarah, based on the events that you like just recently got back from, you finally get to be in your own house. What are your thoughts about what’s going on with AI in the industry right now?

SO: There’s, still a huge topic of conversation. Lots of people are talking about AI, a huge percentage of presentations, you know, had AI in the title or referenced it or talked about it. With that said, it seems like we’re seeing a little more sort of real world, hey, here’s some things we tried, here’s what’s working, here’s what’s not working. 

CC: Mm-hmm.

SO: And I’ll also say that we’re starting to see a really big split between the AI in regulatory environments, which would include the entire EU plus certain kinds of industries and the sort of wild, wild west of we can do anything.

CC: Yeah. So do you feel like it sounds like, know, when AI first came onto the scene, there was mostly, you know, let’s just all adopt this right now. Let’s go for it full steam ahead, especially marketers as a marketer. can I can say that because we’re definitely gung-ho about stuff like that. It sounds like, the perspective has shifted to being more balanced overall. Is that what you would say?

SO: Yeah, I mean, that’s the typical technology adoption curve, right? You know, have your your peak of inflated expectations, and then you have the I think it’s the valley. It’s not the valley of despair, but it’s something like that. But you know, you sort of go from this can do anything. This thing is so cool. Go, go, go, go, go to a more realistic. Okay, what can it actually do? And what you know, does the and this is true for AI or anything else? What can it do? What can’t it do? What does it do well?

CC: Mm.

SO: Where do we need to put some guardrails around it? What are some surprises in terms of things that are and are not working?

CC: Yeah. And at some of the conferences we were at this year, our team had some things to say about AI as well. So we will link some of the recap blog posts we have in the show notes. Sarah, what are some of the things AI can’t do right now? are the still, what are, Sarah, what are some of the big concerns about AI that are still unanswered, unresolved?

SO: So in the big picture, as we’re starting to see people roll out AI-based things in the real world, whether it’s tool sets or content ops or anything else, we’re starting to see some really interesting developments and some really interesting assessments. Number one is that when you look at those little AI snippets that you get now when you do a search and it returns a bunch of search, well, actually it returns a page of ads.

CC: Yes.

SO: And then some real results under the ads. And then above that, it returns an AI overview snippet. So those are surprisingly bad. You do a search on something that you know a little bit of something about and see what you get. And you will see content in there that is just flat wrong. I’m not saying it’s not the best summary. I’m saying it is factually incorrect, right?

CC: Yeah, I hate them right now.

SO: So those are surprisingly bad. And talking about search for a minute, which ties into your question about marketing, there’s some real problems now with SEO, with search engine optimization, because if I’m optimizing my content to be included in an AI overview that is A, wrong, and B, doesn’t actually give me credit, Pre-AI, those snippets that showed up would say, I sourced it from over here.

CC: Mm-hmm.

SO: And in many cases now, the AI overview is just like the sort of summary paragraph with no particular, there’s no citation. It doesn’t say where it came from. So what’s in it for me as a content creator? Why am I creating content that’s going to get taken over by the AI overview and then not lead to people going to my webpage, right? How’s that helped me? 

CC: Yeah. Yeah.

SO: So there’s some real issues there, there’s a move in the direction of thinking about levels of information. So thinking about very superficial information. How much does a cup of flour weigh? That type of thing. That’s just a fact and you can get it pretty much anywhere, we hope. And then there’s deeper information. Why is it better to weigh flour than to measure it? By volume, if you’re a baker.

CC: Yeah.

SO: And what does it look like to use weights? And are there differences among different kinds of flours? And what are some of the things I should consider when I’m going in that direction? So one of those, know, flours, a cup of flour weighs 120, sorry, a cup of all-purpose flour weighs 120 grams is a useful fact. And I don’t know if I really care if people peruse that further or come to my website for more about flour. The deeper information, the more detailed discussion of, you know, whole wheat versus all-purpose versus European flours versus American flours and all these other kinds of things, that requires more in-depth information and that is not so subject to being condensed into an AI summary. So that distinction between, you know, quick and dirty information versus deeper information, information that goes into a topic,

CC: Mm-hmm.

SO: We have a huge problem with disinformation and misinformation with information that is just flat out not either not correct or because of the way AI tools work, is trivially easy to generate content at scale. Tons and tons and tons and tons and tons of content. And because it’s trivially easy,

CC: Mm-hmm.

SO: That means it’s also trivially easy for me to generate, for example, a couple thousand fake reviews for my new product or a couple thousand websites for my fake products. It we can fractionalize down the generation of content. 

CC: Yeah.

SO: And the you know, the interesting part of this is that it implies that you could potentially, you know, we talk about doing A/B testing and marketing. You could do A/B/C/D/E/F/G testing pretty easily because you can generate lots and lots of variants and kind of throw a bunch of stuff against the wall and see what works. But the bad side of this is that you can generate fake news, fake information, fake content that is going to be hig