Links:

• https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf


• https://www.theregister.com/2024/08/05/crowdstrike_is_not_at_all/


• https://www.theverge.com/2024/8/6/24214371/microsoft-delta-letter-crowdstrike-response-comments


• https://www.linkedin.com/posts/alexstamos_why-crowdstrikes-baffling-bsod-disaster-activity-7224046054076243969-1An8?utm_source=combined_share_message&utm_medium=ios_app


• https://www.linkedin.com/posts/choff_why-crowdstrikes-baffling-bsod-disaster-activity-7224078879445958658-ymuc?utm_source=combined_share_message&utm_medium=member_ios


• https://www.securityweek.com/thousands-of-devices-wiped-remotely-following-mobile-guardian-hack/


• https://www.bleepingcomputer.com/news/security/stackexchange-abused-to-spread-malicious-pypi-packages-as-answers/


• https://www.bleepingcomputer.com/news/security/hunters-international-ransomware-gang-targets-it-workers-with-new-sharprhino-malware/

Transcript:

Jerry: Today is Wednesday, August 7th, 2024. And this is episode 275 of the Defensive Security Podcast. My name is Jerry Bell and joining me tonight as always is Mr. Andrew Kalat.

Andrew: Good evening, Jerry. How are you? Good, sir.

Jerry: I am amazing. It is blistering hot at the beach, but it’s awesome.

Andrew: recording from your southern compound.

Jerry: I am.

Andrew: Nice.

Jerry: Yeah, Bell Estate South.

Andrew: And Debbie was not an issue.

Jerry: Debbie not here. We got probably 45 minutes worth of rain.

Andrew: Yeah, it seems, at this point, in real time, stalled out over South Carolina

Jerry: Yeah, it looks several feet of rain hitting like Savannah and That is nuts. But no, it was not a big issue here. I was pretty worried. I packed up all my Milwaukee batteries with lights and whatnot in preparation for the worst got extra tranquilizer for my dog who hates storms.

But no, it’s been absolutely amazing here.

Andrew: So you took the tranks instead? Is that what I’m hearing?

Jerry: Absolutely. You gotta sleep somehow.

Andrew: That’s fair. I’m glad it was a non event, at least for your little neck of the

Jerry: Yeah, it was Nice you could actually see some of the storm clouds off in the distance. And that was the best way to watch a hurricane is when it’s far away.

Andrew: That’s true. That

A few I’ve been through. Stuck on islands, but

Jerry: Yeah, that’s right. since I’ve been here, I have been in the building for two hurricanes, and the building’s been hit by three tornadoes. And then there was also a unsuccessful base jump.

Andrew: So we’re saying you are cursed. Is that what we’re saying?

Jerry: am the human equivalent to a plastic flamingo.

which attracts tornadoes for those who don’t know. Anyway.

Yeah.

Andrew: after that meteorological update,

Jerry: Yeah. just a reminder that the thoughts and opinions we express on the show are ours and do not represent those of our employers past, present, or future.

Andrew: maybe even our

Jerry: Or our pets. my pet is licking me right now and she says, nope, it’s not her opinion.

Andrew: fair,

Jerry: Okay I would say that this is going to be a CrowdStrike heavy episode.

Andrew: three weeks in a row.

Jerry: Yeah, it continues to get more and more interesting. Obviously the main event itself is largely behind us and now we are in the lawyer up phase of the party.

Andrew: the blamestorming

Jerry: blamestorming has indeed begun. The first topic we have to talk about here is the actual formal full root cause analysis was released yesterday by CrowdStrike and it is a 12 page long document. It has lots of marketing fluff in it.

And only I would say a little bit of substance. I don’t think there’s anything that is remarkably telling or revolutionary in the document, but it does indicate technically what went wrong. And it gives some indications of the, potential improvements for their quality assurance, which I think is where a lot of this went wrong.

So the, I’m not going to go through the details in uber technical specificity, but the net is that this channel file update is for this inter process communication agent, for lack of a better term, I’ll call it. And that agent, expects configuration files that have

20 parameters, but through some unfortunate

bad planningtheir test harness actually was Marking the 21st as a catch all, as an asterisk. It was effectively being marked as not used. And so in this particular update, they actually started using it, and that ended up causing their parser to perform what ultimately ended up being an out of bounds read.

Because that parser wasn’t set up to actually read it. And so when that read attempted to happen in kernel space, it tried to access memory. It wasn’t allowed to access, wasn’t allocated. And that caused the blue screen. And because the same thing happened every time it booted up.

You just had this endless boot loop until that particular file got removed. I think the more substantive issue, and that’s the kind of thing that can happen,

Andrew: So let me restate that to make

The application was expecting. a file that had 21 fields in it, and it got a file with 20.

Jerry: Yes.

Andrew: And where it went to read that 21st, it wasn’t allowed to read, and the way that systems protect themselves to do a kernel panic and shut down if you’re trying to read something you’re not allowed to

Jerry: Yes.

Andrew: If you’re in

Jerry: Windows basically says something is horrifically wrong. This should not happen.

Andrew: If I went by that criteria, I’d shut down every day.

Jerry: And so if that were to happen in user space, the application that performed that read would crash. But when it happens in kernel space, Windows attempts to protect itself and it blue screens.

And so the challenge is that testing harness was built assuming that 21st parameter was always set up as a catch all and so effectively was being ignored.

And I think there were really two issues here. One was they didn’t have a very thorough, their testing harness obviously wasn’t, Properly designed, but then they also did not have staged deployments. Like they, what they have a process where once it goes through that test harness and passes it, it goes out far and wide.

There is no staged, deployment ring concept that you have in, let’s say, Microsoft Windows updates and whatnot. And because of that, it, it blasted out. Everybody implicitly trusted CrowdStrike updates and those got applied to pretty much as, fast as they were delivered and the rest is now history.

Andrew: I thi