systemd is a service manager for Linux. It is the first process that runs on many Linux distributions and manages all other user processes. It includes utilities for logging, process isolation, process dependencies, socket activation, and many other tasks.

psystemd is a python library to communicate with systemd over dbus from python as an alternative to shelling out from an application to control services.

Anita Zhang is an engineerd managerd at Meta and Alvaro Levia is a production engineer at Meta.

I attended their systemd workshop at the Southern California Linux Expo.

Topics covered:

• What's systemd?


• Giving talks and workshops


• cgroups and namespaces


• systemd timers vs cron


• Migrating from CentOS 6 to 7


• Production engineers need to go lower in the stack to debug applications


• Meta's Linux userspace team


• Use of public cloud at Meta


• Meta's bootcamp


• Pystemd

Mastodon

• Anita Zhang


• Alvaro Leiva

Workshop

• systemd workshop

Conference talks

• Journey into the Heart of systemd - Scale 19x


• Systemd: why you should care as a Python developer - PyCon 2018


• Move Fast without Breaking things - Scale 18x


• Solving All the Problems with systemd - LISA18


• Using systemd to high level languages - All Systems Go!


• The Curious Case of Memory Growth - Scale 19x

Related Links

• systemd


• psystemd


• systemd-run


• systemd-timers

Transcript

You can help edit this transcript on GitHub.

Introductions

[00:00:00 ] Jeremy: So today I'm talking to Avaro Leiva and Anita Zhang. Avaro is the author of the pystemd library and he's a production engineer at Meta. And Anita is an engineerd managerd at Meta, and I'll let her explain that further.

[00:00:19 ] Jeremy: But thank you both for joining me today.

[00:00:21 ] Anita: Yeah, thanks for having us.

[00:00:24 ] Jeremy: I guess where we could start, Anita, maybe you could explain a little bit your, your title that I just gave you there.

engineerd managerd

[00:00:31 ] Anita: Yeah, so by default I, I should be a software engineering manager, but when I transitioned to management, I was not, Ready to go public with, um, my transition.

So I kind of hid it by, changing the title. we have some weird systems in place that grep on like the word engineer. So I had to keep engineer in there somehow. and so I kind of polled my friends what I should change my title to, and they're like, oh, you're gonna support the systemd team, so you should change it to like managerd.

So I was like, sounds good. engineerd, managerd.

I didn't wanna get kicked out of any workplace groups, for example, that required me to be an engineer.

[00:01:15 ] Jeremy: Oh, okay.

[00:01:17 ] Anita: Or like engineering function, I guess.

[00:01:19 ] Jeremy: Yeah. Yeah. And you just gotta title it yourself, so as long as you got engineer in it, you're good.

[00:01:24 ] Anita: Yeah, pretty much. Some people have really fun titles like Chief Potato Officer and things like that.

[00:01:32 ] Jeremy: So what groups does the, uh, the potato officer get to go in?

[00:01:37 ] Anita: Yeah. Not the C level ones. (laughs)

What's systemd?

[00:01:42 ] Jeremy: I guess maybe to, to start, we should explain to people who aren't familiar, uh, what systemd is. So if either of you wanna wanna take that one.

[00:01:52 ] Alvaro: so people who doesn't know, right?

So systemd is today is your init system, right? Is the thing that manage your, your process. and the best way to understand this, it is like when your computer, it needs to execute something. And that's something is what we call pid one. And that pid one is the thing that is gonna manage everything from now from there on, right?

Uh, in the most basic level, if you remember how to, how does program start, how does like an idea becomes a program? Uh, you need to fork exec, right? So that means that something has to be at the top of that tree and that is systemd. now that can be anything, right? So there was a time where that was like systemv and there was also like upstart, uh, today's systemd is the thing that, uh, it's shipped in most distributions.

[00:02:37 ] Jeremy: Yeah, because I, I definitely remember when I first started working with Linux, uh, it was with CentOS 6, and when I would want to run a service, I would have to go and write a bash script and kind of have all these checks for, is this thing running?

Does it have permission to these things, which user is it running as, and so there was a lot of stuff that I remember having to do before systemd came out.

[00:03:08 ] Alvaro: The good old days as we call them,

[00:03:11 ] Jeremy: Or the bad old days.

[00:03:13 ] Anita: Yeah. Depending on who you ask.

[00:03:15 ] Alvaro: Yeah. So, so that is super interesting because, um, During those time, like you said, you have to write a first script.

That means that you were basically yourself, your own service manager, right? So ideas as simple as, is my program running? There was no real answer. You have to figure it out, right? So if you run a program, uh, you maybe would create a pid file which hold the p or the pid of the process, of the main process, right?

And then something needs to check, oh, is this file exist? Does the file exist and does the content of this file actually match to a process? And then you grab the process. So it was all these ideas that you had to do, and then for, you have to do it for every single software that you would deploy on your machine, right?

That also makes really hard to parallelize stuff, right? Because you have no concept of dependencies. So if your computer has to put, uh, I, I dunno if you remember like long time ago, like Linux machine would, takes like five minutes to boot like your desktop. I remember like openSUSE. I can't remember, like 2008, 2007.

Uh, it would take like five minutes to boot and then Ubuntu came and, and it start like immediately. And it was because, you can parallelize things, but you cannot do that if all you're running are bash script.

Why was systemd chosen to be included in Linux distributions?

[00:04:26 ] Jeremy: I remember before the Linux distributions didn't include it.

And I wonder if you have any insight into how systemd got chosen to be the thing to manage our processes and basically how we got to where we are today.

[00:04:44 ] Anita: I mean, we can kind of speculate a little bit. at the time when Lennart started systemd, um, with. Kai Sievers probably messed up his name there. Um, they were all at Red Hat and Red Hat manages fedora these days and I believe fedoras kind of like the bleeding edge for a lot of the new software ideas. Um, and when they picked up systemd as the defaults, um, eventually it started to trickle down to the rest of their distributions through RHEL and to CentOS and at the same time, I think other distributions started to see how useful it was in terms of managing all the different processes and services.

Um, I know Debian at one point had kind of a vote on like whether they should make systemd either default or like, make it easy to switch between both. And then they decided to just stick with systemd because it's, I mean, the public agrees that it's like easy to use and it's more useful.

It abstracts away a lot of things that they had to manually do before

Who is interested in systemd? Who comes to your talks and workshops?

[00:05:43 ] Jeremy: Something I've been kind of curious about. So just this year at SCaLE uh, you ran a, a workshop teaching people how to use systemd and, and sort of what it is about. I guess when, when you get people coming to these workshops, what are they typically, where are they typically coming from? Are they like system administrators or are they software developers?

Like when you run these workshops, who are you looking for as your audience?