TL;DR:
System stopped booting after an upgrade. Turns out systemd was trying to mount nfs mounts before the network was up, failed, then aborted the entire boot process. Disabling nfs mounts in fstab let me boot into my machine again.
Rant
Ok, major rant here. I.H.A.T.E.S.Y.S.T.E.M.D.! I seriously don’t see the point. I’ve read lots of pro- and con- arguments. At the end of the day, sysvinit just worked. It may be clumsy. It may be ugly. But it works. And it’s nicely transparent and modular, in the Unix tradition.
systemd on the other hand is an opaque binary monolith. And when it breaks, it’s not always easy to figure out why (and it breaks often enough on upgrading). sysvinit was all text files. Easy. grep, vi, (and some web searches) and all could be fixed. Failing all else (or if you were masochistic enough) you could read your way through the various scripts and figure out what it was (trying) to do.
Not so easy with systemd. Maybe it’s just a case of needing to learn the new tools of the trade, but when something esoteric breaks, it’s very difficult to trace down exactly why. Often times it was easier to just reinstall (I cry a bit at this stage. Linux is turning into Windows). In either case, why learn bespoke tools to troubleshoot every subsystem, instead of having the same generic tools used for every subsystem? This is seriously where systemd falls down. I get the feeling it was designed by Microsoft and is being forcibly inserted into Linux to share their pain.
Issue
Anyway, thankfully this particular time I was able to (eventually) find the time and managed to trace down why my box was refusing to boot after the latest apt-get dist-upgrade. Turns out that for some obscure reason, systemd felt the need to ensure my nfs mounts were mounted before the network was up. Uh, right, like that’s going to work! I still haven’t figured out why.
I tried marking the nfs mounts as “_netdev”, and “x-systemd.requires=network-online.target” (despite the fact that both of those should be rather obvious for nfs type mounts, and the documentation claims systemd does default to that), but neither made any difference.
Fix (or is it)
Eventually I just disabled my nfs mounts in fstab to least let me boot into the box again. Hurrah, sort of. Solution? Not really.
I will eventually play around with some more of the systemd-specific markup and see if I can force it to see sense.
