Thread by @thatcks

It's marvelous how systemd-timesyncd.service fails to start on one of our Ubuntu 18.04 machines with the log message 'Failed to create state directory: Permission denied' and there is no way to diagnose this at all (not even a message about WHAT DAMN DIRECTORY).

After chasing many things, timesyncd appears to be failing to start because systemd itself has forgotten or misplaced the 'systemd-timesync' private? user and group, so /var/lib/private/systemd/timesync is (re)created in a broken state.

Indeed, 'getent passwd|group systemd-timesync' works on all Ubuntu 18.04 machines except the one problem one, even though the system still knows what UID and GID to give the directory. So: left hand not talking to right hand, and it's undebuggable.

Thanks, systemd, for returning us to the era where the best answer to system problems for almost everyone is 'hell, I don't know, reboot the system and hope it goes away'. Can't reboot the system? Tough luck, you're up the creek.

It appears most likely that systemd half-forgot its own DynamicUser(s) when a systemd package update happened and systemd re-exec'd itself. The internals of dynamic users are undocumented, of course, so who knows.

I was wrong about the systemd timesyncd problem, it's not DynamicUser. It's more mysterious and screwed up than that and I have no answers, just more frustrations and a blog rant (and a bunch of strace output because why not, whatever).

The systemd timesync problem appears to be because there are lingering dead xrdp-chansrv FUSE mounts in /proc/mounts, which causes systemd to give up when it's trying to create a new namespace and never switch timesyncd to it, so /var/lib/private access fails.

Systemd does not of course log any sort of failure message when it gives up on setting up the DynamicUser private namespace; it just goes ahead and silently runs the service in the regular filesystem, even though it knows that is guaranteed to fail.

Attempting to manually unmount the nominal FUSE filesystem mounts as root gets the great error:

# umount /some/nfs/fs/thinclient_drives
umount: /some/nfs/fs/thinclient_drives: block devices are not permitted on filesystem.

Attempting to unmount the FUSE filesystems as the user fails, ultimately with the error 'Operation not permitted' (if one writes a program to just directly call umount(), since umount will try to stat things and fail with various charming errors due to that).

One can eventually umount these xrdp-chansrv FUSE mounts, *if* one temporarily chmod 755's the directory path to them so the NFS client machine's UID 0 has access to them; otherwise, no.

This is not a robust solution and systemd's handling of this is broken.

The overall end result here is that using DynamicUser is dangerously unreliable on at least the Ubuntu 18.04 version of systemd and perhaps all current versions. Systemd can have spurious setup failures that are not logged and do not abort things.

The simple reproduction of our Ubuntu 18.04 timesyncd failure to restart problem is an active sshfs or smbfs FUSE mount in a NFS-mounted home directory that is mode 700 (or generally not world-accessible).

So that's it for timesyncd for us on 18.04. Thanks, systemd.