Boot optimization

Hi all.
“Inspired” by another question I started to analyze what processes or events “consume” boot up time.
But I found a nuisance: there’s no consistency between different start-ups, so optimization will come from two points: to remove unnecessary modules or programs, and to determine why, sometimes, there are so many variations for the same process.
I explained a little more:
On Wednesday I boot up my machine at morning. It was a “cold” start-up. This is what I got:

$ systemd-analyze time Startup finished in 2.611s (kernel) + 3.387s (initrd) + 36.198s (userspace) = 42.197s
Today I made another cold start-up:

$ systemd-analyze time Startup finished in 2.613s (kernel) + 3.537s (initrd) + 32.722s (userspace) = 38.872s
Well, 3.3 seconds difference, about 8%. And there are almost no differences in kernel time and initrd time. All came from “userspace”.
A note: my box has a Intel i5 at 3.1 GHz, with 4 GB of RAM and 2 hard disks with A LOT of partitions. I don’t know if the 5 seconds dedicated to kernel and initrd are too much or not. Comparing with the whole 40 seconds to boot, I don’t care yet about optimizing kernel and initrd.
Analyzing “individual” boot times, I have got on Wednesday:

$ systemd-analyze blame | head -15 19.999s dkms.service 13.320s ModemManager.service 12.964s firewalld.service 11.303s systemd-logind.service 11.285s avahi-daemon.service 11.283s rtkit-daemon.service 11.283s upower.service 11.282s udisksd.service 7.670s NetworkManager-wait-online.service 5.979s nmb.service 4.859s network-up.service 2.452s plymouth-quit-wait.service 2.383s systemd-udev-settle.service 2.223s chronyd.service 1.934s systemd-fsck-root.service
And today:

$ systemd-analyze blame | head -15 16.881s dkms.service 8.322s ModemManager.service 7.944s NetworkManager-wait-online.service 6.995s firewalld.service 6.534s systemd-logind.service 6.534s avahi-daemon.service 6.532s rtkit-daemon.service 6.532s upower.service 6.531s udisksd.service 6.194s nmb.service 4.223s plymouth-quit-wait.service 4.093s network-up.service 2.549s systemd-udev-settle.service 2.501s network.service 2.430s rpcbind.service

Well…
dkms.service eats up a lot of time. Always. I don’t know if it’s always needed. I understand that I need it if I must update the nvidia driver for example, or the kernel. But, is it always needed?
Some processes vary their boot time a lot, but they always “stay together” (so I guess that some of them influences in others at the time of boot). In my case, systemd-logind.service, avahi-daemon.service, rtkit-daemon.service, upower.service and udisksd.service last almost the same in both cases, even when they are VERY different between boots, almost 45% of difference !! WHY ??
That difference can reach extreme limits. gpm.service lasted 1.804 seconds on Wednesday, and 0.317 seconds today. A 6x factor. But the champions are:
fedora-loadmodules.service: 1.805 s vs. 0.047 s
alsa-restore.service: 1.815 s vs. 0.063 s
Maybe another case of “stay together”? I don’t know, but if so, something happens with fedora-loadmodules.service. Anyway, there’s a 38x factor for fedora service and a 29x factor for alsa. It’s a way too much !

My questions:

1- Why I get so different times with some services?
2- Can those differences be “controled” or optimized in some way?
3- I use a router to my local network. The router is connected to a cablemodem to get Internet. I’m not directly connected to the cablemodem, except in very rare and specific situations (always related to my ISP). Do I need the “ModemManager.service”?
4- Can network times (NetworkManager and network services) be optimized if I don’t use DHCP and force a fixed IP in my network? I guess that’s a point to be easily optimized. I’ll try in some next boot.

Hi Sirius,
here are what I get on my laptop:

$inxi
CPU~Quad core Intel Core i7-6700HQ (-HT-MCP-) speed/max~800/2601 MHz Kernel~4.1.15-nrjQL-desktop-ql1omv x86_64 Up~4 days Mem~5119.2/7884.2MB HDD~1128.2GB(20.0% used) Procs~335 Client~Shell inxi~2.2.27 

$systemd-analyze 
Startup finished in 5.282s (firmware) + 3.002s (loader) + 3.417s (kernel) + 1.356s (initrd) + 9.844s (userspace) = 22.903s

$systemd-analyze blame
          5.268s NetworkManager-wait-online.service
          2.266s mysqld.service
          2.240s httpd.service
           578ms systemd-udev-settle.service
           551ms dkms.service
           518ms firewalld.service
           468ms plymouth-start.service
           355ms systemd-fsck@dev-disk-by\x2duuid-0c78a10b\x2ded35\x2d42ee\x2da465\x2d4a55f3e366d4.service
           344ms postfix.service
           320ms laptop-mode.service
           281ms ldap.service
           231ms ModemManager.service
           195ms saslauthd.service
           194ms accounts-daemon.service
           192ms mnt-wind.mount
           174ms proftpd.service
           168ms xinetd.service
           168ms hddtemp.service
           159ms tuned.service
           122ms bluetooth.service
           118ms systemd-timedated.service
           117ms avahi-daemon.service
           112ms user@500.service
          5.268s NetworkManager-wait-online.service
          2.266s mysqld.service
          2.240s httpd.service
           578ms systemd-udev-settle.service
           551ms dkms.service
           518ms firewalld.service
           468ms plymouth-start.service
           355ms systemd-fsck@dev-disk-by\x2duuid-0c78a10b\x2ded35\x2d42ee\x2da465\x2d4a55f3e366d4.service
           344ms postfix.service
           320ms laptop-mode.service
           281ms ldap.service
           231ms ModemManager.service

You can see that I get shorter delays, especially for dkms. Here is what I get with the plot option (extract) around the network stuffs:

  • dkms: I’s say it depends on which modules are installed. I use dkms-vboxadditions and dkms-virtualbox
  • NetworkManager-wait-online.service: you might try to disable it and see what happens or reduce the timeout setting in /lib/systemd/system/NetworkManager-wait-online.service. See also man nm-online

Tx for answer JC !

In MCC, the only service that includes “dkms” in the name is, well, dkms. But I have som dkms packages installed:

# rpm -qa | grep dkms
dkms-nvidia-current-352.30-0.1-omv2014.0.x86_64
dkms-2.2.0.3.1-0.20130827.6-omv2014.0.noarch
dkms-vboxadditions-5.0.14-0.1-omv2014.0.x86_64
dkms-virtualbox-5.0.14-0.1-omv2014.0.x86_64

So I guess I cannot avoid dkms.

In my system, the plot shows:

I cannot figure why dkms last so much. But it can be seen that something happens that ends several services-load-times at once, as avahi, systemd-logind, rtkit, upower and udisksd all of them finished at almost the same time.

For the moment, I disabled (via MCC) the laptop-mode.service, for one simple reason: my system is not a laptop !!! I don’t know why is loaded at boot time. And it eats some seconds ! (2.207 secs today, 1.758 secs on Wednesday).

Just for comparison here’s what I’m getting in 2014 (hopefully soon to be renamed OM Lx 2):

$ inxi
CPU~Quad core Intel Core i5-4590 (-MCP-) speed/max~3299/3700 MHz Kernel~4.1.15-nrjQL-desktop-1omv         x86_64 Up~0 min Mem~519.9/15927.9MB HDD~2500.5GB(3.1% used) Procs~188 Client~Shell inxi~2.2.27 

$ systemd-analyze
Startup finished in 3.299s (firmware) + 2.946s (loader) + 2.711s (kernel) + 890ms (initrd) + 9.656s (userspace) = 19.503s

$ systemd-analyze blame | head -15
      7.781s NetworkManager-wait-online.service
       461ms plymouth-start.service
       442ms plymouth-read-write.service
       350ms dkms.service
       325ms plymouth-quit-wait.service
       305ms firewalld.service
       249ms saslauthd.service
       247ms hddtemp.service
       239ms xinetd.service
       229ms systemd-udev-settle.service
       228ms tuned.service
       124ms ModemManager.service
       101ms accounts-daemon.service
        93ms systemd-fsck@dev-disk-by\x2duuid-fd099c81\x2d9f23\x2d41d3\x2da607\x2d00349d56bf3b.service
        77ms systemd-timedated.service

And I do have virtualbox installed:

$ rpm -qa | grep virtualbox
virtualbox-doc-5.0.14-0.1-omv2014.0.noarch
dkms-virtualbox-5.0.14-0.1-omv2014.0.x86_64
virtualbox-5.0.14-0.1-omv2014.0.x86_64

Edit: Off topic: Does that </> not indent succeeding lines when we post code. Doesn’t look like it to me. Do we have to manually indent every line after the first one? I’m to old for this. :confounded: Or is there a trick I’m missing? And typing a semi-colon makes smiley window pop up… :persevere:

Don’t know if there is way to get rid of ‘NetworkManager-wait-online.service’ but if there was boot time would be really fast. :cry:

Why don’t you try? :slight_smile:
As said above you have at least 2 solutions:

  • reduce the timeout setting
  • deactivate the service. The first idea
    systemctl disable NetworkManager-wait-online.service
    doesn’t work really. It is still launched by network-online.target
    So a more drastic solution is to mask the service:
    systemctl mask NetworkManager-wait-online.service

Please, see man systemctl for more infos. Trying in a vbox machine, here is what I get:

  • with NetworkManager-wait-online.service unmasked:
    Startup finished in 2.847s (kernel) + 1.045s (initrd) + 8.376s (userspace) = 12.269s
  • with NetworkManager-wait-online.service masked:
    Startup finished in 2.819s (kernel) + 1.036s (initrd) + 3.688s (userspace) = 7.544s

Be aware that might cause problems to services relying on a operational network. You can unmask NetworkManager-wait-online.service with:
systemctl unmask NetworkManager-wait-online.service

Wow…

I: 38 secs to boot (i5 in a box)
JC: 23 secs (i7 in a laptop)
Ben: 19 secs. (i5, probably not a laptop)

However, almost all the difference came with dkms.service !!! In my PC it last from 16 to 20 seconds ! And I want to remove ModemManager soon too !

The difference I can see: nVidia (proprietary) driver. I have it. I don’t know if Ben is using a nVidia card, but at least he doesn’t have the nvidia dkms module. Jean-Claude?

With others services there are almost no differences. For example:

NetworkManager
I: 7.8 secs (with an i5)
JC: 5.3 secs (i7)
Ben: 7.8 secs (i5)

plymouth-start
0.460 secs (I) vs 0.468 secs (JC) vs 0.461 secs (Ben)

But:

firewalld
7 to 12 secs vs 0.52 secs vs 0.305 secs

I think that firewalld is affected by something in my system, as it came together with other services (i blame avahi, because it’s related with networking, but it can be anyone between avahi, systemd-logind, rtkit-daemon, upower and udisksd).

And I remember the original question: why are there so many differences between boot times (measured on a single PC)?

Just for a comparison. I just downloaded the 32-bits ISO of Manjaro XFCE flavor (it’s for an old netbook; I tried to use OMV 2014.2 32.bits, but it didn’t boot up in my box :frowning: ).

Booting from a USB stick:

$ uname -a Linux manjaro 4.1.15-1-MANJARO #1 SMP PREEMPT Tue Dec 15 10:52:46 UTC 2015 i686 GNU/Linux

$ systemd-analyze time
Startup finished in 7.161s (kernel) + 9.313s (userspace) = 16.475s

$ systemd-analyze blame | head -15
5.900s pacman-init.service
5.754s mhwd-live.service
5.134s dev-sdd1.device
4.379s livecd.service
3.835s ModemManager.service
2.562s NetworkManager.service
1.325s polkit.service
1.262s systemd-vconsole-setup.service
946ms systemd-modules-load.service
830ms avahi-daemon.service
593ms systemd-tmpfiles-setup-dev.service
516ms alsa-restore.service
460ms systemd-journald.service
449ms systemd-logind.service
428ms wpa_supplicant.service

It was fast ! It was “visible faster” without using any chronometer. Even more, if I consider that I boot up from a USB drive (usually slower than booting from HDD). Anyway, it’s XFCE vs. KDE.

Kernel time, even in a totally load system as it’s my OMV (installed in June 2014, and updated, never reinstalled), with time consuming modules (as BOINC) included, is better in OMV than in Manjaro. Take into account that I’m comparing 7.16 secs kernel time in Manjaro against 6 seconds kernel+initrd time in OMV.

Both use kernel 4.1.15. Manjaro doesn’t seems to have dkms activated.

$ lspci -v 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) [...]

01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GTX 650] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device 2652
Flags: bus master, fast devsel, latency 0, IRQ 27
Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at f7000000 [disabled] [size=512K]
Capabilities:
Kernel driver in use: nouveau
Kernel modules: nouveau

[…]

Ok. Manjaro is using nouveau (logical). I’m using proprietary nVidia in OMV (which requieres dkms). Also, no VBox in Manjaro.

$ pacman -Q | grep dkms
$

No dkms package installed, as I expected.

NetworkManager (using DHCP) and ModemManager last a much less than in OMV. Also avahi. Just because it’s an “empty” run of Manjaro?? Obviously, I don’t know.

I’ll keep testing.

Actually, dkms is only required when you install a new kernel to build the external modules. If you keep the same kernel, it is safe to disable the dkms service after the first run.
But you might have first to check if the external modules are well installed. Maybe the long time taking by the dkms module in you computer is due to some inconsistency.

I’ll try soon to disconnect dkms.

Today I boot (cold boot) and immediately reboot the system. Then, after a “warm” boot:

$ systemd-analyze time Startup finished in 2.614s (kernel) + 3.409s (initrd) + 31.815s (userspace) = 37.839s

$ systemd-analyze blame | head -15
15.819s dkms.service
8.375s NetworkManager-wait-online.service
6.413s nmb.service
4.881s network-up.service
4.754s ModemManager.service
4.521s firewalld.service
2.893s plymouth-quit-wait.service
2.720s systemd-logind.service
2.704s avahi-daemon.service
2.703s upower.service
2.702s rtkit-daemon.service
2.701s udisksd.service
2.409s systemd-udev-settle.service
2.374s network.service
2.248s chronyd.service

For the moment:

.# systemctl disable laptop-mode laptop-mode.service is not a native service, redirecting to /sbin/chkconfig. Executing /sbin/chkconfig laptop-mode off

.# systemctl -l status ModemManager
ModemManager.service - Modem Manager
Loaded: loaded (/lib/systemd/system/ModemManager.service; disabled)
Active: inactive (dead)

feb 16 08:29:02 sirius2.omv systemd[1]: Starting Modem Manager…
feb 16 08:29:03 sirius2.omv ModemManager[3796]: ModemManager (version 1.2.0) starting…
feb 16 08:29:06 sirius2.omv systemd[1]: Started Modem Manager.
feb 16 08:29:08 sirius2.omv ModemManager[3796]: Couldn’t find support for device at ‘/sys/devices/pci0000:00/0000:00:1c.4/0000:03:00.0’: not supported by any plugin
feb 16 10:51:41 sirius2.omv systemd[1]: Stopping Modem Manager…
feb 16 10:51:41 sirius2.omv ModemManager[3796]: Caught signal, shutting down…
feb 16 10:51:41 sirius2.omv ModemManager[3796]: ModemManager is shut down
feb 16 10:51:41 sirius2.omv systemd[1]: Stopped Modem Manager.

.# lspci -v -s 1c.4
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Prefetchable memory behind bridge: 00000000f2100000-00000000f21fffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
Capabilities: [a0] Power Management version 2
Kernel driver in use: pcieport

(Note 1: under code I cannot use directly # to indicate a root command. That’s why I add a point before)
(Note 2: I manually stopped ModemManager at 10:51:41 from MCC; all keep working without troubles)

I’m not sure about what really means that error with device 1c.4.

Anyway, I believe that ModemManager can be disabled in my case (https://forum.antergos.com/topic/674/solved-boot-times-and-modemmanager-service/3 and ModemManager).

I set a fixed IP for the next boot.

Another quick test.

OMV Lx3, beta, build 14226, Live mode. It doesn’t use nVidia proprietary driver but nouveau.

$ systemd-analyze time Startup finished in 2.258s (kernel) + 3.461s (initrd) + 15.847s (userspace) = 21.568s

$ systemd-analyze blame | head -15
8.483s ldconfig.service
5.084s nmb.service
1.508s firewalld.service
817ms tuned.service
801ms lvm2-monitor.service
607ms systemd-hwdb-update.service
496ms fedora-readonly.service
475ms ModemManager.service
438ms fedora-loadmodules.service
397ms accounts-daemon.service
293ms systemd-localed.service
289ms systemd-journald.service
282ms avahi-daemon.service
275ms speech-dispatcherd.service
272ms tmp.mount

Well well… No dkms. FAST boot, even with DHCP and ModemManager. NetworkManager.service only last 98 milliseconds !