Journalctl takes much storage ⋅ has no config?

1

Hi,

Being on multi-boot each of my four / partition is about 30G on a SSD.
Seems it becomes short nowadays.
My UbuntuBudgie 20.04 sent me some notifications about my system being almost full - only after 3 months of not so intensive usage. So I try to tidy up here and there.

Beware snap do take much storage because of disabled versions and restore ( saved ) feature.
Taking care about these you may easily reduce by 66 % their impact on actual storage.

Now I’m a bit worried about journalctl…

django@ASGARD:~$ journalctl --disk-usage
Archived and active journals take up 1.4G in the file system.
django@ASGARD:~$

… doesn’t 1,4G seem huge for logs ? It looks like it’s stored there forever maybe.
So I read this https://askubuntu.com/questions/864722/where-is-journalctl-data-stored and only config file I found on my UBudgie 20.04 is :

django@ASGARD:~$ cat /etc/systemd/journald.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes

django@ASGARD:~$ cat /etc/systemd/journald.conf.d/*.conf
cat: '/etc/systemd/journald.conf.d/*.conf': Aucun fichier ou dossier de ce type

django@ASGARD:~$ cat /run/systemd/journald.conf.d/*.conf
cat: '/run/systemd/journald.conf.d/*.conf': Aucun fichier ou dossier de ce type

django@ASGARD:~$ cat /usr/lib/systemd/journald.conf.d/*.conf
cat: '/usr/lib/systemd/journald.conf.d/*.conf': Aucun fichier ou dossier de ce type

Would you agree that config’ file does nothing ?
Is this an expected situation ?
Is it the same on your installation⋅s ?
Is it Budgie related or any *buntu does the same ?

And of course any suggestions about « right » settings here welcome.

For the moment I symlinked and moved the /var/log folder to another HDD on a much bigger partition.

My linked journalctl folder actually contains :

django@ASGARD:~$ ls -la /media/DATA/.systems/Budgie/var/log/journal/bdd5b0e4720a4756a32ee35868b9b428
total 1581080
drwxr-sr-x+ 2 root systemd-journal     12288 juil. 11 22:36 .
drwxr-sr-x+ 3 root systemd-journal      4096 mars  22 22:16 ..
-rw-r-----+ 1 root systemd-journal   8388608 avril 10 19:42 system@0005a2f340858a00-a064e278d744f89f.journal~
-rw-r-----+ 1 root systemd-journal   8388608 juil. 11 22:36 system@0005aa306888420f-76db751e8065ddca.journal~
-rw-r-----+ 1 root systemd-journal  16777216 avril  9 16:38 system@4724011553ff4433be8ad767c416c343-0000000000036b0c-0005a2d0478d0680.journal
-rw-r-----+ 1 root systemd-journal  16777216 avril  9 23:50 system@4724011553ff4433be8ad767c416c343-000000000003957d-0005a2dc9193b0f9.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 10 16:59 system@4724011553ff4433be8ad767c416c343-000000000003e4d5-0005a2e29c18f653.journal
-rw-r-----+ 1 root systemd-journal  50331648 avril 17 20:55 system@532c7d0eedbb413c942cf20560afeddb-0000000000000001-0005a2f340830789.journal
-rw-r-----+ 1 root systemd-journal  33554432 avril 24 16:53 system@532c7d0eedbb413c942cf20560afeddb-0000000000062017-0005a3811873c15a.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 25 18:32 system@532c7d0eedbb413c942cf20560afeddb-000000000008d18b-0005a40a87a71fa4.journal
-rw-r-----+ 1 root systemd-journal  16777216 avril 27 17:26 system@532c7d0eedbb413c942cf20560afeddb-000000000008ef5b-0005a42005e8e830.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 28 16:59 system@532c7d0eedbb413c942cf20560afeddb-0000000000091fcc-0005a4475703a490.journal
-rw-r-----+ 1 root systemd-journal  25165824 mai    4 17:24 system@532c7d0eedbb413c942cf20560afeddb-0000000000094c8c-0005a45b13d0b286.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai    5 16:57 system@532c7d0eedbb413c942cf20560afeddb-000000000009bb9f-0005a4d41fd72ce8.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai    8 17:12 system@532c7d0eedbb413c942cf20560afeddb-000000000009d532-0005a4e7dccd25a0.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   10 17:28 system@532c7d0eedbb413c942cf20560afeddb-00000000000a12af-0005a5246c381b50.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   13 18:44 system@532c7d0eedbb413c942cf20560afeddb-00000000000a51be-0005a54ce33ddd76.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   15 17:13 system@532c7d0eedbb413c942cf20560afeddb-00000000000a9246-0005a58a4bd561a4.journal
-rw-r-----+ 1 root systemd-journal  25165824 mai   20 18:52 system@532c7d0eedbb413c942cf20560afeddb-00000000000aadb7-0005a5b1425b817a.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   24 23:14 system@532c7d0eedbb413c942cf20560afeddb-00000000000b6d10-0005a61739b5c50f.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   28 18:05 system@532c7d0eedbb413c942cf20560afeddb-00000000000e856c-0005a66b5b94e7fa.journal
-rw-r-----+ 1 root systemd-journal  25165824 mai   29 17:09 system@532c7d0eedbb413c942cf20560afeddb-00000000000f9471-0005a6b7811567d1.journal
-rw-r-----+ 1 root systemd-journal  58720256 juin  10 16:51 system@532c7d0eedbb413c942cf20560afeddb-00000000000fe43f-0005a6cad3875097.journal
-rw-r-----+ 1 root systemd-journal  25165824 juin  14 17:43 system@532c7d0eedbb413c942cf20560afeddb-0000000000116c62-0005a7bbfa3ee316.journal
-rw-r-----+ 1 root systemd-journal  50331648 juin  27 16:46 system@532c7d0eedbb413c942cf20560afeddb-00000000001280bc-0005a80d2d8f5280.journal
-rw-r-----+ 1 root systemd-journal  33554432 juil.  3 16:39 system@532c7d0eedbb413c942cf20560afeddb-00000000001413d5-0005a911e3603512.journal
-rw-r-----+ 1 root systemd-journal  41943040 juil. 11 12:25 system@532c7d0eedbb413c942cf20560afeddb-0000000000153dc4-0005a98a7d099b65.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil. 11 23:39 system.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil. 11 22:36 user-1000@0005aa30692302f1-f607c6b77cd4290f.journal~
-rw-r-----+ 1 root systemd-journal  41943040 juil.  3 16:39 user-1000@0a7c051ee1d84dafa6a18bf32b82d01b-00000000001415ec-0005a91271dabccd.journal
-rw-r-----+ 1 root systemd-journal 134217728 mai   24 23:14 user-1000@36161651b6664483949d138dc07f91f8-00000000000b706d-0005a617d5960d91.journal
-rw-r-----+ 1 root systemd-journal  50331648 mai   28 18:05 user-1000@36161651b6664483949d138dc07f91f8-00000000000e6ddd-0005a66b5a4e43b5.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   29 17:09 user-1000@36161651b6664483949d138dc07f91f8-00000000000f9475-0005a6b783712220.journal
-rw-r-----+ 1 root systemd-journal  33554432 juin  10 16:51 user-1000@36161651b6664483949d138dc07f91f8-00000000000fe453-0005a6cad61604b7.journal
-rw-r-----+ 1 root systemd-journal  16777216 avril  9 23:50 user-1000@4bff13df4c574316b50a7c35726f8086-00000000000396d3-0005a2dcb4442ae3.journal
-rw-r-----+ 1 root systemd-journal  25165824 avril 10 16:59 user-1000@4bff13df4c574316b50a7c35726f8086-000000000003e4fe-0005a2e29c863aef.journal
-rw-r-----+ 1 root systemd-journal  58720256 avril 17 20:55 user-1000@4bff13df4c574316b50a7c35726f8086-000000000004555e-0005a2f0fc974953.journal
-rw-r-----+ 1 root systemd-journal 117440512 avril 24 16:53 user-1000@4bff13df4c574316b50a7c35726f8086-0000000000062020-0005a3811946ec8d.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 25 18:32 user-1000@4bff13df4c574316b50a7c35726f8086-000000000008d18f-0005a40a8a2a031b.journal
-rw-r-----+ 1 root systemd-journal  16777216 avril 27 17:26 user-1000@4bff13df4c574316b50a7c35726f8086-000000000008ef6f-0005a420081633a0.journal
-rw-r-----+ 1 root systemd-journal  16777216 avril 28 16:59 user-1000@4bff13df4c574316b50a7c35726f8086-0000000000091fd4-0005a44759d4ae0b.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai    4 17:24 user-1000@4bff13df4c574316b50a7c35726f8086-0000000000094c94-0005a45b16525e75.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai    5 16:57 user-1000@4bff13df4c574316b50a7c35726f8086-000000000009bba2-0005a4d420ad3f92.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai    8 17:12 user-1000@4bff13df4c574316b50a7c35726f8086-000000000009d53c-0005a4e7df5a9981.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   10 17:28 user-1000@4bff13df4c574316b50a7c35726f8086-00000000000a12b7-0005a5246eafd9c7.journal
-rw-r-----+ 1 root systemd-journal  16777216 mai   13 18:44 user-1000@4bff13df4c574316b50a7c35726f8086-00000000000a51c4-0005a54ce5edcbb6.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   15 17:13 user-1000@4bff13df4c574316b50a7c35726f8086-00000000000a9256-0005a58a4e6a5a2f.journal
-rw-r-----+ 1 root systemd-journal  25165824 mai   20 18:52 user-1000@4bff13df4c574316b50a7c35726f8086-00000000000aadba-0005a5b1443d8a59.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril  9 16:38 user-1000@b1053316ab0440048f0af8732afaa678-00000000000370d5-0005a2d0d3084b29.journal
-rw-r-----+ 1 root systemd-journal  25165824 juil. 11 12:25 user-1000@ce07206118ad4973ad66085e54c4259f-000000000015449c-0005a98d72320839.journal
-rw-r-----+ 1 root systemd-journal  50331648 juin  14 17:43 user-1000@fd5159432d624056bdd2041c5a508a16-0000000000116ff1-0005a7bc525b0b2d.journal
-rw-r-----+ 1 root systemd-journal  41943040 juin  27 16:46 user-1000@fd5159432d624056bdd2041c5a508a16-00000000001280c0-0005a80d305296c4.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil. 11 23:39 user-1000.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 10 21:51 user-1001@0005a2f50f6556d6-a74b107205698fd1.journal~
-rw-r-----+ 1 root systemd-journal   8388608 avril  9 16:38 user-1001@08b91260ecf94f8f95f6a8cc3b5a0932-0000000000036b10-0005a2d04a64d095.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril  9 23:50 user-1001@08b91260ecf94f8f95f6a8cc3b5a0932-000000000003958f-0005a2dc925e8a8b.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 10 16:59 user-1001@08b91260ecf94f8f95f6a8cc3b5a0932-000000000003e4ce-0005a2e29c12c782.journal
-rw-r-----+ 1 root systemd-journal   8388608 juin  27 16:46 user-1001@1041304f424b49c38e448dcc28dec33c-00000000001382ac-0005a89eae5f79a7.journal
-rw-r-----+ 1 root systemd-journal   8388608 juin  14 17:43 user-1001@253c1d24ee974764b2bf0bd503a02845-000000000012454d-0005a7d3381cac73.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 24 16:53 user-1001@34a49330b9664fc19b5f19357e4c5f23-0000000000064711-0005a393942083cb.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil.  3 16:39 user-1001@38bf7f7feabe4d108bf060371e74aae5-000000000014e2f0-0005a962608e1a2d.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   20 18:52 user-1001@3e7a8c62a01940bf89c9c992211991be-00000000000b3e3f-0005a5ffba7a414f.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   28 18:05 user-1001@4e38d490508a4972aaed99059f9d83ef-00000000000f7623-0005a6a4df06cb28.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   13 18:44 user-1001@58f35710026844ba8616558c518116df-00000000000a543a-0005a54fbe82a3f4.journal
-rw-r-----+ 1 root systemd-journal  25165824 avril 17 20:55 user-1001@948a937e6a25448380d9b15b0ea12b6b-0000000000045a1d-0005a2f50f65436b.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   29 17:09 user-1001@a6de77697ecc4ffda913d345b68643ca-00000000000fc819-0005a6b917947c3d.journal
-rw-r-----+ 1 root systemd-journal   8388608 juin  10 16:51 user-1001@b4232ff5bb2d4547882b8ffe323f96db-0000000000104f2b-0005a70b4381deb4.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai    5 16:57 user-1001@ea0b2976bc6146118de28d7e4f90643d-000000000009bde1-0005a4d61c6431ab.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   28 18:05 user-1002@0ff75380b5aa40958ccd2029ed4629bc-00000000000f23bf-0005a67df6cf23bc.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 17 20:55 user-1002@265ca589a1cb4111aceaab722a2d14cb-00000000000551c2-0005a331d01d5da4.journal
-rw-r-----+ 1 root systemd-journal  16777216 juin  10 16:51 user-1002@29ad4f67f80049fa8c0f4034bc5da6e6-0000000000104241-0005a708fd21dfd2.journal
-rw-r-----+ 1 root systemd-journal   8388608 juin  14 17:43 user-1002@29ad4f67f80049fa8c0f4034bc5da6e6-0000000000116c63-0005a7bbfa8abb5d.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   20 18:52 user-1002@575c20736b054e33be253c92b9677cb1-00000000000aaebb-0005a5b14fd374e9.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   24 23:14 user-1002@575c20736b054e33be253c92b9677cb1-00000000000b6d11-0005a61739de83d0.journal
-rw-r-----+ 1 root systemd-journal   8388608 mai   29 17:09 user-1002@5d371789bc5f4f40bcfcbc89e6d92e42-00000000000fca93-0005a6b9366458ee.journal
-rw-r-----+ 1 root systemd-journal   8388608 avril 24 16:53 user-1002@7debdd8b9abb43d7b0e0f6b85e7a19ea-0000000000062f9d-0005a3913d1b24d1.journal
-rw-r-----+ 1 root systemd-journal  16777216 juin  27 16:46 user-1002@bc03fc5d541f44e0b3c1eacfbb04b697-00000000001296cf-0005a821d2e73bc4.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil.  3 16:39 user-1002@bc03fc5d541f44e0b3c1eacfbb04b697-00000000001413d6-0005a911e3abd1fc.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil. 11 12:25 user-1002@ef0ae2706fa144c5ae076c226561c4aa-0000000000153ddb-0005a98a83149c58.journal
-rw-r-----+ 1 root systemd-journal   8388608 juil. 11 19:58 user-1002.journal
django@ASGARD:~$

Older files here ( april, 9th ) seems to match my upgrade from UBudgie 19.10 to 20.04.

2

Hmm

On our downloads page our min requirements are

Processor speed: 2.4Ghz
Memory: 4GB of RAM
Hard Disk space: 60GB

The emphasis there is minimum.

Think you are going to have to actively look after your undersized space issues

Maybe something like https://unix.stackexchange.com/questions/139513/how-to-clear-journalctl

3

Since when so much space is needed for an Ubuntu ?

Note : there are no personal files here, those sit on a much bigger HDD disk. Only system and hidden files in /home/$USER sit on those 30G on SSD.

I really wonder here why journalctl did not clean itself and why it’s config file seems to actually config nothing at all.

Is this the normal expected default situation ?
Shouldn’t the default config be less greedy ?

I may indeed organize differently my partitions on SSD this is not the real problem.

4

As far as I can gather this is normal behaviour.

Systemd manages its resources by itself

So if resources are low it starts to tidy up itself.

The config is there to allow you to manually tweak stuff rather than accept the defaults.

5

This is somehow « annoying » :

if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either.

…and explain how it can grow big.

As usual, thanks for the links. I’ll do a bit of vacuum and maybe set some limits.

Doing so ( removing unused snaps and debs, moving /var/log to another disk ), I went from

django@ASGARD:~$ df -Th | grep -v loop*
Sys. de fichiers Type     Taille Utilisé Dispo Uti% Monté sur
(…)
/dev/sda1        ext4        24G     21G  2,0G  92% /
(…)
django@ASGARD:~$

to

/dev/sda1        ext4        24G     17G  5,4G  76% /
6

If you really want to use Linux on such a small drive then you may want to consider a reinstall but use btrfs as the file system rather than ext4.

Apparently btrfs (and zfs) can be configured to use transparent disk compression

7

No, no I will resize where I can.

I still have my 16.04 partition that will die in some months - which holds 25G after many years of use, and many adventures, hence my surprise here with 20.04 holding as much only after some months of quiet « home » usage.

I should learn myself to do virtualization for testing distro, but I don’t feel confident and educated enough about it. I guess gnome-boxes might be enough for me but only time I’ve tried it ( in 19.10 ) it coincided with an unexpected bios-reset and motherboard hard-reset… ( probably not related. Hope so at least ).

8

I always specify the #SystemMaxFileSize=500M (or something not too big) as it usually is the same file that grows to a large extent. Specifying this is therefore enough to control the invading nature of that journal file.

I see little interest in keeping big journal files. At some point I had files of over 4 Gb and it really started to take space on a 40 Gb / partition (which is largely enough for the system alone).

I don’t understand why it’s not limited by default to a decent value.

9

Yes to that ↑ I may understand why one wants to keep full journal on server, but on a personal computer ? @fossfreedom shouldn’t UBudgie come with some sane defaults settings there ?

10

It’s the same package for all ubuntu flavours.

We cannot change the package for ourselves.

The best advice I would suggest is to ubuntu-bug systemd and ask just this question if desktops can have a separate config from servers

11

Does it make sense ?

12
13

journald.conf will not remove everything even when configuring it to a smaller max size. It will still eat up the space of (the result of) that journalctl --disk-usage command.

A manual way to clean (vacuum) your log files is

sudo journalctl vacuum-time=15d

(for anything older than 15 days)

Or

sudo journalctl vacuum-size=100M

(to clean anything above the last 100MB)

That first command made me go from 3.8 Gb to 41 Mb. And my journald.conf was set to max 1G. I was not aware it was still eating up space so much.

Would be interesting to make a scheduler that runs these frequently and automatically.

14

Workaround for the moment : I symlinked /var/log/journal onto another disc/partition.

It’s stupid how many people think storage is free / cheap / unlimited anywhere.

And even if it was true, is it necessary to use so much storage for journal on a personal computer ?

Easier storage availability should not prevent from optimizing storage usage.