Docker: Monitor Docker using CheckMk
CheckMk is also a great tool, not just for normal systems, but also for Docker installations.
Especially when running Docker in a production environment, it's highly recommended to monitor all the services.
Fortunately, there's a Docker plugin for CheckMk.
Prerequisites
CheckMk Plugin for Docker
First, you'll have to install the Docker plugin for CheckMk. You'll find it under Setup -> Linux -> Plugins -> mk_docker.py.
Copy it to the plugin directory:
/usr/lib/check_mk_agent/pluginsDocker module for Python
Then, you need the Python module docker. Depending on your system, there are several ways to install it:
# yum install python3-docker
# apt install python3-docker
# python -m pip install dockerCheck plugin
Let's check if the plugin works:
# su -
# cd /usr/lib/check_mk_agent/plugins
# ./mk_docker.py<<<docker_node_info:sep(124)>>>
@docker_version_info|{"PluginVersion": "0.1", "DockerPyVersion": "7.1.0", "ApiVersion": "1.45"}
<<<docker_node_info:sep(0)>>>
{"ID": "4df2dab0-d701-4b5e-a47d-57da81be62e6", "Containers": 7, "ContainersRunning": 7, "ContainersPaused": 0, "ContainersStopped": 0, "Images": 10, "Driver": "overlay2", "DriverStatus": [["Backing Filesystem", "extfs"], ["Supports d_type", "true"], ["Using metacopy", "false"], ["Native Overlay Diff", "true"], ["userxattr", "false"]], "Plugins": {"Volume": ["local"], "Network": ["bridge", "host", "ipvlan", "macvlan", "null", "overlay"], "Authorization": null, "Log": ["awslogs", "fluentd", "gcplogs", "gelf", "journald", "json-file", "local", "splunk", "syslog"]}, "MemoryLimit": false, "SwapLimit": false, "CpuCfsPeriod": true, "CpuCfsQuota": true, "CPUShares": true, "CPUSet": true, "PidsLimit": true, "IPv4Forwarding": true, "BridgeNfIptables": true, "BridgeNfIp6tables": true, "Debug": false, "NFd": 107, "OomKillDisable": false, "NGoroutines": 111, "SystemTime": "2025-01-31T10:37:24.177647149+01:00", "LoggingDriver": "json-file", "CgroupDriver": "systemd", "CgroupVersion": "2", "NEventsListener": 0, "KernelVersion": "6.6.67-v8+", "OperatingSystem": "Debian GNU/Linux trixie/sid", "OSVersion": "", "OSType": "linux", "Architecture": "aarch64", "IndexServerAddress": "https://index.docker.io/v1/", "RegistryConfig": {"AllowNondistributableArtifactsCIDRs": null, "AllowNondistributableArtifactsHostnames": null, "InsecureRegistryCIDRs": ["127.0.0.0/8"], "IndexConfigs": {"docker.io": {"Name": "docker.io", "Mirrors": [], "Secure": true, "Official": true}}, "Mirrors": null}, "NCPU": 4, "MemTotal": 3982454784, "GenericResources": null, "DockerRootDir": "/var/lib/docker", "HttpProxy": "", "HttpsProxy": "", "NoProxy": "", "Name": "pi4", "Labels": [], "ExperimentalBuild": false, "ServerVersion": "26.1.5+dfsg1", "Runtimes": {"io.containerd.runc.v2": {"path": "runc", "status": {"org.opencontainers.runtime-spec.features": "{\"ociVersionMin\":\"1.0.0\",\"ociVersionMax\":\"1.2.0\",\"hooks\":[\"prestart\",\"createRuntime\",\"createContainer\",\"startContainer\",\"poststart\",\"poststop\"],\"mountOptions\":[\"acl\",\"async\",\"atime\",\"bind\",\"defaults\",\"dev\",\"diratime\",\"dirsync\",\"exec\",\"iversion\",\"lazytime\",\"loud\",\"mand\",\"noacl\",\"noatime\",\"nodev\",\"nodiratime\",\"noexec\",\"noiversion\",\"nolazytime\",\"nomand\",\"norelatime\",\"nostrictatime\",\"nosuid\",\"nosymfollow\",\"private\",\"ratime\",\"rbind\",\"rdev\",\"rdiratime\",\"relatime\",\"remount\",\"rexec\",\"rnoatime\",\"rnodev\",\"rnodiratime\",\"rnoexec\",\"rnorelatime\",\"rnostrictatime\",\"rnosuid\",\"rnosymfollow\",\"ro\",\"rprivate\",\"rrelatime\",\"rro\",\"rrw\",\"rshared\",\"rslave\",\"rstrictatime\",\"rsuid\",\"rsymfollow\",\"runbindable\",\"rw\",\"shared\",\"silent\",\"slave\",\"strictatime\",\"suid\",\"symfollow\",\"sync\",\"tmpcopyup\",\"unbindable\"],\"linux\":{\"namespaces\":[\"cgroup\",\"ipc\",\"mount\",\"network\",\"pid\",\"user\",\"uts\"],\"capabilities\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_DAC_READ_SEARCH\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_SETGID\",\"CAP_SETUID\",\"CAP_SETPCAP\",\"CAP_LINUX_IMMUTABLE\",\"CAP_NET_BIND_SERVICE\",\"CAP_NET_BROADCAST\",\"CAP_NET_ADMIN\",\"CAP_NET_RAW\",\"CAP_IPC_LOCK\",\"CAP_IPC_OWNER\",\"CAP_SYS_MODULE\",\"CAP_SYS_RAWIO\",\"CAP_SYS_CHROOT\",\"CAP_SYS_PTRACE\",\"CAP_SYS_PACCT\",\"CAP_SYS_ADMIN\",\"CAP_SYS_BOOT\",\"CAP_SYS_NICE\",\"CAP_SYS_RESOURCE\",\"CAP_SYS_TIME\",\"CAP_SYS_TTY_CONFIG\",\"CAP_MKNOD\",\"CAP_LEASE\",\"CAP_AUDIT_WRITE\",\"CAP_AUDIT_CONTROL\",\"CAP_SETFCAP\",\"CAP_MAC_OVERRIDE\",\"CAP_MAC_ADMIN\",\"CAP_SYSLOG\",\"CAP_WAKE_ALARM\",\"CAP_BLOCK_SUSPEND\",\"CAP_AUDIT_READ\",\"CAP_PERFMON\",\"CAP_BPF\",\"CAP_CHECKPOINT_RESTORE\"],\"cgroup\":{\"v1\":true,\"v2\":true,\"systemd\":true,\"systemdUser\":true},\"seccomp\":{\"enabled\":true,\"actions\":[\"SCMP_ACT_ALLOW\",\"SCMP_ACT_ERRNO\",\"SCMP_ACT_KILL\",\"SCMP_ACT_KILL_PROCESS\",\"SCMP_ACT_KILL_THREAD\",\"SCMP_ACT_LOG\",\"SCMP_ACT_NOTIFY\",\"SCMP_ACT_TRACE\",\"SCMP_ACT_TRAP\"],\"operators\":[\"SCMP_CMP_EQ\",\"SCMP_CMP_GE\",\"SCMP_CMP_GT\",\"SCMP_CMP_LE\",\"SCMP_CMP_LT\",\"SCMP_CMP_MASKED_EQ\",\"SCMP_CMP_NE\"],\"archs\":[\"SCMP_ARCH_AARCH64\",\"SCMP_ARCH_ARM\",\"SCMP_ARCH_MIPS\",\"SCMP_ARCH_MIPS64\",\"SCMP_ARCH_MIPS64N32\",\"SCMP_ARCH_MIPSEL\",\"SCMP_ARCH_MIPSEL64\",\"SCMP_ARCH_MIPSEL64N32\",\"SCMP_ARCH_PPC\",\"SCMP_ARCH_PPC64\",\"SCMP_ARCH_PPC64LE\",\"SCMP_ARCH_RISCV64\",\"SCMP_ARCH_S390\",\"SCMP_ARCH_S390X\",\"SCMP_ARCH_X32\",\"SCMP_ARCH_X86\",\"SCMP_ARCH_X86_64\"]},\"apparmor\":{\"enabled\":true},\"selinux\":{\"enabled\":true}},\"annotations\":{\"io.github.seccomp.libseccomp.version\":\"2.5.5\",\"org.opencontainers.runc.checkpoint.enabled\":\"true\",\"org.opencontainers.runc.commit\":\"1.1.15+ds1-1+b1\",\"org.opencontainers.runc.version\":\"1.1.15+ds1\"}}"}}, "runc": {"path": "runc", "status": {"org.opencontainers.runtime-spec.features": "{\"ociVersionMin\":\"1.0.0\",\"ociVersionMax\":\"1.2.0\",\"hooks\":[\"prestart\",\"createRuntime\",\"createContainer\",\"startContainer\",\"poststart\",\"poststop\"],\"mountOptions\":[\"acl\",\"async\",\"atime\",\"bind\",\"defaults\",\"dev\",\"diratime\",\"dirsync\",\"exec\",\"iversion\",\"lazytime\",\"loud\",\"mand\",\"noacl\",\"noatime\",\"nodev\",\"nodiratime\",\"noexec\",\"noiversion\",\"nolazytime\",\"nomand\",\"norelatime\",\"nostrictatime\",\"nosuid\",\"nosymfollow\",\"private\",\"ratime\",\"rbind\",\"rdev\",\"rdiratime\",\"relatime\",\"remount\",\"rexec\",\"rnoatime\",\"rnodev\",\"rnodiratime\",\"rnoexec\",\"rnorelatime\",\"rnostrictatime\",\"rnosuid\",\"rnosymfollow\",\"ro\",\"rprivate\",\"rrelatime\",\"rro\",\"rrw\",\"rshared\",\"rslave\",\"rstrictatime\",\"rsuid\",\"rsymfollow\",\"runbindable\",\"rw\",\"shared\",\"silent\",\"slave\",\"strictatime\",\"suid\",\"symfollow\",\"sync\",\"tmpcopyup\",\"unbindable\"],\"linux\":{\"namespaces\":[\"cgroup\",\"ipc\",\"mount\",\"network\",\"pid\",\"user\",\"uts\"],\"capabilities\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_DAC_READ_SEARCH\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_SETGID\",\"CAP_SETUID\",\"CAP_SETPCAP\",\"CAP_LINUX_IMMUTABLE\",\"CAP_NET_BIND_SERVICE\",\"CAP_NET_BROADCAST\",\"CAP_NET_ADMIN\",\"CAP_NET_RAW\",\"CAP_IPC_LOCK\",\"CAP_IPC_OWNER\",\"CAP_SYS_MODULE\",\"CAP_SYS_RAWIO\",\"CAP_SYS_CHROOT\",\"CAP_SYS_PTRACE\",\"CAP_SYS_PACCT\",\"CAP_SYS_ADMIN\",\"CAP_SYS_BOOT\",\"CAP_SYS_NICE\",\"CAP_SYS_RESOURCE\",\"CAP_SYS_TIME\",\"CAP_SYS_TTY_CONFIG\",\"CAP_MKNOD\",\"CAP_LEASE\",\"CAP_AUDIT_WRITE\",\"CAP_AUDIT_CONTROL\",\"CAP_SETFCAP\",\"CAP_MAC_OVERRIDE\",\"CAP_MAC_ADMIN\",\"CAP_SYSLOG\",\"CAP_WAKE_ALARM\",\"CAP_BLOCK_SUSPEND\",\"CAP_AUDIT_READ\",\"CAP_PERFMON\",\"CAP_BPF\",\"CAP_CHECKPOINT_RESTORE\"],\"cgroup\":{\"v1\":true,\"v2\":true,\"systemd\":true,\"systemdUser\":true},\"seccomp\":{\"enabled\":true,\"actions\":[\"SCMP_ACT_ALLOW\",\"SCMP_ACT_ERRNO\",\"SCMP_ACT_KILL\",\"SCMP_ACT_KILL_PROCESS\",\"SCMP_ACT_KILL_THREAD\",\"SCMP_ACT_LOG\",\"SCMP_ACT_NOTIFY\",\"SCMP_ACT_TRACE\",\"SCMP_ACT_TRAP\"],\"operators\":[\"SCMP_CMP_EQ\",\"SCMP_CMP_GE\",\"SCMP_CMP_GT\",\"SCMP_CMP_LE\",\"SCMP_CMP_LT\",\"SCMP_CMP_MASKED_EQ\",\"SCMP_CMP_NE\"],\"archs\":[\"SCMP_ARCH_AARCH64\",\"SCMP_ARCH_ARM\",\"SCMP_ARCH_MIPS\",\"SCMP_ARCH_MIPS64\",\"SCMP_ARCH_MIPS64N32\",\"SCMP_ARCH_MIPSEL\",\"SCMP_ARCH_MIPSEL64\",\"SCMP_ARCH_MIPSEL64N32\",\"SCMP_ARCH_PPC\",\"SCMP_ARCH_PPC64\",\"SCMP_ARCH_PPC64LE\",\"SCMP_ARCH_RISCV64\",\"SCMP_ARCH_S390\",\"SCMP_ARCH_S390X\",\"SCMP_ARCH_X32\",\"SCMP_ARCH_X86\",\"SCMP_ARCH_X86_64\"]},\"apparmor\":{\"enabled\":true},\"selinux\":{\"enabled\":true}},\"annotations\":{\"io.github.seccomp.libseccomp.version\":\"2.5.5\",\"org.opencontainers.runc.checkpoint.enabled\":\"true\",\"org.opencontainers.runc.commit\":\"1.1.15+ds1-1+b1\",\"org.opencontainers.runc.version\":\"1.1.15+ds1\"}}"}}}, "DefaultRuntime": "runc", "Swarm": {"NodeID": "", "NodeAddr": "", "LocalNodeState": "inactive", "ControlAvailable": false, "Error": "", "RemoteManagers": null}, "LiveRestoreEnabled": false, "Isolation": "", "InitBinary": "docker-init", "ContainerdCommit": {"ID": "1.7.24~ds1-4", "Expected": "1.7.24~ds1-4"}, "RuncCommit": {"ID": "1.1.15+ds1-1+b1", "Expected": "1.1.15+ds1-1+b1"}, "InitCommit": {"ID": "", "Expected": ""}, "SecurityOptions": ["name=seccomp,profile=builtin", "name=cgroupns"], "CDISpecDirs": [], "Warnings": ["WARNING: No memory limit support", "WARNING: No swap limit support"]}
[...]
<<<<f9623127cf20>>>>
<<<docker_container_diskstat:sep(124)>>>
@docker_version_info|{"PluginVersion": "0.1", "DockerPyVersion": "7.1.0", "ApiVersion": "1.45"}
<<<docker_container_diskstat:sep(0)>>>
{"io_service_bytes_recursive": [{"major": 8, "minor": 0, "op": "read", "value": 5451776}, {"major": 8, "minor": 0, "op": "write", "value": 774144}, {"major": 179, "minor": 0, "op": "read", "value": 117915648}, {"major": 179, "minor": 0, "op": "write", "value": 0}], "io_serviced_recursive": null, "io_queue_recursive": null, "io_service_time_recursive": null, "io_wait_time_recursive": null, "io_merged_recursive": null, "io_time_recursive": null, "sectors_recursive": null, "time": 1738316246.7696357, "names": {"7:1": "loop1", "1:2": "ram2", "1:0": "ram0", "1:9": "ram9", "179:0": "mmcblk0", "7:6": "loop6", "1:14": "ram14", "1:7": "ram7", "7:4": "loop4", "1:12": "ram12", "1:5": "ram5", "7:2": "loop2", "1:10": "ram10", "1:3": "ram3", "7:0": "loop0", "1:1": "ram1", "7:7": "loop7", "1:15": "ram15", "8:0": "sda", "1:8": "ram8", "7:5": "loop5", "1:13": "ram13", "1:6": "ram6", "7:3": "loop3", "1:11": "ram11", "1:4": "ram4"}}
<<<<>>>>
<<<<492f8bcf463d>>>>
<<<docker_container_diskstat:sep(124)>>>
@docker_version_info|{"PluginVersion": "0.1", "DockerPyVersion": "7.1.0", "ApiVersion": "1.45"}
<<<docker_container_diskstat:sep(0)>>>
{"io_service_bytes_recursive": [{"major": 179, "minor": 0, "op": "read", "value": 43995136}, {"major": 179, "minor": 0, "op": "write", "value": 0}], "io_serviced_recursive": null, "io_queue_recursive": null, "io_service_time_recursive": null, "io_wait_time_recursive": null, "io_merged_recursive": null, "io_time_recursive": null, "sectors_recursive": null, "time": 1738316246.7751245, "names": {"7:1": "loop1", "1:2": "ram2", "1:0": "ram0", "1:9": "ram9", "179:0": "mmcblk0", "7:6": "loop6", "1:14": "ram14", "1:7": "ram7", "7:4": "loop4", "1:12": "ram12", "1:5": "ram5", "7:2": "loop2", "1:10": "ram10", "1:3": "ram3", "7:0": "loop0", "1:1": "ram1", "7:7": "loop7", "1:15": "ram15", "8:0": "sda", "1:8": "ram8", "7:5": "loop5", "1:13": "ram13", "1:6": "ram6", "7:3": "loop3", "1:11": "ram11", "1:4": "ram4"}}
<<<<>>>>
<<<<93aecad39239>>>>
<<<docker_container_diskstat:sep(124)>>>
@docker_version_info|{"PluginVersion": "0.1", "DockerPyVersion": "7.1.0", "ApiVersion": "1.45"}
<<<docker_container_diskstat:sep(0)>>>
{"io_service_bytes_recursive": [{"major": 179, "minor": 0, "op": "read", "value": 7680000}, {"major": 179, "minor": 0, "op": "write", "value": 0}, {"major": 8, "minor": 0, "op": "read", "value": 21774336}, {"major": 8, "minor": 0, "op": "write", "value": 8716984320}], "io_serviced_recursive": null, "io_queue_recursive": null, "io_service_time_recursive": null, "io_wait_time_recursive": null, "io_merged_recursive": null, "io_time_recursive": null, "sectors_recursive": null, "time": 1738316246.771221, "names": {"7:1": "loop1", "1:2": "ram2", "1:0": "ram0", "1:9": "ram9", "179:0": "mmcblk0", "7:6": "loop6", "1:14": "ram14", "1:7": "ram7", "7:4": "loop4", "1:12": "ram12", "1:5": "ram5", "7:2": "loop2", "1:10": "ram10", "1:3": "ram3", "7:0": "loop0", "1:1": "ram1", "7:7": "loop7", "1:15": "ram15", "8:0": "sda", "1:8": "ram8", "7:5": "loop5", "1:13": "ram13", "1:6": "ram6", "7:3": "loop3", "1:11": "ram11", "1:4": "ram4"}}
<<<<>>>>Configure CheckMk
Go to Setup -> Hosts and choose the yellow box for the host where the plugin was deployed to run service discovery. Then we already see the new services:
The easiest way is to
and then
After that, accept the changes:
CheckMk needs some time to collect all the data,
but after a few minutes, everything is fine:
But, please keep in mind, that after changing something inside Docker, you'll have to remove outdated and implement new services.