Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-2b13f1cc4cb8386a5ff1323cd7b3487cbdd2060e237e1d767301e07f1238f867.scope/cpuacct.stat', container doesn't run

[ghost]

I am running zabbix agent with the zabbix_docker_monitoring.so module. The agent can "see" the cdocker conatiner running however it can not read any of the metrics files. I have attached a clean log file from the last attempt and debug level is set to 4.

zabbix_agentd.log

[jangaraj]

Please try to compile/use latest master version (v0.6.8). There is a fix #80, which may also fix your problem.

[ghost]

Thanks for the advice and information. I first compliled v0.6.8 and used the new binary derived from that. Unfortunately I got the same issue. I then used your master branch, where the code from issue 80 had been merged to. I also got the same issue there.

I spun up another container for the tests above:

19353:20171116:134135.281 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-2b13f1cc4cb8386a5ff1323cd7b3487cbdd2060e237e1d767301e07f1238f867.scope/cpuacct.stat'
 19355:20171116:135112.254 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-1c76382cd7348f344971ac0865e5b247a6886f850c068d61aa2213bc564ea62a.scope/cpuacct.stat'
 19354:20171116:135113.051 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-1c76382cd7348f344971ac0865e5b247a6886f850c068d61aa2213bc564ea62a.scope/memory.stat'
 19354:20171116:135114.186 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-1c76382cd7348f344971ac0865e5b247a6886f850c068d61aa2213bc564ea62a.scope/memory.stat'
 19354:20171116:135115.228 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-1c76382cd7348f344971ac0865e5b247a6886f850c068d61aa2213bc564ea62a.scope/memory.stat'
 19354:20171116:135119.220 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-2b13f1cc4cb8386a5ff1323cd7b3487cbdd2060e237e1d767301e07f1238f867.scope/cpuacct.stat'
 19354:20171116:135120.292 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-2b13f1cc4cb8386a5ff1323cd7b3487cbdd2060e237e1d767301e07f1238f867.scope/memory.stat'
 19354:20171116:135121.318 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-2b13f1cc4cb8386a5ff1323cd7b3487cbdd2060e237e1d767301e07f1238f867.scope/memory.stat'
 19354:20171116:135122.342 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-2b13f1cc4cb8386a5ff1323cd7b3487cbdd2060e237e1d767301e07f1238f867.scope/memory.stat'
 19355:20171116:135132.667 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-dd9bc138341a7e0af617c2b067ac4f7e8bb442cf10872b170dc0eff87f1c5022.scope/cpuacct.stat'
 19354:20171116:135133.705 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-dd9bc138341a7e0af617c2b067ac4f7e8bb442cf10872b170dc0eff87f1c5022.scope/memory.stat'
 19355:20171116:135134.733 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-dd9bc138341a7e0af617c2b067ac4f7e8bb442cf10872b170dc0eff87f1c5022.scope/memory.stat'
 19354:20171116:135135.769 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-dd9bc138341a7e0af617c2b067ac4f7e8bb442cf10872b170dc0eff87f1c5022.scope/memory.stat'
 19355:20171116:135139.882 Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-c2215070ecba5736371de18f5e83956e0897bc18842b3934f59d75819dc44302.scope/cpuacct.stat'
 19355:20171116:135140.925 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-c2215070ecba5736371de18f5e83956e0897bc18842b3934f59d75819dc44302.scope/memory.stat'
 19354:20171116:135141.940 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-c2215070ecba5736371de18f5e83956e0897bc18842b3934f59d75819dc44302.scope/memory.stat'
 19354:20171116:135142.957 Cannot open metric file: '/sys/fs/cgroup/memory/system.slice/docker-c2215070ecba5736371de18f5e83956e0897bc18842b3934f59d75819dc44302.scope/memory.stat'

Note I am running this on Red Hat 7. Appreciate any further advice you might have.

[jangaraj]

Thanks for the info. I need to debug it. Could you provide an output of these commands, please?

$ cat /etc/*release*
$ uname -a
$ docker info
$ find /sys/fs/cgroup/cpu,cpuacct/ -name "*docker*"
$ ls -lah /sys/fs/cgroup/cpu,cpuacct/docker

Do you have any special setup of your cgroups?

Also could you try dockbix agent, please?

$ docker run \
  --name=dockbix-agent-xxl \
  --net=host \
  --privileged \
  -v /:/rootfs \
  -v /var/run:/var/run \
  --restart unless-stopped \
  -e "ZA_Server=127.0.0.1" \
  -d monitoringartist/dockbix-agent-xxl-limited:latest
$ docker run \
  --rm -ti \
  --net host \
  --entrypoint=zabbix_get \
  monitoringartist/dockbix-agent-xxl-limited:latest \
  -s 127.0.0.1 -k docker.mem[/dockbix-agent-xxl,rss]

[ghost]

Thanks for the information. I will work on the pieces above and get back to you.

[ghost]

Here is the information for your debugging. Apologies for the delay on responding on this piece:

 cat /etc/*release*
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Red Hat Enterprise Linux Server release 7.4 (Maipo)
cpe:/o:redhat:enterprise_linux:7.4:ga:server

uname -a
Linux lxv5225 3.10.0-693.2.2.el7.x86_64 #1 SMP Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

docker info
Containers: 15
 Running: 1
 Paused: 0
 Stopped: 14
Images: 6
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: vg_redhat-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 3.847 GB
 Data Space Total: 6.853 GB
 Data Space Available: 3.006 GB
 Metadata Space Used: 1.417 MB
 Metadata Space Total: 163.6 MB
 Metadata Space Available: 162.2 MB
 Thin Pool Minimum Free Space: 685.2 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-693.2.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 4
Total Memory: 15.51 GiB
Name: lxv5225
ID: 2NMF:NBL4:GFKI:NYL2:5QNZ:ETJN:43HN:GYTU:Y5CC:55G5:7K7Q:EJOU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
Registries: docker.io (secure)


find /sys/fs/cgroup/cpu,cpuacct/ -name "*docker*"
/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-c2215070ecba5736371de18f5e83956e0897bc18842b3934f59d75819dc44302.scope
/sys/fs/cgroup/cpu,cpuacct/system.slice/docker.service

ls -lah /sys/fs/cgroup/cpu
lrwxrwxrwx. 1 root root 11 Nov 16 11:47 /sys/fs/cgroup/cpu -> cpu,cpuacct

 ls -lah /sys/fs/cgroup/cpu,cpuacct/docker
ls: cannot access /sys/fs/cgroup/cpu,cpuacct/docker: No such file or directory

I will try out dockbix this week.

[ghost]

Additionally using the docker comands above got error:
ZBX_NOTSUPPORTED: Cannot open memory.stat file

[jangaraj]

Weird. Path from the find command vs zabbix module path looks ok:
MODULE: /sys/fs/cgroup/cpu,cpuacct/system.slice/docker- 1c76382cd7348f344971ac0865e5b247a6886f850c068d61aa2213bc564ea62a.scope/cpuacct.stat
FIND: /sys/fs/cgroup/cpu,cpuacct/system.slice/docker- c2215070ecba5736371de18f5e83956e0897bc18842b3934f59d75819dc44302.scope/