[BUG] Unable to open boltdb.

[sbcrumb]

Describe the bug
New Install with docker-run or docker compose and I get an error about boltdb

Log Files
docker run -it --rm -p 8080:8080 -p 8086:8086 -v pwd/scrutiny:/opt/scrutiny/config -v pwd/influxdb2:/opt/scrutiny/influxdb -v /run/udev:/run/udev:ro --cap-add SYS_RAWIO --device=/dev/sda --device=/dev/sdb --name scrutiny ghcr.io/analogj/scrutiny:master-omnibus [s6-init] making user provided files available at /var/run/s6/etc...exited 0. [s6-init] ensuring user provided files have correct perms...exited 0. [fix-attrs.d] applying ownership & permissions fixes... [fix-attrs.d] done. [cont-init.d] executing container initialization scripts... [cont-init.d] 01-timezone: executing... [cont-init.d] 01-timezone: exited 0. [cont-init.d] 50-cron-config: executing... [cont-init.d] 50-cron-config: exited 0. [cont-init.d] done. [services.d] starting services waiting for influxdb waiting for scrutiny service to start starting cron [services.d] done. starting influxdb influxdb not ready scrutiny api not ready 2023-11-05T15:31:11.801156Z info Welcome to InfluxDB {"log_id": "0lK1aaEG000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:11.839344Z error Failed opening bolt {"log_id": "0lK1aaEG000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:12.840054Z info Welcome to InfluxDB {"log_id": "0lK1aeHl000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:12.840476Z error Failed opening bolt {"log_id": "0lK1aeHl000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:13.847608Z info Welcome to InfluxDB {"log_id": "0lK1aiDl000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:13.847852Z error Failed opening bolt {"log_id": "0lK1aiDl000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:14.851747Z info Welcome to InfluxDB {"log_id": "0lK1am8l000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:14.852174Z error Failed opening bolt {"log_id": "0lK1am8l000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:15.845124Z info Welcome to InfluxDB {"log_id": "0lK1aq1G000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:15.845525Z error Failed opening bolt {"log_id": "0lK1aq1G000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb influxdb not ready scrutiny api not ready 2023-11-05T15:31:16.827357Z info Welcome to InfluxDB {"log_id": "0lK1atrl000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:16.827633Z error Failed opening bolt {"log_id": "0lK1atrl000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:17.843753Z info Welcome to InfluxDB {"log_id": "0lK1axpl000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:17.844179Z error Failed opening bolt {"log_id": "0lK1axpl000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:18.858097Z info Welcome to InfluxDB {"log_id": "0lK1b0nW000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:18.858566Z error Failed opening bolt {"log_id": "0lK1b0nW000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:19.855647Z info Welcome to InfluxDB {"log_id": "0lK1b4gl000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:19.855969Z error Failed opening bolt {"log_id": "0lK1b4gl000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device See 'influxd -h' for help influxdb config file already exists. skipping. starting influxdb 2023-11-05T15:31:20.832154Z info Welcome to InfluxDB {"log_id": "0lK1b8W0000", "version": "v2.2.0", "commit": "a2f8538837", "build_date": "2022-04-06T17:36:40Z"} 2023-11-05T15:31:20.832702Z error Failed opening bolt {"log_id": "0lK1b8W0000", "error": "unable to open boltdb: no such device"} Error: unable to open boltdb: no such device

The log files will be available on your host in the config directory. Please attach them to this issue.

Please also provide the output of docker info

`Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 5
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc version: v1.1.9-0-gccaecfc
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-0.deb11.11-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.48GiB
Name: deathstar
ID: d1d0ebec-0c54-4364-b38c-deea68d38ea2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false`

[AnalogJ]

Thats super strange. Never seen this issue before. Did you get this fixed or is this still happening?

[sbcrumb]

Still happening.

[delize]

@AnalogJ this has started to happen for me as well in the past 12 days:

 scrutiny:
        container_name: scrutiny
        image: ghcr.io/analogj/scrutiny:master-omnibus
        privileged: true
        cap_add:
            - SYS_RAWIO
            - SYS_ADMIN
        # ports:
        #     - "8082:8080" # webapp
            # - "8087:8086" # influxDB admin
        environment:
            - SCRUTINY_WEB_INFLUXDB_INIT_USERNAME=${SCRUTINY_WEB_INFLUXDB_INIT_USERNAME}
            - SCRUTINY_WEB_INFLUXDB_INIT_PASSWORD=${SCRUTINY_WEB_INFLUXDB_INIT_PASSWORD}
        volumes:
            - /run/udev:/run/udev:ro
            - ${BASE_PATH}/scrutiny/config:/opt/scrutiny/config
            - ${BASE_PATH}/influxdb:/opt/scrutiny/influxdb
        restart: 'unless-stopped'
        devices:
            - "/dev/sda"
            - "/dev/sdb"
            - "/dev/sdc"
            - "/dev/sdd"
            - "/dev/sde"
            - "/dev/sdf"
            - "/dev/sdg"
            - "/dev/sdh"
            - "/dev/nvme0"
            - "/dev/nvme1"
        networks:
            - lame
            - backend

Resulting in:

$ docker logs -f scrutiny                                                                                   [15:12:54]
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/01-timezone
cont-init: info: /etc/cont-init.d/01-timezone exited 0
cont-init: info: running /etc/cont-init.d/50-cron-config
cont-init: info: /etc/cont-init.d/50-cron-config exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun collector-once (no readiness notification)
services-up: info: copying legacy longrun cron (no readiness notification)
services-up: info: copying legacy longrun influxdb (no readiness notification)
services-up: info: copying legacy longrun scrutiny (no readiness notification)
s6-rc: info: service legacy-services successfully started
waiting for influxdb
starting cron
waiting for scrutiny service to start
influxdb config file already exists. skipping.
starting influxdb
influxdb not ready
scrutiny api not ready
ts=2024-08-11T13:12:53.305972Z lvl=info msg="Welcome to InfluxDB" log_id=0qxOyaEG000 version=v2.2.0 commit=a2f8538837 build_date=2022-04-06T17:36:40Z
ts=2024-08-11T13:12:54.259152Z lvl=error msg="Failed opening bolt" log_id=0qxOyaEG000 error="unable to open boltdb: timeout"
Error: unable to open boltdb: timeout
See 'influxd -h' for help
influxdb config file already exists. skipping.
starting influxdb
ts=2024-08-11T13:12:55.369567Z lvl=info msg="Welcome to InfluxDB" log_id=0qxOyiIG000 version=v2.2.0 commit=a2f8538837 build_date=2022-04-06T17:36:40Z
ts=2024-08-11T13:12:56.322736Z lvl=error msg="Failed opening bolt" log_id=0qxOyiIG000 error="unable to open boltdb: timeout"
Error: unable to open boltdb: timeout
See 'influxd -h' for help
influxdb config file already exists. skipping.
starting influxdb
ts=2024-08-11T13:12:57.431215Z lvl=info msg="Welcome to InfluxDB" log_id=0qxOyqLl000 version=v2.2.0 commit=a2f8538837 build_date=2022-04-06T17:36:40Z
influxdb not ready
scrutiny api not ready
ts=2024-08-11T13:12:58.385093Z lvl=error msg="Failed opening bolt" log_id=0qxOyqLl000 error="unable to open boltdb: timeout"
Error: unable to open boltdb: timeout
See 'influxd -h' for help
influxdb config file already exists. skipping.
starting influxdb
ts=2024-08-11T13:12:59.514165Z lvl=info msg="Welcome to InfluxDB" log_id=0qxOyyUW000 version=v2.2.0 commit=a2f8538837 build_date=2022-04-06T17:36:40Z
ts=2024-08-11T13:13:00.468258Z lvl=error msg="Failed opening bolt" log_id=0qxOyyUW000 error="unable to open boltdb: timeout"
Error: unable to open boltdb: timeout
See 'influxd -h' for help
influxdb config file already exists. skipping.
starting influxdb
ts=2024-08-11T13:13:01.580843Z lvl=info msg="Welcome to InfluxDB" log_id=0qxOz5Z0000 version=v2.2.0 commit=a2f8538837 build_date=2022-04-06T17:36:40Z

I have had scrutiny running without issue for 6 months or so since February. No changes to my side in terms of the docker file, but this only started happening within the past 12 days.

[delize]

influxdata/influxdb#24320

When checking into:

root@adcfb9a3fd72:/opt/scrutiny/influxdb# ls -al
total 68
drwx------ 3 1000 root      6 Jul 18 18:40 .
drwxr-xr-x 1 root root      3 Aug 11 13:12 ..
-rw-r--r-- 1 1000 root    142 Jul 18 18:40 config.yaml
drwx------ 5 1000 root      5 Jul 18 17:42 engine
-rw------- 1 1000 1000 131072 Aug 11 00:15 influxd.bolt
-rw-r--r-- 1 1000 1000 122880 Jul 18 17:32 influxd.sqlite

Copied it as a backup, then removed the file.

Checked logs again:

luxdb/influxd.sqlite
ts=2024-08-11T13:20:46.067539Z lvl=error msg="Failed to apply SQL migrations" log_id=0qxPQRuG000 error="DB contains record of unknown migration \"0006_migrate_replications_foreign_key\" - if you are downgrading from a more recent version of influxdb, please run the \"influxd downgrade\" command from that version to revert your metadata to be compatible with this version prior to starting influxd."
Error: DB contains record of unknown migration "0006_migrate_replications_foreign_key" - if you are downgrading from a more recent version of influxdb, please run the "influxd downgrade" command from that version to revert your metadata to be compatible with this version prior to starting influxd.
See 'influxd -h' for help
influxdb config file already exists. skipping.
starting influxdb
ts=2024-08-11T13:20:47.161901Z lvl=info msg="Welcome to InfluxDB" log_id=0qxPQWEG000 version=v2.2.0 commit=a2f8538837 build_date=2022-04-06T17:36:40Z
ts=2024-08-11T13:20:47.169666Z lvl=info msg="Resources opened" log_id=0qxPQWEG000 service=bolt path=/opt/scrutiny/influxdb/influxd.bolt
ts=2024-08-11T13:20:47.169721Z lvl=info msg="Resources opened" log_id=0qxPQWEG000 service=sqlite path=/opt/scrutiny/influxdb/influxd.sqlite
ts=2024-08-11T13:20:47.175764Z lvl=error msg="Failed to apply SQL migrations" log_id=0qxPQWEG000 error="DB contains record of unknown migration \"0006_migrate_replications_foreign_key\" - if you are downgrading from a more recent version of influxdb, please run the \"influxd downgrade\" command from that version to revert your metadata to be compatible with this version prior to starting influxd."
Error: DB contains record of unknown migration "0006_migrate_replications_foreign_key" - if you are downgrading from a more recent version of influxdb, please run the "influxd downgrade" command from that version to revert your metadata to be compatible with this version prior to starting influxd.
See 'influxd -h' for help
^Ccontext canceled

If I remove the whole mounted folder, and try to have Scrutiny rebuild everything from scratch in the volume mount - also fails with the above errors.

For reference, I have my own instance of InfluxDB running, in a stand alone container. But use the omni since it's an AIO package.

I attempted to downgrade the image to v0.80 omnibus, removed the volume mount, and have rebooted - and am still experiencing the issue.