Traefik, Let's Encrypt DNS Challenge, but Grafana in subfolder: Not Found

[Servery]

I modified the "Advanced installation with Traefik, Let's Encrypt & HTTP Basic Auth" so that the SSL challenge is performed via DNS with Cloudflare API. It work's!

https://teslamate.subdomain.domain.de/ is pointing to TeslaMate, but
https://teslamate.subdomain.domain.de/grafana only shows: "Not Found"
If I now stop the teslamate container with docker-compose stop teslamate Grafana works great in it's subfolder. What is wrong with my config?

docker-compose.yml

version: "3"

services:
teslamate:
image: teslamate/teslamate:latest
restart: always
depends_on:
- database
environment:
- DATABASE_USER=${TM_DB_USER}
- DATABASE_PASS=${TM_DB_PASS}
- DATABASE_NAME=${TM_DB_NAME}
- DATABASE_HOST=database
- MQTT_HOST=mosquitto
- VIRTUAL_HOST=${FQDN_TM}
- CHECK_ORIGIN=true
- TZ=${TM_TZ}
volumes:
- ./import:/opt/app/import
labels:
- "traefik.enable=true"
- "traefik.port=4000"
- "traefik.http.middlewares.redirect.redirectscheme.scheme=https"
- "traefik.http.middlewares.teslamate-auth.basicauth.realm=teslamate"
- "traefik.http.middlewares.teslamate-auth.basicauth.usersfile=/auth/.htpasswd"
- "traefik.http.routers.teslamate-insecure.rule=Host(${FQDN_TM})"
- "traefik.http.routers.teslamate-insecure.middlewares=redirect"
- "traefik.http.routers.teslamate-ws.rule=Host(${FQDN_TM}) && Path(/live/websocket)"
- "traefik.http.routers.teslamate-ws.entrypoints=websecure"
- "traefik.http.routers.teslamate-ws.tls"
- "traefik.http.routers.teslamate.rule=Host(${FQDN_TM})"
- "traefik.http.routers.teslamate.middlewares=teslamate-auth"
- "traefik.http.routers.teslamate.entrypoints=websecure"
- "traefik.http.routers.teslamate.tls.certresolver=myresolver"
cap_drop:
- all

database:
image: postgres:13
restart: always
environment:
- POSTGRES_USER=${TM_DB_USER}
- POSTGRES_PASSWORD=${TM_DB_PASS}
- POSTGRES_DB=${TM_DB_NAME}
volumes:
- teslamate-db:/var/lib/postgresql/data

grafana:
image: teslamate/grafana:latest
restart: always
environment:
- DATABASE_USER=${TM_DB_USER}
- DATABASE_PASS=${TM_DB_PASS}
- DATABASE_NAME=${TM_DB_NAME}
- DATABASE_HOST=database
- GRAFANA_PASSWD=${GRAFANA_PW}
- GF_SECURITY_ADMIN_USER=${GRAFANA_USER}
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PW}
- GF_AUTH_ANONYMOUS_ENABLED=false
- GF_SERVER_DOMAIN=${FQDN_TM}
- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana
- GF_SERVER_SERVE_FROM_SUB_PATH=true

volumes:
  - teslamate-grafana-data:/var/lib/grafana
labels:
  - "traefik.enable=true"
  - "traefik.port=3000"
  - "traefik.http.middlewares.redirect.redirectscheme.scheme=https"
  - "traefik.http.routers.grafana-insecure.rule=Host(`${FQDN_TM}`)"
  - "traefik.http.routers.grafana-insecure.middlewares=redirect"
  - "traefik.http.routers.grafana.rule=Path(`/grafana`) || PathPrefix(`/grafana/`)"
  - "traefik.http.routers.grafana.entrypoints=websecure"
  - "traefik.http.routers.grafana.tls.certresolver=myresolver"

mosquitto:
image: eclipse-mosquitto:1.6
restart: always
ports:
- 127.0.0.1:1883:1883
volumes:
- mosquitto-conf:/mosquitto/config
- mosquitto-data:/mosquitto/data

proxy:
image: traefik:v2.3
restart: always
command:
- "--global.sendAnonymousUsage=false"
- "--providers.docker"
- "--providers.docker.exposedByDefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.dnschallenge=true"
- "--certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudflare"
- "--certificatesresolvers.myresolver.acme.email=${LETSENCRYPT_EMAIL}"
- "--certificatesresolvers.myresolver.acme.storage=/etc/acme/acme.json"
ports:
- 80:80
- 443:443
environment:
- "CF_API_EMAIL=${CLOUDFLARE_EMAIL}" # Account email
- "CF_DNS_API_TOKEN=${CLOUDFLARE_API_TOKEN}" # API token with DNS: Edit permission (since v3.1.0)
volumes:
- ./.htpasswd:/auth/.htpasswd
- ./acme/:/etc/acme/
- /var/run/docker.sock:/var/run/docker.sock:ro

volumes:
teslamate-db:
teslamate-grafana-data:
mosquitto-conf:
mosquitto-data:

[tobiasehlert]

From what I can see, it looks like this should be pretty correct actually.

What do you see in the log output in your docker container, when trying to access the grafana path?
Run docker-compose logs --tail=5 --follow and make a request to your grafana folder.

Maybe it can help you find the issue, if you add more logging to Traefik to see, what's going on by adding following line under command section:
- "--log.level=DEBUG"

[Servery]

docker-compose up -d

root@DockerTeslaMate:~/TeslaMate# docker-compose logs --tail=10
Attaching to teslamate_teslamate_1, teslamate_database_1, teslamate_grafana_1, teslamate_mosquitto_1, teslamate_proxy_1
database_1 |
database_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
database_1 |
database_1 | 2021-02-28 22:44:54.901 UTC [1] LOG: starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
database_1 | 2021-02-28 22:44:54.902 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
database_1 | 2021-02-28 22:44:54.902 UTC [1] LOG: listening on IPv6 address "::", port 5432
database_1 | 2021-02-28 22:44:54.916 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
database_1 | 2021-02-28 22:44:54.925 UTC [28] LOG: database system was shut down at 2021-02-28 22:44:25 UTC
database_1 | 2021-02-28 22:44:54.940 UTC [1] LOG: database system is ready to accept connections
grafana_1 | t=2021-02-28T22:44:54+0000 lvl=info msg="App mode production" logger=settings
grafana_1 | t=2021-02-28T22:44:54+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
grafana_1 | t=2021-02-28T22:44:54+0000 lvl=info msg="Starting DB migrations" logger=migrator
grafana_1 | t=2021-02-28T22:44:54+0000 lvl=info msg="Starting plugin search" logger=plugins
grafana_1 | t=2021-02-28T22:44:54+0000 lvl=info msg="Registering plugin" logger=plugins id=input
grafana_1 | t=2021-02-28T22:44:55+0000 lvl=info msg="Registering plugin" logger=plugins id=grafana-piechart-panel
grafana_1 | t=2021-02-28T22:44:55+0000 lvl=info msg="Registering plugin" logger=plugins id=grafana-map-panel
grafana_1 | t=2021-02-28T22:44:55+0000 lvl=info msg="Registering plugin" logger=plugins id=natel-discrete-panel
grafana_1 | t=2021-02-28T22:44:55+0000 lvl=info msg="Registering plugin" logger=plugins id=pr0ps-trackmap-panel
grafana_1 | t=2021-02-28T22:44:55+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl=/grafana socket=
mosquitto_1 | 1614552294: mosquitto version 1.6.13 starting
mosquitto_1 | 1614552294: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto_1 | 1614552294: Opening ipv4 listen socket on port 1883.
mosquitto_1 | 1614552294: Opening ipv6 listen socket on port 1883.
mosquitto_1 | 1614552294: mosquitto version 1.6.13 running
mosquitto_1 | 1614552301: New connection from 172.23.0.6 on port 1883.
mosquitto_1 | 1614552301: New client connected from 172.23.0.6 as TESLAMATE_302E3338373 (p2, c1, k60).
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=teslamate-ws@docker serviceName=teslamate-teslamate
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Creating server 0 http://172.23.0.6:4000" entryPointName=websecure routerName=teslamate-ws@docker serviceName=teslamate-teslamate serverName=0
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Added outgoing tracing middleware teslamate-teslamate" entryPointName=websecure routerName=teslamate-ws@docker middlewareName=tracing middlewareType=TracingForwarder
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
proxy_1 | time="2021-02-28T22:44:58Z" level=warning msg="No domain found in rule Path(/grafana) || PathPrefix(/grafana/), the TLS options applied for this router will depend on the hostSNI of each request" entryPointName=websecure routerName=grafana@docker
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Adding route for dockerteslamate.local.mydomain.com with TLS options default" entryPointName=websecure
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Try to challenge certificate for domain [dockerteslamate.local.mydomain.com] found in HostSNI rule" providerName=tmhttpchallenge.acme routerName=teslamate@docker rule="Host(DockerTeslaMate.local.mydomain.com)"
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Looking for provided certificate(s) to validate ["dockerteslamate.local.mydomain.com"]..." providerName=tmhttpchallenge.acme routerName=teslamate@docker rule="Host(DockerTeslaMate.local.mydomain.com)"
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="No ACME certificate generation required for domains ["dockerteslamate.local.mydomain.com"]." rule="Host(DockerTeslaMate.local.mydomain.com)" providerName=tmhttpchallenge.acme routerName=teslamate@docker
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="No domain parsed in provider ACME" providerName=tmhttpchallenge.acme routerName=grafana@docker rule="Path(/grafana) || PathPrefix(/grafana/)"
teslamate_1 | 2021-02-28 23:44:57.852 [info] Migrations already up
teslamate_1 | 2021-02-28 23:45:01.618 [info] Version: 1.21.5
teslamate_1 | 2021-02-28 23:45:01.705 [info] Running TeslaMateWeb.Endpoint with cowboy 2.8.0 at :::4000 (http)
teslamate_1 | 2021-02-28 23:45:01.707 [info] Access TeslaMateWeb.Endpoint at http://DockerTeslaMate.local.mydomain.com
teslamate_1 | 2021-02-28 23:45:01.719 [info] MQTT connection has been established
teslamate_1 | 2021-02-28 23:45:05.135 [info] tzdata release in place is from a file last modified Tue, 22 Dec 2020 23:35:21 GMT. Release file on server was last modified Sun, 24 Jan 2021 19:35:23 GMT.
teslamate_1 | 2021-02-28 23:45:08.420 [info] Tzdata has updated the release from 2020e to 2021a

browsing to https://dockerteslamate.local.mydomain.com/grafana/

teslamate_1 | 2021-02-28 23:48:34.607 [info] GET /grafana/
teslamate_1 | 2021-02-28 23:48:34.607 [info] Sent 404 in 243µs
proxy_1 | time="2021-02-28T22:44:58Z" level=warning msg="No domain found in rule Path(/grafana) || PathPrefix(/grafana/), the TLS options applied for this router will depend on the hostSNI of each request" entryPointName=websecure routerName=grafana@docker
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Adding route for dockerteslamate.local.mydomain.com with TLS options default" entryPointName=websecure
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Try to challenge certificate for domain [dockerteslamate.local.mydomain.com] found in HostSNI rule" providerName=tmhttpchallenge.acme routerName=teslamate@docker rule="Host(DockerTeslaMate.local.mydomain.com)"
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="Looking for provided certificate(s) to validate ["dockerteslamate.local.mydomain.com"]..." providerName=tmhttpchallenge.acme routerName=teslamate@docker rule="Host(DockerTeslaMate.local.mydomain.com)"
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="No ACME certificate generation required for domains ["dockerteslamate.local.mydomain.com"]." rule="Host(DockerTeslaMate.local.mydomain.com)" providerName=tmhttpchallenge.acme routerName=teslamate@docker
proxy_1 | time="2021-02-28T22:44:58Z" level=debug msg="No domain parsed in provider ACME" providerName=tmhttpchallenge.acme routerName=grafana@docker rule="Path(/grafana) || PathPrefix(/grafana/)"
proxy_1 | time="2021-02-28T22:48:34Z" level=debug msg="Authentication succeeded" middlewareName=teslamate-auth@docker middlewareType=BasicAuth

[tobiasehlert]

Hi @Servery

Looks weird.. it should work with your configuration. I'm not sure about the Cloudflare config, if there could be something there.
Have you doublechecked if that is as it should be?
Have you tried to run the default letsencrypt way as well, to see if that maybe behaves more correct?

[Servery]

Let's Encrypt works perfectly: my Subdomain is getting a valid SSL certificate

[DrMichael]

I am a bit confused by dockerteslamate.local.mydomain.com. Didn't you say teslamate.subdomain.domain.de? Are those indeed the right FQDN's?

[Servery]

I replaced my real FQDN with some dummy data.

[Servery]

In my logs there is a warning:

No domain found in rule Path(/grafana) || PathPrefix(/grafana/), the TLS options applied for this router will depend on the hostSNI of each request

maybe this config for Grafana is not correct:

"traefik.http.routers.grafana.rule=Path(/grafana) || PathPrefix(/grafana/)"

the teslamate config is a little different:

"traefik.http.routers.teslamate.rule=Host(${FQDN_TM})"

Is there everything okay in the Grafana config?