All apps missing from "list"

[tylercal]

Description of problem

I'm not sure if this is somehow related to the fix for #215, but all of my let's encrypt apps appear to be missing from dokku letsencrypt:list

How reproducible

not sure

Steps to Reproduce

I was going through the steps to test the fix for #215 (mostly verifying dokku letsencrypt:cron-job --add would execute). After that was successful, I ran dokku letsencrypt:list and saw one should auto-renew via cron in a few days.

Actual Results

Today when I came to check, none of my apps were listed in letsencrypt and the certificate for the app that would have been scheduled for renew had not been updated.

Expected Results

All of my previously enabled apps would still be in the list.

Environment Information

~> dokku report
-----> uname: Linux DokkuVM 4.15.0-132-generic #136-Ubuntu SMP Tue Jan 12 14:58:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
                     total        used        free      shared  buff/cache   available
       Mem:           7977        2924         241          68        4810        4736
       Swap:             0           0           0
-----> docker version:
       Client: Docker Engine - Community
        Version:           19.03.13
        API version:       1.40
        Go version:        go1.13.15
        Git commit:        4484c46d9d
        Built:             Wed Sep 16 17:02:59 2020
        OS/Arch:           linux/amd64
        Experimental:      false

       Server: Docker Engine - Community
        Engine:
         Version:          19.03.13
         API version:      1.40 (minimum version 1.12)
         Go version:       go1.13.15
         Git commit:       4484c46d9d
         Built:            Wed Sep 16 17:01:30 2020
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.3.7
         GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
        runc:
         Version:          1.0.0-rc10
         GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
        docker-init:
         Version:          0.18.0
         GitCommit:        fec3683
-----> docker daemon info:
       Client:
        Debug Mode: true

       Server:
        Containers: 31
         Running: 23
         Paused: 0
         Stopped: 8
        Images: 87
        Server Version: 19.03.13
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
        runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
        init version: fec3683
        Security Options:
WARNING: No swap limit support
         apparmor
         seccomp
          Profile: default
        Kernel Version: 4.15.0-132-generic
        Operating System: Ubuntu 18.04.5 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 2
        Total Memory: 7.79GiB
        Name: DokkuVM
        ID: CA3Z:GOHT:SO22:RWHD:BQCZ:QF4Y:ZIUH:POZD:HOZE:D7LY:HD6Y:YQCR
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> git version: git version 2.17.1
-----> sigil version: 0.6.0
-----> herokuish version:
       herokuish: 0.5.27
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v225
         heroku-buildpack-nodejs    v183
         heroku-buildpack-clojure   v87
         heroku-buildpack-python    v191
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v35
         heroku-buildpack-scala     v90
         heroku-buildpack-play      v26
         heroku-buildpack-php       v190
         heroku-buildpack-go        v153
         buildpack-nginx            v14
         buildpack-null             v3
-----> dokku version: dokku version 0.24.3
-----> plugn version: plugn: 0.6.1
-----> dokku plugins:
         00_dokku-standard    0.24.3 enabled    dokku core standard plugin
         20_events            0.24.3 enabled    dokku core events logging plugin
         app-json             0.24.3 enabled    dokku core app-json plugin
         apps                 0.24.3 enabled    dokku core apps plugin
         builder              0.24.3 enabled    dokku core builder plugin
         builder-dockerfile   0.24.3 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.24.3 enabled    dokku core builder-herokuish plugin
         builder-pack         0.24.3 enabled    dokku core builder-pack plugin
         buildpacks           0.24.3 enabled    dokku core buildpacks plugin
         certs                0.24.3 enabled    dokku core certificate management plugin
         checks               0.24.3 enabled    dokku core checks plugin
         common               0.24.3 enabled    dokku core common plugin
         config               0.24.3 enabled    dokku core config plugin
         cron                 0.24.3 enabled    dokku core cron plugin
         docker-options       0.24.3 enabled    dokku core docker-options plugin
         domains              0.24.3 enabled    dokku core domains plugin
         enter                0.24.3 enabled    dokku core enter plugin
         git                  0.24.3 enabled    dokku core git plugin
         hostname             0.2.0 enabled    Sets the docker hostname option for dokku
         letsencrypt          0.11.2 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.24.3 enabled    dokku core logs plugin
         logspout             0.4.0 enabled    sends dokku app stdout to a logging service
         network              0.24.3 enabled    dokku core network plugin
         nginx-vhosts         0.24.3 enabled    dokku core nginx-vhosts plugin
         plugin               0.24.3 enabled    dokku core plugin plugin
         postgres             1.0.0 enabled    dokku postgres service plugin
         proxy                0.24.3 enabled    dokku core proxy plugin
         ps                   0.24.3 enabled    dokku core ps plugin
         redis                1.12.5 enabled    dokku redis service plugin
         repo                 0.24.3 enabled    dokku core repo plugin
         resource             0.24.3 enabled    dokku core resource plugin
         scheduler-docker-local 0.24.3 enabled    dokku core scheduler-docker-local plugin
         shell                0.24.3 enabled    dokku core shell plugin
         ssh-keys             0.24.3 enabled    dokku core ssh-keys plugin
         storage              0.24.3 enabled    dokku core storage plugin
         tags                 0.24.3 enabled    dokku core tags plugin
         tar                  0.24.3 enabled    dokku core tar plugin
         trace                0.24.3 enabled    dokku core trace plugin

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

installed via apt on Azure VM

Additional information

Everything has been working fine for some time. I do try to keep my version of dokku up to date, so it could have been an update with that or some 3rd cause unrelated to the 11.0/1/2 release of this plugin.

[m-z-b]

Just to note that this was one of several issues I had trying to do a simple upgrade from 0.9.4.
(Dokku 0.21.4 on Ubuntu 20.04). So you're not alone.

I assumed it was some left over state from a previous version, but couldn't figure out what it was. Given it was a 3 day old release of letsencrypt and running on a live server, I ended up reverting to 0.9.4.

[josegonzalez]

Does re-adding the cron make it "work" again?

Seems like the switch to letsgo broke this because the filepaths changed, but I think this is fixable by adding support for the old and new style.

[tylercal]

re-adding the cron via letsencrypt:cron-job --add did not restore the apps to the "list". I ended up re-enabling all the apps via letsencrypt:enable

[bradbajuz]

I just wanted to chime in that I was having the same issue and had to re-enable both apps using the letsencrypt command letsencrypt:enable. Updating the plugin and adding the cronjob didn't make a difference. My apps had certificates past the 30 day renewal period; both up for renewal on 4/20/21.

[josegonzalez]

A potential fix was released as part of 0.11.4. I'll leave this ticket open for a week in case someone can confirm/deny the latest version fixes the problem for them.

[arusa]

I experienced the same problem. My monitoring once again reported, that the certificates are going to expire, but now I can't even renew manually, because dokku letsencrypt:list didn't show any certificates/apps at all anymore.

After installing 0.11.4 I could see them all again. Thank you! :)

[arusa]

JFI: Because I think that it also has to do with the same changes:
While running a manual renew I see a lot of these messages:

Reloading nginx
rm: cannot remove '/home/dokku/APPNAME/tls/server.letsencrypt.crt': No such file or directory
-----> Configuring APP_HOSTNAME...(using built-in template)

[josegonzalez]

Hopefully I've fixed that in the latest release @arusa. Try 0.11.5 now.

[ifree92]

I've faced with the same issue.
I have been upgraded to 0.11.2 and lost all apps on the list.

I have anew enabled my apps and just a moment before upgraded to 0.11.6 and again - lost all my enabled apps.
dokku letsencrypt:list returns nothing.

UPD 0.11.6: when I'm trying to enable dokku letsencrypt:enable my-super-app, the process is good, but then dokku letsencrypt:list returns nothing for me. Very strange bug.
Now I'm updating the certs blindly.

[benwinding]

Still an issue in letsencrypt 0.11.6 cannot see any apps in letsencrypt:list, even after running letsencrypt:enable myapp for each one.

I was able to downgrade by using the following command:

dokku plugin:update letsencrypt 0.10.1

[arusa]

It worked in 0.11.5 again, but after upgrading to 0.11.6 the letsencrypt:list is empty again

[arusa]

Thanks @benwinding
It also works with:

dokku plugin:update letsencrypt 0.11.5

[josegonzalez]

Ah feck I realize what I did wrong here, let me fix it and will release it in 0.11.7.

[josegonzalez]

This is what I get for not having comprehensive integration tests for this plugin... I really should roll this into the core and have it tested there...

[josegonzalez]

Alright we're in business, @arusa mind checking if 0.11.7 does the trick? I think it should fix it for both old and new setups.

[arusa]

Pinning the update to 0.11.5 maybe wasn't the best idea. I can't find out how to unpin it.

I updated (and pinned) it to 0.11.7 now and the letsencrypt:list works again, but renewing a cert and enabling letsencrypt currently doesn't work :(

       2021/04/06 08:01:26 Could not obtain certificates:
       	error: one or more domains had a problem:
.... acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: The key authorization file from the server did not match this challenge "3Ib04L5qx0MeYAKFFyfGlJe9BPbbkRarK1B5jjeuqq4.jL0TD0c7WAQGp9amPlI0PVJSn4EoLRiSvaSP18kiWa8" != "TEST"

[josegonzalez]

How are you triggering a renewal? Seems like one side is responding with TEST. If you could show the entire output, that might help debug.

[josegonzalez]

I'll be offline for a few hours, but if you can hop on to our slack channel during the afternoon/evening EST, that would be great.

[benwinding]

I'll be offline for a few hours, but if you can hop on to our slack channel during the afternoon/evening EST, that would be great.

All good, thanks for the updates @josegonzalez and I appreciate your work and quick reaction time!

Cheers,
Ben

[arusa]

First I tried it with letsencrypt:auto-renew and then with letsencrypt:enable, but both brought the same result.

Where can I find the slack channel? I am in CEST (UTC+2), but maybe we can find the right time ;-)

$ dokku letsencrypt:enable myapp
=====> Enabling letsencrypt for myapp
-----> Enabling ACME proxy for myapp...
       Reloading nginx configuration (via systemctl): nginx.service.
-----> Getting letsencrypt certificate for myapp...
        - Domain 'myapp.at'
        - Domain 'www.myapp.at'
       2021/04/06 08:00:55 No key found for account [email protected]. Generating a P256 key.
       2021/04/06 08:00:55 Saved key to /certs/accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected]
       2021/04/06 08:00:56 [INFO] acme: Registering account for [email protected]
       !!!! HEADS UP !!!!

       Your account credentials have been saved in your Let's Encrypt
       configuration directory at "/certs/accounts".

       You should make a secure backup of this folder now. This
       configuration directory will also contain certificates and
       private keys obtained from Let's Encrypt so making regular
       backups of this folder is ideal.
       2021/04/06 08:00:56 [INFO] [myapp.dokku.mydomain.at, myapp.at, www.myapp.at] acme: Obtaining bundled SAN certificate
       2021/04/06 08:00:57 [INFO] [myapp.at] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645770
       2021/04/06 08:00:57 [INFO] [myapp.dokku.mydomain.at] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645771
       2021/04/06 08:00:57 [INFO] [www.myapp.at] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645773
       2021/04/06 08:00:57 [INFO] [myapp.at] acme: Could not find solver for: tls-alpn-01
       2021/04/06 08:00:57 [INFO] [myapp.at] acme: use http-01 solver
       2021/04/06 08:00:57 [INFO] [myapp.dokku.mydomain.at] acme: Could not find solver for: tls-alpn-01
       2021/04/06 08:00:57 [INFO] [myapp.dokku.mydomain.at] acme: use http-01 solver
       2021/04/06 08:00:57 [INFO] [www.myapp.at] acme: Could not find solver for: tls-alpn-01
       2021/04/06 08:00:57 [INFO] [www.myapp.at] acme: use http-01 solver
       2021/04/06 08:00:57 [INFO] [myapp.at] acme: Trying to solve HTTP-01
       2021/04/06 08:00:58 [INFO] [myapp.at] Served key authentication
       2021/04/06 08:00:59 [INFO] [myapp.at] Served key authentication
       2021/04/06 08:00:59 [INFO] [myapp.at] Served key authentication
       2021/04/06 08:01:00 [INFO] [myapp.at] Served key authentication
       2021/04/06 08:01:04 [INFO] [myapp.at] The server validated our request
       2021/04/06 08:01:04 [INFO] [myapp.dokku.mydomain.at] acme: Trying to solve HTTP-01
       2021/04/06 08:01:04 [INFO] [myapp.dokku.mydomain.at] Served key authentication
       2021/04/06 08:01:05 [INFO] [myapp.dokku.mydomain.at] Served key authentication
       2021/04/06 08:01:06 [INFO] [myapp.dokku.mydomain.at] Served key authentication
       2021/04/06 08:01:15 [INFO] [myapp.dokku.mydomain.at] Served key authentication
       2021/04/06 08:01:20 [INFO] [myapp.dokku.mydomain.at] The server validated our request
       2021/04/06 08:01:20 [INFO] [www.myapp.at] acme: Trying to solve HTTP-01
       2021/04/06 08:01:21 [WARN] Received request for domain myapp.at:443 with method GET but the domain did not match any challenge. Please ensure your are passing the Host header properly.
       2021/04/06 08:01:22 [WARN] Received request for domain myapp.at:443 with method GET but the domain did not match any challenge. Please ensure your are passing the Host header properly.
       2021/04/06 08:01:26 [INFO] Skipping deactivating of valid auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645770
       2021/04/06 08:01:26 [INFO] Skipping deactivating of valid auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645771
       2021/04/06 08:01:26 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645773
       2021/04/06 08:01:26 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/12124645773
       2021/04/06 08:01:26 Could not obtain certificates:
       	error: one or more domains had a problem:
       [www.myapp.at] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: The key authorization file from the server did not match this challenge "3Ib04L5qx0MeYAKFFyfGlJe9BPbbkRarK1B5jjeuqq4.jL0TD0c7WAQGp9amPlI0PVJSn4EoLRiSvaSP18kiWa8" != "TEST"
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for myapp...
       Reloading nginx configuration (via systemctl): nginx.service.
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

[josegonzalez]

Link here: https://glider-slackin.herokuapp.com

[drguildo]

I just installed 0.11.7 (for the first time, not an upgrade), successfully enabled it on an app (HTTPS works), but the app isn't listed under dokku letsencrypt:list:

$ sudo dokku letsencrypt:list
-----> App name           Certificate Expiry        Time before expiry        Time before renewal      
$

[michaelyork]

cross-posting from #229

+1 to the issue @drguildo is facing on 0.11.7. For what it's worth, it also seems to break HTTPS after rebooting the host.

[josegonzalez]

Okay, the is_active check should work in 0.11.8. Can someone verify this?

[josegonzalez]

@michaelyork mind verifying 0.11.8 fixes the reboot issue as well?

[michaelyork]

Looking good! Thanks so much for the quick fix @josegonzalez. list working properly and certs survived reboot.

[drguildo]

0.11.8 seems to fix the app listing for me, too. Thanks.

[arusa]

Unfortunately I still can't renew or create letsencrypt certificates :-(

I still get this error:

acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: The key authorization file from the server did not match this challenge "15woyWlob1bZsXCslOusTeuZu1PE-ydO3u2J_RDFFZs.BOxuB9GIFepTlJg0Y2RR4noVhIIRCWkCn6UkiyKW6AI" != "TEST"

As you can see above #224 (comment)

[josegonzalez]

@arusa we diagnosed your issue on slack and it seems to be #228.

Closing this issue though, since the list problem is fixed :)

[rome2o]

I'm on 0.11.8v, it isn't working for me!

I can see the :list but when I do auto-renew it says Letsencrypt not enabled

Can you please advise?

[josegonzalez]

@rome2o can you enable it for the app first?

[rome2o]

@josegonzalez I've it already enabled, so the reason I got this issue is that certificate expired, the app stopped receiving webhook and client complained. So I went to renew certificate manually through enabling it again.

I've setup a staging server to reproduce the issue and yes the issue remain intact. No matter how many times we enable it, it somehow says it's not enabled when we run auto-renew command.

Here's the report for the app

                     total        used        free      shared  buff/cache   available
       Mem:           7961        2409         345          72        5206        5188
       Swap:           511          22         489
-----> docker version: 
       Client: Docker Engine - Community
        Version:           20.10.5
        API version:       1.41
        Go version:        go1.13.15
        Git commit:        55c4c88
        Built:             Tue Mar  2 20:18:20 2021
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true
       
       Server: Docker Engine - Community
        Engine:
         Version:          20.10.5
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.13.15
         Git commit:       363e9a8
         Built:            Tue Mar  2 20:16:15 2021
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.4.4
         GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
        runc:
         Version:          1.0.0-rc93
         GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
       
       Server:
        Containers: 12
         Running: 6
         Paused: 0
         Stopped: 6
        Images: 22
        Server Version: 20.10.5
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Cgroup Version: 1
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: x
        runc version: x
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
        Kernel Version: 5.4.0-70-generic
        Operating System: Ubuntu 20.04.2 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 4
        Total Memory: 7.775GiB
        Name: localhost
        ID: SDFF:D7J4:FGZW:5C7J:2JII:WFFA:YEBE:36OQ:QZRL:USWC:D3LG:EVOZ
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
WARNING: No swap limit support
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> git version: git version 2.25.1
-----> sigil version: 0.6.0
-----> herokuish version: 
       herokuish: 0.5.27
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v225
         heroku-buildpack-nodejs    v183
         heroku-buildpack-clojure   v87
         heroku-buildpack-python    v191
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v35
         heroku-buildpack-scala     v90
         heroku-buildpack-play      v26
         heroku-buildpack-php       v190
         heroku-buildpack-go        v153
         buildpack-nginx            v14
         buildpack-null             v3
-----> dokku version: dokku version 0.24.3
-----> plugn version: plugn: 0.6.1
-----> dokku plugins: 
         00_dokku-standard    0.24.3 enabled    dokku core standard plugin
         20_events            0.24.3 enabled    dokku core events logging plugin
         app-json             0.24.3 enabled    dokku core app-json plugin
         apps                 0.24.3 enabled    dokku core apps plugin
         builder              0.24.3 enabled    dokku core builder plugin
         builder-dockerfile   0.24.3 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.24.3 enabled    dokku core builder-herokuish plugin
         builder-pack         0.24.3 enabled    dokku core builder-pack plugin
         buildpacks           0.24.3 enabled    dokku core buildpacks plugin
         certs                0.24.3 enabled    dokku core certificate management plugin
         checks               0.24.3 enabled    dokku core checks plugin
         common               0.24.3 enabled    dokku core common plugin
         config               0.24.3 enabled    dokku core config plugin
         cron                 0.24.3 enabled    dokku core cron plugin
         docker-options       0.24.3 enabled    dokku core docker-options plugin
         domains              0.24.3 enabled    dokku core domains plugin
         enter                0.24.3 enabled    dokku core enter plugin
         git                  0.24.3 enabled    dokku core git plugin
         letsencrypt          0.11.8 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.24.3 enabled    dokku core logs plugin
         mysql                1.12.1 enabled    dokku mysql service plugin
         network              0.24.3 enabled    dokku core network plugin
         nginx-vhosts         0.24.3 enabled    dokku core nginx-vhosts plugin
         plugin               0.24.3 enabled    dokku core plugin plugin
         proxy                0.24.3 enabled    dokku core proxy plugin
         ps                   0.24.3 enabled    dokku core ps plugin
         redis                1.13.0 enabled    dokku redis service plugin
         repo                 0.24.3 enabled    dokku core repo plugin
         resource             0.24.3 enabled    dokku core resource plugin
         scheduler-docker-local 0.24.3 enabled    dokku core scheduler-docker-local plugin
         shell                0.24.3 enabled    dokku core shell plugin
         slack                0.4.0 enabled    Slack notifier on deploy
         ssh-keys             0.24.3 enabled    dokku core ssh-keys plugin
         storage              0.24.3 enabled    dokku core storage plugin
         tags                 0.24.3 enabled    dokku core tags plugin
         tar                  0.24.3 enabled    dokku core tar plugin
         trace                0.24.3 enabled    dokku core trace plugin
=====> playground app information
       App deploy source:             
       App dir:                       /home/dokku/playground
       App locked:                    false
=====> playground builder information
       Builder computed selected:     
       Builder global selected:       
       Builder selected:              
=====> playground buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
CN = app.x.com.au
error 20 at 0 depth lookup: unable to get local issuer certificate
=====> playground ssl information
       Ssl dir:                       /home/dokku/playground/tls
       Ssl enabled:                   true                     
       Ssl hostnames:                 app-bypass.x.com.au app.x.com.au
       Ssl expires at:                Oct  6 09:11:06 2021 GMT 
       Ssl issuer:                    C = US, O =              
       Ssl starts at:                 Jul  8 09:11:07 2021 GMT 
       Ssl subject:                   subject=CN = app.x.com.au
       Ssl verified:                  self signed              
=====> playground checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
=====> playground cron information
       Cron task count:               2
=====> playground docker options information
       Docker options build:          --link dokku.mysql.playground-mysql:dokku-mysql-playground-mysql --link dokku.redis.playground-redis:dokku-redis-playground-redis 
       Docker options deploy:         --link dokku.mysql.playground-mysql:dokku-mysql-playground-mysql --link dokku.redis.playground-redis:dokku-redis-playground-redis --restart=on-failure:10 
       Docker options run:            --link dokku.mysql.playground-mysql:dokku-mysql-playground-mysql --link dokku.redis.playground-redis:dokku-redis-playground-redis 
=====> playground domains information
       Domains app enabled:           true                     
       Domains app vhosts:            app.x.com.au app-bypass.x.com.au
       Domains global enabled:        true                     
       Domains global vhosts:         app.x.com.au     
=====> playground git information
       Git deploy branch:             master                   
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                       e80c8e8b                 
       Git last updated at:           1622811163               
=====> playground logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector sink:              
=====> playground network information
       Network attach post create:    
       Network attach post deploy:    
       Network bind all interfaces:   false
       Network web listeners:         172.17.0.8:5000
=====> playground nginx information
       Nginx access log format:                                
       Nginx access log path:         /var/log/nginx/playground-access.log
       Nginx bind address ipv4:                                
       Nginx bind address ipv6:       ::                       
       Nginx client max body size:                             
       Nginx disable custom config:   false                    
       Nginx error log path:          /var/log/nginx/playground-error.log
       Nginx global hsts:             true                     
       Nginx computed hsts:           true                     
       Nginx hsts:                                             
       Nginx hsts include subdomains: true                     
       Nginx hsts max age:            15724800                 
       Nginx hsts preload:            false                    
       Nginx proxy buffer size:       4096                     
       Nginx proxy buffering:         on                       
       Nginx proxy buffers:           8 4096                   
       Nginx proxy busy buffers size: 8192                     
       Nginx proxy read timeout:      60s                      
       Nginx last visited at:         1625740686               
       Nginx x forwarded for value:   $remote_addr             
       Nginx x forwarded port value:  $server_port             
       Nginx x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
=====> playground proxy information
       Proxy enabled:                 true
       Proxy port map:                http:80:5000 https:443:5000
       Proxy type:                    nginx
=====> playground ps information
       Deployed:                      true
       Processes:                     2
       Ps can scale:                  false
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: 3e1c54d8a08)
       Status worker 1:               running (CID: 3597ed06126)
=====> playground resource information
=====> playground scheduler-docker-local information
       Scheduler docker local disable chown:                          
=====> playground storage information
       Storage build mounts:                                   
       Storage deploy mounts:                                  
       Storage run mounts:   ```

Do we have any hot fix for that?

[josegonzalez]

Mind upgrading dokku to the latest? That's still a few patch versions behind.

Additionally, seems like I may have fixed something here for in dokku-letsencrypt 0.11.9. Mind upgrading the plugin to latest?

[rome2o]

Mind upgrading dokku to the latest? That's still a few patch versions behind.

Additionally, seems like I may have fixed something here for in dokku-letsencrypt 0.11.9. Mind upgrading the plugin to latest?

It looks like it has fixed this, is there any way to know if a certificate renewal has failed? Just in case.

For anyone digging deeper into this, I've updated the dokku to 0.24.10 and updated the plugin using the following command:

dokku plugin:update letsencrypt 0.11.9

The issue seems to be resolved! We just have to see it for it to renew as required. I wish there was a way to notify that the certificate is expired! Maybe should look into an alternative link watcher thing that can do the job?

Thanks @josegonzalez
Ali

[bfontaine]

I had dokku 0.26.8 and dokku-letsencrypt 0.12.1 and still had the issue. I was able to fix it by doing letsencrypt:disable followed by letsencrypt:enable again. Doing letsencrypt:enable without disabling it first didn’t have any effect on the issue.