letsencrypt renew cronjob gets lost every now and then

[arusa]

Description of problem

I have 2 small dokku servers running and both have lost their letsencrypt renewal cronjobs multiple times.

I have no idea why this is happening, but I experience it on two different servers running dokku version 0.23.7 on Ubuntu 18.04 and 20.04

How reproducible

Steps to Reproduce

1. Add cronjob

$ dokku letsencrypt:cron-job --add
no crontab for dokku
-----> Added cron job to dokku's crontab.

$ crontab -l
@daily /var/lib/dokku/plugins/available/letsencrypt/cron-job

2. wait

3. realize that certificates did not renew and cronjob is gone

Every few months I realize that certificates will expire soon and when I go check it on the server the cronjob is suddenly gone:

$ dokku letsencrypt:cron-job
no crontab for dokku
       No job added. Use --add to add the cron job.

$ crontab -l
no crontab for dokku

Actual Results

Cronjobs get lost after a while

Expected Results

Cronjobs should remain

Environment Information

2 Servers:

Ubuntu 18.04: dokku version 0.23.7
Ubuntu 20.04: dokku version 0.23.7

dokku report APP_NAME output

The problem does not affect an app but the global letsencrypt cronjob.

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Ubuntu 18.04 server:
deb https://packagecloud.io/dokku/dokku/ubuntu/ bionic main

Ubuntu 20.04 server:
deb https://packagecloud.io/dokku/dokku/ubuntu/ focal main

Additional information

$ dokku plugin:list
...
  letsencrypt          0.9.4 enabled    Automated installation of let's encrypt TLS certificates
...

Just wanted to document this problem to see if others also experience it too.

[josegonzalez]

Fuck I meant to fix this.

In a new cron-entries trigger, we should check the dokku version and check if it's at least a specific version - 0.23.2 - and echo out the $SCHEDULE;$FULL_COMMAND;$ARBITRARY_DATA (semi-colon delimited, as shown).

Also, the thing that currently writes out the schedule should do the same version check, and only write if its lower than 0.23.0 (the hook was implemented in 0.23.2 but scheduled cron tasks were implemented in 0.23.0), and on versions 0.23.2 and up, trigger the cron-write plugin trigger.

Just writing this down in case you have time to fix it, I won't till the weekend.

[stockmind]

Same issue on my machines (dokku version updated to 0.24.0)!

The cron file also gets emptied out completely (clearing out even the others entries inside), and this also happens when you add the cron-job for the first time with letsencrypt:cron-job --add

[arusa]

I can't think of a reason why an existing crontab entry gets removed.
It would be really easy to just add this crontab entry manually, but something seems to be clearing the dokku users crontab repeatedly.

[tylercal]

I can't find the docs now, but if I recall correctly a recent release (0.23 maybe) of Dokku changed the way cron works. I thought what I remembered reading was that if you manually maintained your crontab it would be wiped out (this is what happened for several of my non-letsencrypt cron jobs).

In any case, I'm on 0.24.2 and this will be the second time my letsencrypt cron has been wiped out in as many months.

[josegonzalez]

This was fixed, please upgrade to 0.10.0 and rerun dokku letsencrypt:cron-job --add to get thing working again. New deploys should also not wipe this out anymore :)

[nerg4l]

When running dokku letsencrypt:cron-job --add I receive the following error:

touch: cannot touch '/var/lib/dokku/data/letsencrypt/autorenew': No such file or directory

According to plugin:list I have the latest version:

letsencrypt          0.10.1 enabled    Automated installation of let's encrypt TLS certificates

Edit: dokku report

-----> uname: Linux urahara 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
                     total        used        free      shared  buff/cache   available
       Mem:          32026       20824        1391          23        9811       11246
       Swap:         20109         117       19991
-----> docker version:
       Client: Docker Engine - Community
        Version:           20.10.1
        API version:       1.40
        Go version:        go1.13.15
        Git commit:        831ebea
        Built:             Tue Dec 15 04:34:58 2020
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true

       Server: Docker Engine - Community
        Engine:
         Version:          19.03.14
         API version:      1.40 (minimum version 1.12)
         Go version:       go1.13.15
         Git commit:       5eb3275d40
         Built:            Tue Dec  1 19:18:53 2020
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.4.3
         GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
        runc:
         Version:          1.0.0-rc92
         GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
        docker-init:
         Version:          0.18.0
         GitCommit:        fec3683
-----> docker daemon info:
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Build with BuildKit (Docker Inc., v0.5.0-docker)

       Server:
        Containers: 12
         Running: 12
         Paused: 0
         Stopped: 0
        Images: 46
        Server Version: 19.03.14
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
WARNING: No swap limit support
        Swarm: inactive
        Runtimes: runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
        runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
        init version: fec3683
        Security Options:
         apparmor
         seccomp
          Profile: default
        Kernel Version: 5.4.0-65-generic
        Operating System: Ubuntu 20.04.1 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 40
        Total Memory: 31.28GiB
        Name: urahara
        ID: QJIR:FUS4:WEB7:MYQ2:UU3J:VAUE:7RRV:BJAD:J5D2:TRUM:UGKY:NPMI
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false

-----> git version: git version 2.25.1
-----> sigil version: 0.6.0
-----> herokuish version:
       herokuish: 0.5.25
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v223
         heroku-buildpack-nodejs    v183
         heroku-buildpack-clojure   v86
         heroku-buildpack-python    v188
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v34
         heroku-buildpack-scala     v88
         heroku-buildpack-play      v26
         heroku-buildpack-php       v189
         heroku-buildpack-go        v151
         buildpack-nginx            v14
         buildpack-null             v3
-----> dokku version: dokku version 0.24.3
-----> plugn version: plugn: 0.6.1
-----> dokku plugins:
         00_dokku-standard    0.24.3 enabled    dokku core standard plugin
         20_events            0.24.3 enabled    dokku core events logging plugin
         app-json             0.24.3 enabled    dokku core app-json plugin
         apps                 0.24.3 enabled    dokku core apps plugin
         builder              0.24.3 enabled    dokku core builder plugin
         builder-dockerfile   0.24.3 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.24.3 enabled    dokku core builder-herokuish plugin
         builder-pack         0.24.3 enabled    dokku core builder-pack plugin
         buildpacks           0.24.3 enabled    dokku core buildpacks plugin
         certs                0.24.3 enabled    dokku core certificate management plugin
         checks               0.24.3 enabled    dokku core checks plugin
         clickhouse           0.0.3 enabled    dokku clickhouse service plugin
         common               0.24.3 enabled    dokku core common plugin
         config               0.24.3 enabled    dokku core config plugin
         cron                 0.24.3 enabled    dokku core cron plugin
         docker-options       0.24.3 enabled    dokku core docker-options plugin
         domains              0.24.3 enabled    dokku core domains plugin
         enter                0.24.3 enabled    dokku core enter plugin
         git                  0.24.3 enabled    dokku core git plugin
         global-cert          0.4.5 enabled    manages a global certificate for dokku
         letsencrypt          0.10.1 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.24.3 enabled    dokku core logs plugin
         mysql                1.12.1 enabled    dokku mysql service plugin
         network              0.24.3 enabled    dokku core network plugin
         nginx-vhosts         0.24.3 enabled    dokku core nginx-vhosts plugin
         plugin               0.24.3 enabled    dokku core plugin plugin
         proxy                0.24.3 enabled    dokku core proxy plugin
         ps                   0.24.3 enabled    dokku core ps plugin
         redis                1.13.0 enabled    dokku redis service plugin
         repo                 0.24.3 enabled    dokku core repo plugin
         resource             0.24.3 enabled    dokku core resource plugin
         scheduler-docker-local 0.24.3 enabled    dokku core scheduler-docker-local plugin
         shell                0.24.3 enabled    dokku core shell plugin
         ssh-keys             0.24.3 enabled    dokku core ssh-keys plugin
         storage              0.24.3 enabled    dokku core storage plugin
         tags                 0.24.3 enabled    dokku core tags plugin
         tar                  0.24.3 enabled    dokku core tar plugin
         trace                0.24.3 enabled    dokku core trace plugin

[nerg4l]

Running mkdir -p /var/lib/dokku/data/letsencrypt fixed the problem but I think this should be part of the update process.

[josegonzalez]

Alright, that should be fixed now as of 0.11.0 (which also switches to acme/lego). If you want to try it out and verify that things worked, that'd be great.

[tylercal]

Doesn't seem to fix it for me

letsencrypt 0.11.0 enabled
~> dokku letsencrypt:cron-job --add
touch: cannot touch '/var/lib/dokku/data/letsencrypt/autorenew': No such file or directory

[nerg4l]

Same here.

# dokku plugin:update letsencrypt
Plugin (letsencrypt) updated
-----> Priming bash-completion cache
# dokku letsencrypt:cron-job --add
touch: cannot touch '/var/lib/dokku/data/letsencrypt/autorenew': No such file or directory

  letsencrypt          0.11.0 enabled    Automated installation of let's encrypt TLS certificates

[josegonzalez]

Okay now it should be fixed. We never had a symlink for the update trigger to the install trigger 😅

[josegonzalez]

Update to 0.11.1 to get the fix :)

[nerg4l]

Now it works

[tylercal]

Ditto, works for me now too. Do I remember correctly that, dokku letsencrypt:cron-job used to show the status of the scheduled cron job? E.g. that it had been added?

The help command shows letsencrypt:cron-job [--add --remove] with the square brackets making it seem like the switches are optional but the output when running without a switch, dokku letsencrypt:cron-job, is Specify --add or --remove to modify the cron-job.

The readme seems to more clearly show the switches as required, letsencrypt:cron-job <--add|--remove>

[josegonzalez]

Can you open a bug for current status?

[josegonzalez]

I filed the issue here: #221