Skip to content
This repository has been archived by the owner on Nov 19, 2021. It is now read-only.

Deployment Notes

Jump to bottom
John Barrett edited this page Feb 24, 2014 · 17 revisions

Community Platform Deployment

release.sh

In script/ddgc_release.sh, supplying the first parameter "view" or "prod" will set the target host to staging or live platform :

if [ "$1" = "view" ];
then
	DDGC_RELEASE_HOSTNAME="view.dukgo.com"
else 	if [ "$1" = "prod" ];
	then
		DDGC_RELEASE_HOSTNAME="duck.co"
	fi
fi

The second parameter is a release tarball name, so to deploy v0.940 to staging it would be invoked like so:

community-platform $ script/ddgc_release.sh view DDGC-0.940.tar.gz

The script runs several commands over ssh as the ddgc user. We'll go through each of these now.

	rm -rf ~/deploy &&
	mkdir ~/deploy

~/deploy is the target directory for the installation.

We then copy the release tarball over to this remote directory with scp, and start running commands over ssh again:

	. /home/ddgc/perl5/perlbrew/etc/bashrc &&
	. /home/ddgc/ddgc_config.sh &&

Initialising perlbrew and DDGC config - ddgc_config.sh exports environment variables for the DDGC application, database/email/XMPP etc. credentials. ddgc_config.sh is not and should not be in public revision control, it is created on a per-host basis.

	cd ~/deploy &&
	tar xz --strip-components=1 -f $2 &&

Extract the tarball, omitting the top level directory from path names (DDGC-0.940/ in our example).

	cpanm -n --installdeps . &&
	duckpan DDGC::Static &&

Install dependencies and static site elements.

	touch ~/ddgc_web_maintenance &&
	echo Stopping current system... && 
	sudo /usr/local/sbin/stop_ddgc.sh &&

Create a "maintenance" flag - other processes (e.g. nginx) can check for the existence of this to detect ongoing maintenance.

The stop_ddgc.sh script signals daemontools to stop the ddgc_web_fastcgi.pl process.

	echo Copying new files in place... && 
	. /home/ddgc/perl5/perlbrew/etc/bashrc &&
	. /home/ddgc/ddgc_config.sh &&

Initialising perlbrew again / DDGC config... is this necessary here?

	mv ~/live ~/backup/$CURRENT_DATE_FILENAME &&
	mv ~/deploy ~/live &&

Backup live, move deployed code to live endpoint.

	rm -rf ~/cache &&
	mkdir ~/cache &&

Recreate cache directory.

	cp -ar ~/live/share/docroot/* ~/docroot/ &&
	cp -ar ~/live/share/docroot_duckpan/* ~/ddgc/duckpan/ &&

Relocate static elements - ~/live/share/docroot/ contains DDGC static/error pages, ~/live/share/docroot_duckpan/ contains duckpan.org static/error pages.

	echo Starting new system... && 
	sudo /usr/local/sbin/start_ddgc.sh &&

start_ddgc.sh signals daemontools to start ddgc_web_fastcgi.pl process and removes the ddgc_web_maintenance flag.

	. /home/ddgc/perl5/perlbrew/etc/bashrc &&
	. /home/ddgc/ddgc_config.sh &&

...

	cd ~/live &&
	script/ddgc_add_duckpan_dist.pl ddgc $2

Adds the community-platform dist to duckpan.

dzil release

This is invoked to actually perform the upgrade script on staging. DuckPAN-Overview#wiki-git-tagging explains some of the steps which occur here to create a dist for upload and tag on github, in summary:

  • Version is incremented
  • Tarball created, e.e. DDGC-0.940.tar.gz
  • Test suite performed.
  • ddgc_release view DDGC-0.940.tar.gz
  • Git tag created from new version and pushed to Github

Each tasks depends on the success of the last. To deploy the same code to live:

ddgc_release prod DDGC-0.940.tar.gz

Questions / Recommendations:

When is script/ddgc_db_autoupgrade.pl performed?

While you can pass the the option --doupgrade option to perform the upgrades in the diff manually, the output of this script is currently reviewed and performed by hand during deployment.

Would deploying to a dated directory (e.g. ~/deploy/ddgc-201402181145) and making ~/live a symlink to allow for atomic upgrade process and rollback in one step?

This should probably be explored.

Does the perlbrew/DDGC environment need to be imported multiple times?

No, this is an artifact of an earlier iteration of the deployment method and can safely be removed.

Observations on deployment run-through

Issues:

  • Syncing versions between staging and live, especially regarding deployed schemas.
  • Following on from this, there appears to be no easy way to know the state of a given server, the last deployed version and so on.
  • Schema changes require no small amount of manual intervention.
  • Dependency resolution appears to be incomplete, deployment often stalled on errors to do with missing cpan dists.
  • Travis CI configs (and possibly other files) may be changed by processes invoked by release.

Recommendations:

  • Start with dzil build or dzil test --release to ensure we are in a fit state to deploy and generated files like travis.yml are committed.
  • While The ability to have some elasticity in schema definition is incredibly helpful to iterative development processes, there should perhaps be some discussion of proposed schema changes. Deployment difficulties and data loss are potential issues. (Is it going too far to suggest a database backup be performed as part of deployment any time there are schema changes?).
  • There should be some note taken of code and schema versions which have been deployed to staging/live platforms (i.e. how far ahead staging's version is, the steps required to bring live to the same state - a short deployment doc with bullet point steps required. Should there be a footer element with current version visible to admins?). Potentially one could add new deployment steps to git log messages under a deployment header to be pulled out by a script which could parse these out of log entries between tagged versions. Something as simple as this in the commit message - ideally these cases should be handled automatically, but some awareness up-front of potential issues never hurts:
commit c85320d9ddb90c13f4a215f1f0a87b531ab3a410
Author: John Barrett <[email protected]>
Date:   Mon Feb 24 17:44:42 2014 +0000

New foobar magic.

Deployment:

- This version has schema changes - run ddgc_db_autoupgrade.pl
- New dependency - Foo::Bar::Magic
  • Perhaps Dist::Zilla::Plugin::AutoPrereqs could be added to dist.ini for additional dependency discovery. This generates a file which could trivially be incorporated into an automation system like Chef.
  • Commands with useful output in scripts should not be silenced (ssh -q in deploy script)
  • Are all merges to master discussed? Should there be a long-term 'dev' branch to focus development activity on with master branch reserved for deployment ready versions?

ddgc_db_autoupgrade.pl

This script generates a SQL::Translator::Diff between the currently deployed schema and the new one described in DBIx::Class schema in DDGC::DB.

Questions / Recommendations:

Should we use schema versioning + DBIx::Class::DeploymentHandler?

It should perhaps be explored. Changes should occur within a transaction either way.

ddgc_web_fastcgi.pl

Launched by daemontools like so:

perl /home/ddgc/live/script/ddgc_web_fastcgi.pl --listen 127.0.0.1:8989 --nproc 5 -p /home/ddgc/web.pid ddgc 2>&1

Uses Catalyst::ScriptRunner:

Catalyst::ScriptRunner->run('DDGC::Web', 'FastCGI');

...to run Catalyst::Script::FastCGI with the DDGC::Web application loaded.

nginx configs - (view.)duck.co

With 5 workers available on http://127.0.0.1:8989/, much of the work here amounts to serving this port. Notable config elements:

upstream ddgc {
        server 127.0.0.1:8989;
}

Uses HttpUpstreamModule to create a http endpoint, ddgc. HttpUpstreamModule can take several servers and will round-robin load balance between them.

The server definition defines some static elements, among them:

        root            /home/ddgc/docroot;

        location /generated_css {
                alias /home/ddgc/ddgc/cache/generated_css;
        }

        location /robots.txt {
                alias /home/ddgc/ddgc/robots.txt;
        }

Others are defined in a similar fashion.

        include         fastcgi_params_plack;
        fastcgi_param   HTTPS   on;

The fastcgi_params_plack file contains parameters which should be passed to the application backend, such as REQUEST_URI. We also sets the HTTPS parameter (Apache would usually pass this parameter to its (Fast)CGI applications).

        location @ddgc {
                if (-f /home/ddgc/ddgc_web_maintenance_long) {
                        return 503;
                }
                if (-f /home/ddgc/ddgc_web_maintenance) {
                        return 503;
                }
                fastcgi_pass ddgc;
        }

This is what passes requests to upstream ddgc, but only if there is no maintenance ongoing.

There is a / location matcher which redirects to ``@ddgcwhen no other conditions are matched - this attempts to find static files at$uri` before forwarding the request to `@ddgc`:

        location / {
                try_files $uri @ddgc;
        }

The @maintenance location decides which error page to serve depending on the flag file present in ~/ddgc:

        location @maintenance {
                if (-f /home/ddgc/ddgc_web_maintenance) {
                        rewrite ^ /maintenance.html break;
                }
                if (-f /home/ddgc/ddgc_web_maintenance_long) {
                        rewrite ^ /long_maintenance.html break;
                }
        }       

        error_page 503 @maintenance;

Errors are redirected to a static page, apart from 404s, which go to the Roboduck 404 page in DDGC:

        error_page 403 405 502 504 @error;
        error_page 404 @ddgc;

        location @error {
                rewrite ^ /error.html break;
        }

The remaining sections redirect HTTP requests to HTTPS and help.(view.)dukgo.com requests to (view.)dukgo.com/help/

Questions / Recommendations:

The definition for location /ddgc_static refers to site_perl/5.16.2/auto/share/dist/DDGC-Static with a warning "will need to changed if we update perl" - should this be moved completely? It appears to be the only thing depending on our perl version.

nginx configs - (view.)duckpan.org

Simply serves the static pages in /home/ddgc/ddgc/duckpan, in summary:

        index   index.html;
        location / {
                root                    /home/ddgc/ddgc/duckpan;
        }

Set the default index page to index.html (index pages contain the Dashing DuckPAN Duck ASCII : http://duckpan.org/ - smashing!)

        error_page      403     /forbidden.html;

Serve forbidden.html for any forbidden pages - requests for directory listings and such.

        error_page      404     = @404;
        
        location @404 {
                rewrite ^/perldoc/(.*) https://metacpan.org/module/$1 redirect;
                rewrite ^ /notfound.html redirect;
        }

Requests for non-existant pages are:

  • served notfound.html in normal circumstances or...
  • Redirected to metacpan if they are a request for non-existent perl module docs.

Questions / Recommendations:

Should duckpan.org have a config for HTTPS (even with a self signed cert)?

A misbehaving or misconfigured client requesting HTTPS urls with be redirected to the first site nginx configured with SSL (in this case, duck.co) and be returned a 404.

Is the metacpan redirect useful? Not sure what the use-case is.

Clone this wiki locally