Setup IAM users
- Create the users: "Attach existing policies directly" with "Administrator Access" and "Billing" to view billing
- Install awscli
$ pip3 install awscli
$ aws --version
aws-cli/1.18.39 Python/3.7.7 Darwin/19.4.0 botocore/1.15.39
- Configure AWS CLI:
- AWS regions and codes: doc
- US East (N. Virginia), code: us-east-1, slightly cheaper than Singapore
- Asia Pacific (Singapore), code
ap-southeast-1
- AWS CLI uses two files to store the sensitive credential information (in ~/.aws/credentials) separated from the less sensitive configuration options (in ~/.aws/config). See the doc.
- Run the following commands to setup and check cli is working (s3 does not have region)
$ aws configure AWS Access Key ID [None]: A6NxxxxxZDKKB AWS Secret Access Key [None]: VLxxxx2fZWhr Default region name [None]: us-east-1 Default output format [None]: json $ ls ~/.aws config credentials $ aws s3 ls - AWS regions and codes: doc
AWS ECR (Elastic Container Registry) is part of AWS ECS (Elastic Container Service):
Go to Amazon ECS, click "Repositories", and then add two new repositories:
- fred-backend:
aws ecr create-repository --repository-name fred-backend --region us-east-1 - fred-frontend:
aws ecr create-repository --repository-name fred-frontend --region us-east-1
Get the <AWS_ACCOUNT_ID> and the related URIs as shown in the screenshot above and build the images locally with tags (dev in this case):
docker build \
-f services/backend/Dockerfile \
-t <ECR-fred-backend-repo-URI>:dev \
./services/backend
docker build \
-f services/frontend/Dockerfile \
-t <ECR-fred-frontend-repo-URI>:dev \
./services/frontend
After this, do docker image ls locally, you should see the two newly built images:
Now, we are going to push the dev build images to ECR. If you check the repositories your created above, they should be empty:
Next, let's push the two images you just built locally on your computer to the repositories.
First, you need to login to AWS ECR by using the following command: aws ecr get-login-password --region us-east-1 generates the long password, which is piped to docker login to login with default username "AWS" - don't forget to change the account id or the region you chose. See doc
aws ecr get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin \
<AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
Next, use the following commands to push the images to the ECR repositories (it may take a while given the image sizes are about 400M and 800M):
docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/fred-backend:dev
docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/fred-frontend:dev
Once this is done, you should be able to the images in ECR:
Create/update the following files:
- update
services/db/create.sql: addCREATE DATABASE app_prod; - add
services/backend/Dockerfile.prod - add
services/frontend/Dockerfile.prod - add
services/frontend/nginx/aws/default.conf: this file is used to overwrite the default nginx configuration - add
docker-compose.prod.yml
Then, test out the production images locally:
$ docker-compose down -v
$ docker-compose -f docker-compose.prod.yml up -d --build
$ docker-compose exec backend python manage.py reset_db
$ docker-compose exec backend python manage.py load_data
Test it out at http://localhost:3007/
Build the production images with the prod tag:
$ docker build \
-f services/backend/Dockerfile.prod \
-t <ECR-fred-backend-repo-URI>:prod \
./services/backend
$ docker build \
-f services/frontend/Dockerfile.prod \
-t <ECR-fred-frontend-repo-URI>:prod \
--build-arg NODE_ENV=production \
--build-arg REACT_APP_USERS_SERVICE_URL=${REACT_APP_USERS_SERVICE_URL} \
./services/frontend
You should see the new production images using docker image ls (you can see the frontend production image size is much smaller than the dev image):
Now, push the production images to ECR:
$ docker push <ECR-fred-backend-repo-URI>:prod
$ docker push <ECR-fred-frontend-repo-URI>:prod
You can see the production images on ECR:
Go to the CodeBuild dashboard and click "Create project":
Follow the screenshots below:
NOTE: add the AmazonEC2ContainerRegistryPowerUser policy to the codebuild-my-fra-service-role role using IAM dashboard.
Then, add buildspec.yml, push the code to Github and build in CodeBuild. Make sure it succeeded:
My AWS is quite old and was using EC2-Class. The new accounts are created only for EC2-VPC. I have to contact AWS to switch to the new EC2-VPC.
Each AWS has a default VPC for each region. If the default VPC is deleted. You can use the following command to re-create one: aws ec2 create-default-vpc
Create "Application Load Balancer" (ALB):
Don't forget to change the value of REACT_APP_USERS_SERVICE_URL in buildspec.yml to the DNS name of the ALB.
Essentially, one ALB is created with one listener at port 80 and then have two rules to forward traffic:
- if path is /, forward to the frontend task group
- if other paths (/users /auth /etc.), forward to the backend task group at port 5000. NOTE only five paths can be specified in the rule.
Get the password with master user name: fredapp:
Get the status of database:
$ aws --region us-east-1 rds describe-db-instances \
--db-instance-identifier fred-db \
--query 'DBInstances[].{DBInstanceStatus:DBInstanceStatus}'
$ aws --region us-east-1 rds describe-db-instances \
--db-instance-identifier fred-db
Run the following command to get the address of the database:
aws --region us-east-1 rds describe-db-instances \
--db-instance-identifier fred-db \
--query 'DBInstances[].{Address:Endpoint.Address}'
[
{
"Address": "fred-db.coaqhaja4wc9.us-east-1.rds.amazonaws.com"
}
]
So, the production URI is:
postgres://fredapp:<YOUR_PASSWORD>@<YOUR_ADDRESS>:5432/fred_prod
update security group rule to make healthy check pass:
- Task Definition:
- Tasks
- Services
- Clusters
Cluster has services, service has tasks
get the EC2 instance of the backend to login to initialize the database:
dami:fred harrywang$ chmod 400 ~/sandbox/keys/fred-aws.pem
dami:fred harrywang$ ssh -i ~/sandbox/keys/fred-aws.pem [email protected]
[ec2-user@ip-172-31-84-8 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
017ff3d54c29 991046682610.dkr.ecr.us-east-1.amazonaws.com/fred-backend:prod "/bin/sh -c 'gunicor…" 4 minutes ago Up 4 minutes 0.0.0.0:32770->5000/tcp ecs-fred-backend-td-1-backend-e09bb2fc80e288904f00
de0d72895173 amazon/amazon-ecs-agent:latest "/agent" 22 minutes ago Up 22 minutes (healthy) ecs-agent
$ docker exec -it 017ff3d54c29 bash
root@017ff3d54c29:/usr/src/app# python manage.py reset_db
database reset done!
root@017ff3d54c29:/usr/src/app# python manage.py load_data
user table loaded
author and quote tables loaded
Now go to http://LOAD_BALANCER_DNS_NAME to test out!!
add the following files:
- ecs/ecs_backend_task_definition.json
- ecs/ecs_frontend_task_definition.json
- deploy.sh
update buildspec.yml:
post_build:
commands:
- echo pushing prod images to ecr...
- docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/fred-backend:prod
- docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/fred-frontend:prod
- chmod +x ./deploy.sh
- bash deploy.sh
Then, go to CodeBuild to add more environment variables:
Then, add a security policy to the role:
4 EC2 instances were started - you may want to shutdown them to avoid charges.
Delete the clusters will terminate all related EC2 instances.