feat: grant cross-account access to Superset

[patheard]

Summary

Add S3 bucket policies and a Glue data catalog policy that allow Superset to access the data lake's transformed and curated data.

Related

• https://github.com/cds-snc/platform-core-services/issues/610

[github-actions]

Production: buckets 🪣

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 3 to add, 0 to change, 0 to destroy

Show summary

CHANGE	NAME
add	aws_s3_bucket_policy.athena_bucket
	aws_s3_bucket_policy.curated_bucket
	aws_s3_bucket_policy.transformed_bucket

Show plan

Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket_policy.athena_bucket will be created
  + resource "aws_s3_bucket_policy" "athena_bucket" {
      + bucket = "cds-data-lake-athena-production"
      + id     = (known after apply)
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "s3:PutObject",
                          + "s3:ListBucket",
                          + "s3:GetObject",
                          + "s3:GetBucketLocation",
                          + "s3:AbortMultipartUpload",
                        ]
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::066023111852:role/SupersetAthenaRead"
                        }
                      + Resource  = [
                          + "arn:aws:s3:::cds-data-lake-athena-production/*",
                          + "arn:aws:s3:::cds-data-lake-athena-production",
                        ]
                      + Sid       = "SupersetReadWrite"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
    }

  # aws_s3_bucket_policy.curated_bucket will be created
  + resource "aws_s3_bucket_policy" "curated_bucket" {
      + bucket = "cds-data-lake-curated-production"
      + id     = (known after apply)
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "s3:ListBucket",
                          + "s3:GetObject",
                          + "s3:GetBucketLocation",
                        ]
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::066023111852:role/SupersetAthenaRead"
                        }
                      + Resource  = [
                          + "arn:aws:s3:::cds-data-lake-curated-production/*",
                          + "arn:aws:s3:::cds-data-lake-curated-production",
                        ]
                      + Sid       = "SupersetRead"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
    }

  # aws_s3_bucket_policy.transformed_bucket will be created
  + resource "aws_s3_bucket_policy" "transformed_bucket" {
      + bucket = "cds-data-lake-transformed-production"
      + id     = (known after apply)
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "s3:ListBucket",
                          + "s3:GetObject",
                          + "s3:GetBucketLocation",
                        ]
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::066023111852:role/SupersetAthenaRead"
                        }
                      + Resource  = [
                          + "arn:aws:s3:::cds-data-lake-transformed-production/*",
                          + "arn:aws:s3:::cds-data-lake-transformed-production",
                        ]
                      + Sid       = "SupersetRead"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Warning: Argument is deprecated

  with module.athena_bucket.aws_s3_bucket.this,
  on .terraform/modules/athena_bucket/S3/main.tf line 8, in resource "aws_s3_bucket" "this":
   8: resource "aws_s3_bucket" "this" {

Use the aws_s3_bucket_lifecycle_configuration resource instead

(and 22 more similar warnings elsewhere)

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"

Show Conftest results

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions

[github-actions]

Production: glue 🧴

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 1 to add, 0 to change, 0 to destroy

Show summary

CHANGE	NAME
add	aws_glue_resource_policy.cross_account_access

Show plan

Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_glue_resource_policy.cross_account_access will be created
  + resource "aws_glue_resource_policy" "cross_account_access" {
      + id     = (known after apply)
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "glue:GetTables",
                          + "glue:GetTableVersions",
                          + "glue:GetTableVersion",
                          + "glue:GetTable",
                          + "glue:GetPartitions",
                          + "glue:GetPartition",
                          + "glue:GetDatabases",
                          + "glue:GetDatabase",
                          + "glue:BatchGetPartition",
                        ]
                      + Effect    = "Allow"
                      + Principal = {
                          + AWS = "arn:aws:iam::066023111852:role/SupersetAthenaRead"
                        }
                      + Resource  = "arn:aws:glue:ca-central-1:739275439843:*"
                      + Sid       = "SupersetReadAccess"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"

Show Conftest results

WARN - plan.json - main - Missing Common Tags: ["aws_glue_catalog_database.operations_aws_production"]
WARN - plan.json - main - Missing Common Tags: ["aws_glue_crawler.operations_aws_production_account_tags"]
WARN - plan.json - main - Missing Common Tags: ["aws_glue_crawler.operations_aws_production_cost_usage_report"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.glue_crawler"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.glue_etl"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.glue_crawler"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.glue_etl"]
WARN - plan.json - main - Missing Common Tags: ["aws_kms_key.aws_glue"]

27 tests, 19 passed, 8 warnings, 0 failures, 0 exceptions