feat: add Cost and Usage Glue visual ETL job

[patheard]

Summary

Add an export of the Cost and Usage Glue visual ETL job definition. There is currently no way to manage visual ETL jobs using Terraform, but this will give us a backup of the job definition in case we need to revert a change.

A future PR will have a nightly sync job that checks for new or modified visual ETL jobs and creates automated PRs with the changes.

⚠️ Note that this PR also renames the existing Glue crawlers so they have a consistent naming convention.

Related

• https://github.com/cds-snc/platform-core-services/issues/610

[github-actions]

Production: glue 🧴

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 2 to add, 0 to change, 2 to destroy

Show summary

CHANGE	NAME
recreate	aws_glue_crawler.operations_aws_production_account_tags
	aws_glue_crawler.operations_aws_production_cost_usage_report

Show plan

Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_glue_crawler.operations_aws_production_account_tags must be replaced
-/+ resource "aws_glue_crawler" "operations_aws_production_account_tags" {
      ~ arn                    = "arn:aws:glue:ca-central-1:739275439843:crawler/Organization Account Tags" -> (known after apply)
      ~ configuration          = jsonencode(
          ~ {
              ~ Version              = 1.0 -> 1
                # (2 unchanged attributes hidden)
            }
        )
      ~ id                     = "Organization Account Tags" -> (known after apply)
      ~ name                   = "Organization Account Tags" -> "Operations / AWS / Organization / Account Tags" # forces replacement
      ~ role                   = "service-role/AWSGlueCrawler-DataLake" -> "arn:aws:iam::739275439843:role/service-role/AWSGlueCrawler-DataLake"
      - tags                   = {} -> null
        # (7 unchanged attributes hidden)

      - lake_formation_configuration {
          - use_lake_formation_credentials = false -> null
            # (1 unchanged attribute hidden)
        }

      - lineage_configuration {
          - crawler_lineage_settings = "DISABLE" -> null
        }

      - recrawl_policy {
          - recrawl_behavior = "CRAWL_EVERYTHING" -> null
        }

      ~ s3_target {
          - exclusions          = [] -> null
          - sample_size         = 0 -> null
            # (4 unchanged attributes hidden)
        }

      - schema_change_policy {
          - delete_behavior = "DEPRECATE_IN_DATABASE" -> null
          - update_behavior = "UPDATE_IN_DATABASE" -> null
        }
    }

  # aws_glue_crawler.operations_aws_production_cost_usage_report must be replaced
-/+ resource "aws_glue_crawler" "operations_aws_production_cost_usage_report" {
      ~ arn                    = "arn:aws:glue:ca-central-1:739275439843:crawler/Cost and Usage Report 2.0" -> (known after apply)
      - classifiers            = [] -> null
      ~ configuration          = jsonencode(
          ~ {
              ~ Version              = 1.0 -> 1
                # (2 unchanged attributes hidden)
            }
        )
      ~ id                     = "Cost and Usage Report 2.0" -> (known after apply)
      ~ name                   = "Cost and Usage Report 2.0" -> "Operations / AWS / Cost and Usage Report" # forces replacement
      ~ role                   = "service-role/AWSGlueCrawler-DataLake" -> "arn:aws:iam::739275439843:role/service-role/AWSGlueCrawler-DataLake"
      - tags                   = {} -> null
        # (6 unchanged attributes hidden)

      - lake_formation_configuration {
          - use_lake_formation_credentials = false -> null
            # (1 unchanged attribute hidden)
        }

      - lineage_configuration {
          - crawler_lineage_settings = "DISABLE" -> null
        }

      - recrawl_policy {
          - recrawl_behavior = "CRAWL_EVERYTHING" -> null
        }

      ~ s3_target {
          - exclusions          = [] -> null
          - sample_size         = 0 -> null
            # (4 unchanged attributes hidden)
        }

      - schema_change_policy {
          - delete_behavior = "DEPRECATE_IN_DATABASE" -> null
          - update_behavior = "UPDATE_IN_DATABASE" -> null
        }
    }

Plan: 2 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"

Show Conftest results

WARN - plan.json - main - Missing Common Tags: ["aws_glue_catalog_database.operations_aws_production"]
WARN - plan.json - main - Missing Common Tags: ["aws_glue_crawler.operations_aws_production_account_tags"]
WARN - plan.json - main - Missing Common Tags: ["aws_glue_crawler.operations_aws_production_cost_usage_report"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.glue_crawler"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.glue_etl"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.glue_crawler"]
WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.glue_etl"]
WARN - plan.json - main - Missing Common Tags: ["aws_kms_key.aws_glue"]

27 tests, 19 passed, 8 warnings, 0 failures, 0 exceptions