Skip to content

Commit

Permalink
Merge branch 'main' into marrobi/service-bus-standard
Browse files Browse the repository at this point in the history
  • Loading branch information
jonnyry authored Jan 10, 2025
2 parents 33effe2 + 3704002 commit f56fa2d
Show file tree
Hide file tree
Showing 38 changed files with 557 additions and 569 deletions.
26 changes: 18 additions & 8 deletions .github/workflows/build_validation_develop.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
name: Build Validation

on: # yamllint disable-line rule:truthy
on: # yamllint disable-line rule:truthy
pull_request:
branches:
- main
- 'feature/**'
- "feature/**"

# for each ref (branch/pr) run just the most recent,
# cancel other pending/running ones
Expand Down Expand Up @@ -52,6 +52,16 @@ jobs:
terraform_workspace_services:
- templates/workspace_services/**/terraform/**/*.tf
- uses: hashicorp/setup-terraform@v3
if: ${{ steps.filter.outputs.terraform == 'true' }}
with:
terraform_version: "1.9.8"

- uses: hashicorp/setup-terraform@v3
if: ${{ steps.filter.outputs.terraform == 'true' }}
with:
terraform_version: "1.9.8"

- name: Terraform format check
if: ${{ steps.filter.outputs.terraform == 'true' }}
run: terraform fmt -check -recursive
Expand Down Expand Up @@ -107,7 +117,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VALIDATE_TERRAFORM_TFLINT: true
TERRAFORM_TFLINT_CONFIG_FILE: .tflint_core.hcl
FILTER_REGEX_INCLUDE: './core/.*'
FILTER_REGEX_INCLUDE: "./core/.*"

- name: Workspace Tags
if: ${{ steps.filter.outputs.terraform_workspaces == 'true' }}
Expand All @@ -118,7 +128,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VALIDATE_TERRAFORM_TFLINT: true
TERRAFORM_TFLINT_CONFIG_FILE: .tflint_workspaces.hcl
FILTER_REGEX_INCLUDE: './templates/workspaces/.*'
FILTER_REGEX_INCLUDE: "./templates/workspaces/.*"

- name: Workspace Services Tags
if: ${{ steps.filter.outputs.terraform_workspace_services == 'true' }}
Expand All @@ -129,8 +139,8 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VALIDATE_TERRAFORM_TFLINT: true
TERRAFORM_TFLINT_CONFIG_FILE: .tflint_workspace_services.hcl
FILTER_REGEX_INCLUDE: './templates/workspaces/.*'
FILTER_REGEX_EXCLUDE: '.*user_resource.*'
FILTER_REGEX_INCLUDE: "./templates/workspaces/.*"
FILTER_REGEX_EXCLUDE: ".*user_resource.*"

- name: User Resources Tags
if: ${{ steps.filter.outputs.terraform_workspace_services == 'true' }}
Expand All @@ -141,7 +151,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VALIDATE_TERRAFORM_TFLINT: true
TERRAFORM_TFLINT_CONFIG_FILE: .tflint_user_resources.hcl
FILTER_REGEX_INCLUDE: './templates/workspace_services/.*/user_resources/.*'
FILTER_REGEX_INCLUDE: "./templates/workspace_services/.*/user_resources/.*"

- name: Shared Services Tags
if: ${{ steps.filter.outputs.terraform_shared_services == 'true' }}
Expand All @@ -152,4 +162,4 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VALIDATE_TERRAFORM_TFLINT: true
TERRAFORM_TFLINT_CONFIG_FILE: .tflint_shared_services.hcl
FILTER_REGEX_INCLUDE: './templates/shared_services/.*'
FILTER_REGEX_INCLUDE: "./templates/shared_services/.*"
8 changes: 4 additions & 4 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@
#
name: "CodeQL"

on: # yamllint disable-line rule:truthy
on: # yamllint disable-line rule:truthy
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '41 3 * * 5'
- cron: "41 3 * * 5"

# for each ref (branch/pr) run just the most recent,
# cancel other pending/running ones
Expand All @@ -29,7 +29,7 @@ concurrency:
jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
permissions:
actions: read
contents: read
Expand All @@ -38,7 +38,7 @@ jobs:
strategy:
fail-fast: false
matrix:
language: ['python', 'java', 'javascript', 'typescript']
language: ["python", "java", "javascript", "typescript"]

steps:
- name: Checkout repository
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ ENHANCEMENTS:
* Split log entries with [Log chunk X of Y] for better readability. ([#3992](https://github.com/microsoft/AzureTRE/issues/3992))
* Expose APP_SERVICE_SKU build variable to allow enablement of App Gateway WAF ([#4111](https://github.com/microsoft/AzureTRE/pull/4111))
* Update Terraform to use Azure AD authentication rather than storage account keys ([#4103](https://github.com/microsoft/AzureTRE/issues/4103))
* Consolidate Terraform upgrade scripts ([#4099](https://github.com/microsoft/AzureTRE/issues/4099))
* Storage accounts should use infrastructure encryption ([#4001](https://github.com/microsoft/AzureTRE/issues/4001))
* Update obsolete Terraform properties ([#4136](https://github.com/microsoft/AzureTRE/issues/4136))
* Update Guacamole version and dependencies ([#4140](https://github.com/microsoft/AzureTRE/issues/4140))
Expand All @@ -31,6 +32,8 @@ ENHANCEMENTS:
* Upgrade Python version from 3.8 to 3.12 ([#3949](https://github.com/microsoft/AzureTRE/issues/3949))Upgrade Python version from 3.8 to 3.12 (#3949)
* Disable storage account key usage ([[#4227](https://github.com/microsoft/AzureTRE/issues/4227)])
* Update Guacamole dependencies ([[#4232](https://github.com/microsoft/AzureTRE/issues/4232)])
* Add option to force tunnel TRE's Firewall ([#4237](https://github.com/microsoft/AzureTRE/issues/4237))
* Add EventGrid diagnostics to identify airlock issues ([#4258](https://github.com/microsoft/AzureTRE/issues/4258))

BUG FIXES:
* Update KeyVault references in API to use the version so Terraform cascades the update ([#4112](https://github.com/microsoft/AzureTRE/pull/4112))
Expand Down
8 changes: 5 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ terraform-upgrade:
&& . ${MAKEFILE_DIR}/devops/scripts/check_dependencies.sh env \
&& . ${MAKEFILE_DIR}/devops/scripts/load_and_validate_env.sh \
&& . ${MAKEFILE_DIR}/devops/scripts/load_env.sh ${DIR}/.env \
&& cd ${DIR}/terraform/ && ./upgrade.sh
&& ./devops/scripts/upgrade.sh ${DIR}

terraform-import:
$(call target_title, "Importing ${DIR} with Terraform") \
Expand Down Expand Up @@ -309,8 +309,10 @@ deploy-shared-service:
&& ${MAKEFILE_DIR}/devops/scripts/deploy_shared_service.sh $${PROPS}

firewall-install:
$(MAKE) bundle-build bundle-publish bundle-register deploy-shared-service \
DIR=${MAKEFILE_DIR}/templates/shared_services/firewall/ BUNDLE_TYPE=shared_service
. ${MAKEFILE_DIR}/devops/scripts/check_dependencies.sh env \
&& $(MAKE) bundle-build bundle-publish bundle-register deploy-shared-service \
DIR=${MAKEFILE_DIR}/templates/shared_services/firewall/ BUNDLE_TYPE=shared_service \
PROPS="$${FIREWALL_SKU+--firewall_sku $${FIREWALL_SKU} }$${FIREWALL_FORCE_TUNNEL_IP+--firewall_force_tunnel_ip $${FIREWALL_FORCE_TUNNEL_IP} }"

static-web-upload:
$(call target_title, "Uploading to static website") \
Expand Down
3 changes: 2 additions & 1 deletion config.sample.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ management:
acr_name: __CHANGE_ME__
# ID of external Key Vault to store CMKs in (only required if enable_cmk_encryption is true)
# external_key_store_id: __CHANGE_ME__
# Name of Key Vault for encryption keys, required only if enable_cmk_encryption is true and not using external_key_store_id
# Name of Key Vault for encryption, required if enable_cmk_encryption is true and external_key_store_id is not set
# encryption_kv_name: __CHANGE_ME__
# Azure Resource Manager credentials used for CI/CD pipelines
arm_subscription_id: __CHANGE_ME__
Expand Down Expand Up @@ -46,6 +46,7 @@ tre:
# The TRE Web UI is deployed by default.
# Uncomment the following to disable deployment of the Web UI.
# deploy_ui: false
# firewall_force_tunnel_ip: __CHANGE_ME__
firewall_sku: Standard
# The SKU of the Service Bus to use. Options are "Standard" or "Premium". For production, use Premium
servicebus_sku: Standard
Expand Down
8 changes: 0 additions & 8 deletions core/terraform/airlock/airlock_processor.tf
Original file line number Diff line number Diff line change
@@ -1,11 +1,3 @@
data "local_file" "airlock_processor_version" {
filename = "${path.root}/../../airlock_processor/_version.py"
}

locals {
version = replace(replace(replace(data.local_file.airlock_processor_version.content, "__version__ = \"", ""), "\"", ""), "\n", "")
}

resource "azurerm_service_plan" "airlock_plan" {
name = "plan-airlock-${var.tre_id}"
resource_group_name = var.resource_group_name
Expand Down
21 changes: 21 additions & 0 deletions core/terraform/airlock/data.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
data "local_file" "airlock_processor_version" {
filename = "${path.root}/../../airlock_processor/_version.py"
}

data "azurerm_private_dns_zone" "eventgrid" {
name = module.terraform_azurerm_environment_configuration.private_links["privatelink.eventgrid.azure.net"]
resource_group_name = var.resource_group_name
}

data "azurerm_container_registry" "mgmt_acr" {
name = var.mgmt_acr_name
resource_group_name = var.mgmt_resource_group_name
}

data "azurerm_monitor_diagnostic_categories" "eventgrid_custom_topics" {
resource_id = azurerm_eventgrid_topic.airlock_notification.id
}

data "azurerm_monitor_diagnostic_categories" "eventgrid_system_topics" {
resource_id = azurerm_eventgrid_system_topic.export_approved_blob_created.id
}
52 changes: 47 additions & 5 deletions core/terraform/airlock/eventgrid_topics.tf
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
data "azurerm_private_dns_zone" "eventgrid" {
name = module.terraform_azurerm_environment_configuration.private_links["privatelink.eventgrid.azure.net"]
resource_group_name = var.resource_group_name
}

# Below we assign a SYSTEM-assigned identity for the topics. note that a user-assigned identity will not work.

# Event grid topics
Expand Down Expand Up @@ -511,3 +506,50 @@ resource "azurerm_eventgrid_event_subscription" "export_approved_blob_created" {
]
}

resource "azurerm_monitor_diagnostic_setting" "eventgrid_custom_topics" {
for_each = merge({
(azurerm_eventgrid_topic.airlock_notification.name) = azurerm_eventgrid_topic.airlock_notification.id,
(azurerm_eventgrid_topic.step_result.name) = azurerm_eventgrid_topic.step_result.id,
(azurerm_eventgrid_topic.status_changed.name) = azurerm_eventgrid_topic.status_changed.id,
(azurerm_eventgrid_topic.data_deletion.name) = azurerm_eventgrid_topic.data_deletion.id,
},
var.enable_malware_scanning ? { (azurerm_eventgrid_topic.scan_result[0].name) = azurerm_eventgrid_topic.scan_result[0].id } : null
)

name = "${each.key}-diagnostics"
target_resource_id = each.value
log_analytics_workspace_id = var.log_analytics_workspace_id
dynamic "enabled_log" {
for_each = data.azurerm_monitor_diagnostic_categories.eventgrid_custom_topics.log_category_types
content {
category = enabled_log.value
}
}

metric {
category = "AllMetrics"
}
}

resource "azurerm_monitor_diagnostic_setting" "eventgrid_system_topics" {
for_each = {
(azurerm_eventgrid_system_topic.import_inprogress_blob_created.name) = azurerm_eventgrid_system_topic.import_inprogress_blob_created.id,
(azurerm_eventgrid_system_topic.import_rejected_blob_created.name) = azurerm_eventgrid_system_topic.import_rejected_blob_created.id,
(azurerm_eventgrid_system_topic.import_blocked_blob_created.name) = azurerm_eventgrid_system_topic.import_blocked_blob_created.id,
(azurerm_eventgrid_system_topic.export_approved_blob_created.name) = azurerm_eventgrid_system_topic.export_approved_blob_created.id,
}

name = "${each.key}-diagnostics"
target_resource_id = each.value
log_analytics_workspace_id = var.log_analytics_workspace_id
dynamic "enabled_log" {
for_each = data.azurerm_monitor_diagnostic_categories.eventgrid_system_topics.log_category_types
content {
category = enabled_log.value
}
}

metric {
category = "AllMetrics"
}
}
5 changes: 0 additions & 5 deletions core/terraform/airlock/identity.tf
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
data "azurerm_container_registry" "mgmt_acr" {
name = var.mgmt_acr_name
resource_group_name = var.mgmt_resource_group_name
}

resource "azurerm_user_assigned_identity" "airlock_id" {
resource_group_name = var.resource_group_name
location = var.location
Expand Down
2 changes: 2 additions & 0 deletions core/terraform/airlock/locals.tf
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
locals {
version = replace(replace(replace(data.local_file.airlock_processor_version.content, "__version__ = \"", ""), "\"", ""), "\n", "")

# STorage AirLock EXternal
import_external_storage_name = lower(replace("stalimex${var.tre_id}", "-", ""))
# STorage AirLock IMport InProgress
Expand Down
2 changes: 1 addition & 1 deletion core/terraform/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ set -o nounset
# shellcheck disable=SC1091
source ./migrate.sh

PLAN_FILE="tfplan$$"
TS=$(date +"%s")
PLAN_FILE="${TS}-tre-core.tfplan"
LOG_FILE="${TS}-tre-core.log"

# This variables are loaded in for us
Expand Down
13 changes: 0 additions & 13 deletions core/terraform/upgrade.sh

This file was deleted.

2 changes: 1 addition & 1 deletion core/version.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.11.15"
__version__ = "0.11.17"
61 changes: 61 additions & 0 deletions devops/scripts/upgrade.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/bin/bash
set -e

# This script is used to upgrade terraform providers in a specified directory

# Usage: ./upgrade.sh <directory>

DIR=$1

# Load environment variables from .env file
if [ -f "$DIR/.env" ]; then
set -a
# shellcheck source=/dev/null
. "$DIR/.env"
set +a
fi

# Ensure TF_VAR_mgmt_resource_group_name is set
if [ -z "${TF_VAR_mgmt_resource_group_name}" ]; then
echo "Error: TF_VAR_mgmt_resource_group_name is not set."
exit 1
fi

# Ensure TF_VAR_mgmt_storage_account_name is set
if [ -z "${TF_VAR_mgmt_storage_account_name}" ]; then
echo "Error: TF_VAR_mgmt_storage_account_name is not set."
exit 1
fi

# Ensure TF_VAR_terraform_state_container_name is set
if [ -z "${TF_VAR_terraform_state_container_name}" ]; then
echo "Error: TF_VAR_terraform_state_container_name is not set."
exit 1
fi

# Ensure TRE_ID is set
if [ -z "${TRE_ID}" ]; then
echo "Error: TRE_ID is not set."
exit 1
fi

# Infer the key from the directory names
PARENT_DIR=$(basename "$(dirname "$DIR")")
GRANDPARENT_DIR=$(basename "$(dirname "$(dirname "$DIR")")")

if [[ "$GRANDPARENT_DIR" == "workspaces" || "$GRANDPARENT_DIR" == "shared_services" ]]; then
KEY="${TRE_ID?}_${TF_VAR_id?}_${PARENT_DIR}"
elif [[ "$GRANDPARENT_DIR" == "workspace_services" ]]; then
KEY="${TRE_ID?}_${TF_VAR_workspace_id?}_${TF_VAR_id?}_${PARENT_DIR}"
elif [[ "$GRANDPARENT_DIR" == "user_resources" ]]; then
KEY="${TRE_ID?}_${TF_VAR_workspace_id?}_${TF_VAR_workspace_service_id?}_${TF_VAR_id?}_${PARENT_DIR}"
else
KEY="${TRE_ID?}_${PARENT_DIR}"
fi

# Run terraform init with upgrade and reconfigure options
terraform -chdir="$DIR/terraform" init -upgrade -reconfigure -input=false -backend=true \
-backend-config="resource_group_name=${TF_VAR_mgmt_resource_group_name}" \
-backend-config="storage_account_name=${TF_VAR_mgmt_storage_account_name}" \
-backend-config="container_name=${TF_VAR_terraform_state_container_name}" \
-backend-config="key=${KEY}"
13 changes: 0 additions & 13 deletions devops/terraform/upgrade.sh

This file was deleted.

21 changes: 21 additions & 0 deletions docs/tre-admins/configure-firewall-force-tunneling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Forced Tunneling to External Firewall in TRE

Azure TRE deploys and manages an Azure firewall to ensure creation of workspace level rules can be automated when TRE workspaces and other services are created without manual intervention.
It is highly recommended leaving the Azure TRE firewall in place. If there is still the requirement to send all traffic through a centralized enterprise firewall, such as that deployed as part of an Azure landing zone, then forced tunnelling should be used. The centralized firewall will need a superset of rules used by the TRE.

To setup forced tunneling to an external firewall, follow these steps:

## 1. Set the firewall_force_tunnel_ip parameter in the config.yaml file
Provide the external firewall's IP address:

```json
firewall_force_tunnel_ip: 192.168.0.4
```
This automatically creates a route table to direct TRE’s traffic to the specified IP.

## 2. Manually Connect TRE to Your Firewall
Configure connectivity between TRE’s VNet and your external firewall using one of the following methods:

1. **VNet Peering**: Peer the TRE VNet with your firewall’s VNet.
1. **ExpressRoute**: Use a private connection for firewalls located on-premises.
1. **Site-to-Site VPN**: Establish a VPN connection as an alternative.
Loading

0 comments on commit f56fa2d

Please sign in to comment.