Skip to content

Latest commit

 

History

History
870 lines (692 loc) · 38.3 KB

README.NewResourceFromTerraform.md

File metadata and controls

870 lines (692 loc) · 38.3 KB

Before You Start

This README entails instructions to implement a new resource CRD through Terraform Provider Google.

Add a New Resource

The instructions will help you add a new resource to Config Connector. A resource is a K8s representation of a GCP resource.

Overview

To add a new resource you add new metadata to ServiceMappings, generate a CRD, register a controller, enable the controller access to the CRD, add test contexts, test resource data, and update the documentation with information about the new resource as well as samples.

This example is written from the perspective of a developer adding a new resource, SpannerInstance. However, this resource has already been implemented! The intention is for you to map these instructions to the resource you are adding. For example, if you were to add PubSubTopic then you would replace instances of the string Spanner with PubSub and instances of the string SpannerInstance with PubSubTopic, etc.

Study the Resource

First, you'll need to find information on the resource. There are two sources of truth for the resource; the documentation of the Google Resource and API itself, and the documentation and code of the provider we use to wrap it.

Determine the Resource Name

The resource name is divided into two parts. First, the service name indicates the name of the GCP service that the resource belongs to, then, a specific name for that resource within the context of the service finishes the name.

The resource name is always Pascal case and singular (e.g. ComputeHealthCheck), while the API will use camel case for service and resource name, dot-separated and pluralized (e.g. compute.healthChecks), and Terraform uses snake case, sometimes with google_ prepended (e.g. google_compute_health_check).

Information from the provider

We use Terraform Provider google-beta as the underlying provider library for Config Connector resources, with a few custom patches. You should find a file from our provider under third_party/github.com/hashicorp/terraform-provider-google-beta/google-beta in the approximate format resource_<resource name>.go. If the file is not there, we likely need to update terraform provider.

If the resource is locational, there may be a locationality indicator, such as regional or global, between the service name and specific name.

If there are variations on the filename(s) for your resource other than locationality indicators or _migration after the resource name, the resource may be more difficult to implement.

Check to make sure these really are for your resource, not a different resources with a longer name (e.g. ComputeInstance, ComputeTargetInstanceGroup, and ComputeInstanceGroupManager are three different resources which all start with 'ComputeInstance').

Technically, all the information you need about the provider is in this file. However, it's not necessarily the most readable. Fortunately, the provider comes with documentation which is submitted to and published by Terraform. Search the Terraform documentation using the 'FILTER' box atop the list on the left. You should be able to find documentation for your resource. Make sure it's a resource and not a data source.

Information from Google

One nice thing about the Terraform documentation mentioned above is that it will often link to the relevant Google documentation. If it doesn't, you can attempt to find the documentation by searching the Google product offerings or just Googling it.

Reading the Google documentation is very important for making sure you have the correct service name, knowing how the resource is called in the API, and gaining context on how the resource might actually be used.

Information from other Config Connector contributors

Implementing a resource requires extensive knowledge of GCP, ServiceMappings, and the API quirks of your resource. If you're stuck or unclear on anything, ask for help!

Generate the Custom Resource Definition

Custom Resource Definitions (CRDs), are the mechanism by which K8s developers can add new APIs to their cluster. In the case of Config Connector, for each GCP resource there will eventually be an associated Config Connector CRD. The CRDs are generated by the pkg/crd/crdgeneration program.

API Server Objects with the Kind of a registered CRD are called Custom Resources (CRs). For Config Connector CRDs, there is a single dynamic controller which is responsible for actuating the declared state of all CRs onto GCP. The dynamic controller makes extensive use of the terraform library to communicate with GCP.

Both the crdgeneration program and the dynamic controller need additional metadata to properly create and operate on Config Connector CRDs. The metadata is stored in config/servicemappings. The service mappings are documented inline with their definition at pkg/apis/core/v1alpha1/servicemapping_types.go.

Create the Service Mappings file

For each GCP service, there is an associated Service Mappings file. Create the Service Mappings file:

  1. Determine the name of the service for the resource you are adding. In the case of SpannerInstance the name of the service is Spanner.

  2. Determine the name of the service mappings file. The file's name is $(lowercaseservicename).yaml. So, in the case of the Spanner service, the name of the file is spanner.yaml.

  3. If the file already exists, skip the rest of the steps in this section.

  4. In config/servicemappings, open spanner.yaml for editing.

  5. Paste the following into the file.

    apiVersion: core.cnrm.cloud.google.com/v1alpha1
kind: ServiceMapping
metadata:
  name: spanner.cnrm.cloud.google.com
  namespace: cnrm-system
spec:
  name: Spanner
  version: v1beta1
  serviceHostName: "spanner.googleapis.com"
  resources:

  6. Replace all instances of spanner and Spanner with the appropriate service name you determined in step 1.

Tip: You can use the auto-generated service mappings as a reference.

Add the Resource Config to Service Mappings

For each GCP resource, there is an associated ResourceConfig in a ServiceMappings file. Add the ResourceConfig for your resource:

Tip: You can use the auto-generated service mappings as a reference.

  1. Open the Service Mappings file from the previous section for editing. In the case of the Spanner service the file is named spanner.yaml.

  2. In the file, under resources, paste the following:

    resources:
  - name: google_spanner_instance
    kind: SpannerInstance
    metadataMapping:
      name: name
      labels: labels

  3. Replace the value for name with the name of the Terraform resource associated with your GCP resource.

  4. Replace the value for kind with the ConfigConnector Kind for your resource. The name should be in pascal case, i.e. ServiceNameResourceName, but with acronyms in uppercase.

  5. Edit the metadataMapping as appropriate. For this you will need to take a look at the terraform resource documentation, ResourceConfig examples for similar resources, and the pkg/apis/core/servicemapping_types documentation.

  6. Add directives if necessary. For reference, a "directive" is a specific term we're using to describe a special Terraform provider field that maps a unique behavior on top of the underlying resource, which can't be accomplished as part of a GET response. Similar to the step above, you'll need to look at Terraform resource documentation, Google Cloud API documentation, and the pkg/apis/core/servicemapping_types documentation to determine if any fields are directives. There isn't a foolproof way to tell whether a field is a directive , but a general rule of thumb to start with is to determine if there are any fields specific to Terraform and also don't appear in the Google Cloud API docs.

    For example, with Container Cluster we first compare the Terraform documentation to the the Google Cloud resource API documentation. One helpful thing to notice is that there is usually a note on the bottom of the Terraform documentation that will inform you if there are any Terraform-specific behaviors on this resource. With Container Cluster, you can see that remove_default_node_pool is not present in the API. Therefore, the container cluster resource in the container.yaml service mapping will have a directives section like this:

    - name: google_container_cluster
  kind: ContainerCluster
  ...
  directives:
    - remove_default_node_pool

  7. Add mutableButUnreadableFields if necessary. Fields may not exist in the underlying GCP resource, but are returned by Terraform on read and it's mutable. We need to add them into the mutableButUnreadableFields list in the service mapping. The difference between mutableButUnreadableFields and directives is, The mutableButUnreadableFields fields will still be a part of resource CRD spec, they are unreadable but the value of the fields can be modified. Similarly, you'll need to look at Terraform resource documentation, Google Cloud API documentation, and the pkg/apis/core/servicemapping_types documentation to determine if any fields are mutableButUnreadableFields. An example is password field. It is mutable because users may update their password through the API. However, GCP resource usually will not return the password value for security considerations.

     - name: google_container_cluster
   kind: ContainerCluster
   ...
   mutableButUnreadableFields:
      - min_master_version

  8. Add ignoredFields if necessary. Fields can't be supported or will result in the suboptimal UX (e.g. multi-kind reference in the legacy style) due to lack of feature support, and we need to ignore the fields right now: Add the fields into the ignoredFields list in the service mapping, file a GitHub issue to support this field, and add a TODO with the issue ID right above the entry in the ignoredFields list.

    - name: google_container_cluster
  kind: ContainerCluster
  ...
  ignoredFields:
      # TODO: https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/000
      - node_pool

  9. Fill out the resourceAvailableInAssetInventory. Set to false.

  10. Add hierarchicalReferences if they exist. Determine if the corresonding Terraform resource exposes one or more parent resources through top-level fields. These fields can have names like org_id, parent_org_id, parent_folder_id, folder, project, etc. Additionally, we consider billing_account to be a hierarchical reference, but this is a unique case and should be brought to the attention when you create your PR. Hierarchical references can be added to the servicemapping like so:

    // The hierarchical reference should be configured
// under both 'hierarchicalReferences` and `resourceReferences`
// lists.
hierarchicalReferences:
  - type: project    // one of 'project', 'folder', 'organization' and 'billingAccount'
    key: projectRef  // one of 'projectRef', 'folderRef', 'organizationRef' and 'billingAccountRef'
resourceReferences:
  - tfField: project
    key: projectRef. // must match the key defined in hierarchicalReferences list
    description: |-
      The project that this resource belongs to.
    gvk:
      kind: Project
      version: v1beta1
      group: resourcemanager.cnrm.cloud.google.com

    Additional examples can be found here: projectRef, folderRef, and organizationRef

    Note: The containers field has been deprecated, and hierarchicalReferences should be used instead.

  11. Follow the resource reference guidelines here to identify and configure potential resource reference fields.

Generate the CRD

  1. Run make manifests.
  2. Run git status. Verify that there is a new CRD in config/crds.

Note that you need to run make manifests every time when you update the service mapping file, and check the crd has been updated accordingly.

Run Unit Tests

make test

Ensure all tests pass.

Add the Service to Testing and Release

For testing to work, the service must be enabled for the test project. Add the service to the SUPPORTED_SERVICES block in scripts/shared-vars-public.sh if it is not there.

Test Your Changes

To verify your changes, add test resource YAMLs, run targeted tests for your resource.

Add necessary YAML files to testdata/

Add create.yaml and update.yaml to pkg/test/resourcefixture/testdata/basic in this folder structure. Make sure that the bottommost folder is the name of the resource in lowercase.

Tip: You can use the auto-generated testdata as a reference when adding test YAML files under pkg/test/resourcefixture/testdata/basic.

pkg/test/resourcefixture/testdata/basic
├── spanner
|   ├── v1alpha2
|       ├── spannerinstance
|           └── create.yaml
|           └── update.yaml

Create.yaml

Paste this into the create.yaml file (feel free to change labels/name/spec, but do not change the ${uniqueId} or ${projectId} replacement strings):

apiVersion: spanner.cnrm.cloud.google.com/v1alpha2
kind: SpannerInstance
metadata:
  labels:
    label-one: "value-one"
  name: spannerinstance-${uniqueId}
spec:
  config: regional-us-west1
  displayName: Spanner Instance Sample
  numNodes: 2

Please refer to the test data naming convention when changing name field. labels field is not needed if it wasn't included in the resource's servicemapping.

Update.yaml

The update.yaml file should look about exactly the same, except modify or add any fields/labels. Note that not all fields can be added/modified. Check the resource's Terraform implementation to see which fields support updates. For example, the Terraform implementation for SpannerInstance is at third_party/github.com/hashicorp/terraform-provider-google-beta/google-beta/services/spanner/resource_spanner_instance.go. We can see that the name field does not support updates since ForceNew is enabled.

In some cases, none of the fields of a resource support updates, in which case you can skip adding the update.yaml. You will also need to set SkipUpdate to true in the test context.

Dependencies.yaml

In some cases, the resource being added depends on other resources being created first. You can put the configurations for these dependencies in a dependencies.yaml file.

For example, the resource SQLDatabase depends on a SQLInstance resource already existing, so the dependencies.yaml for SQLDatabase contains the configuration for a SQLInstance resource.

Note: The dependent resources are created sequentially in the integration test, so the resources in dependencies.yaml MUST follow the topological ordering. E.g. if both a SQLInstance and a SQLUser should be in dependencies.yaml, and the SQLUser need to reference the SQLInstance, then SQLInstance must be defined before SQLUser.

Note: if one of the resource's dependencies is on ComputeNetwork and/or ComputeSubnetwork, see the Using the Default ComputeNetwork/ComputeSubnetwork section under the Appendix.

Run Tests

  1. Set [test-target] to the kind (in all lower-case letters) in the following command to run all the tests for your resource.

    go test -v -tags=integration ./pkg/controller/dynamic/ -test.run TestCreateNoChangeUpdateDelete -run-tests [test-target] -timeout 900s

    If you have multiple test cases and just want to run a single test case of your resource, you can set [test-target] to a specific test case folder name. For example:

    pkg/test/resourcefixture/testdata/basic
├── pubsub
|   ├── v1beta1
|       ├── pubsubsubscription
|           ├── basicpubsubsubscription
|               └── create.yaml
|               └── dependencies.yaml
|               └── update.yaml
|           ├── bigquerypubsubsubscription
|               └── create.yaml
|               └── dependencies.yaml
|               └── update.yaml

    go test -v -tags=integration ./pkg/controller/dynamic/ -test.run TestCreateNoChangeUpdateDelete -run-tests basicpubsubsubscription -timeout 900s

  2. If the dynamic test data contains constant placeholder strings, e.g. "TEST_FOLDER_ID", you need to set them as env variables first as follows:

       # Export the environment variables needed in the dynamic tests if you haven't done it.
   TEST_FOLDER_ID=123456789 go test -v -tags=integration ./pkg/controller/dynamic/ -test.run TestCreateNoChangeUpdateDelete -run-tests cloudschedulerjob -timeout 900s

Optionally Add the Resource to Test Contexts

If the tests passed then skip this section.

If the update or acquire test fail, it may mean that you need to disable the acquire and update test because the resource cannot support those functions. Note that disabling either of these tests should ONLY be done if the resource does not support the feature. Otherwise, you will need to debug the failures, understand the root cause, and make the appropriate fix(es).

To disable the acquire and update test:

  1. In pkg/test/resourcefixture/contexts/ open a file named ${service}_context.go for editing where ${service} is the lowercase name of the service for your resource. In the case of SpannerInstance it would be spanner and the file would be named spanner_context.go.

  2. Paste the following into the file, if the file already exists, do not paste in the func init() portion.

    func init() {
    resourceContextMap["spannerinstance"] = ResourceContext{
        ResourceKind:   "SpannerInstance",
        SkipUpdate:     true, // Update is not supported because ...
        SkipAcquire:    true, // Acquire is not supported because ...
    }
}

  3. Go line by line and convert the logic from SpannerInstance to your resource type. This may require making additional changes in other files.

  4. Run through the following list of things to implement and verify:

    • change the ResourceContextMap key to the lowercase resource name (no spaces, no underscores)
    • change ResourceKind to the same resource name as the key, but in UpperCamelCase
    • change SkipUpdate to true (and add a comment documenting the reason) or delete it
    • change SkipAcquire to true (and add a comment documenting the reason) or delete it

Optionally Add the Test Case to Long Running Test List

If the test takes more than 10 minutes to finish, add it to LONG_RUNNING_CRUD_TESTS_REGEX in ./scripts/shared-vars-public.sh.

Optionally enable GCP services API endpoints

Check the SUPPORTED_SERVICES in ./scripts/shared-vars-public.sh to make sure the GCP services for your resource is in the list, if not, you need to add it into the list.

Resource skeleton test cases

Skeleton test cases create a minimal resource definition for the provider to match to after ingesting the resource from an alternate source, such as a Cloud Asset or URI. To determine exactly what the expected skeleton should look like, you may need to run the pkg/resourceskeleton tests a few times to refine the resource skeleton you enter based on the diffs you get.

These tests depend on the TF provider import, which means they are sensitive to the idTemplate provided in the resources and the Terraform import itself. If you find an issue in the resource's Terraform import then patch terraform or create an issue for terraform to fix the importer before skipping tests.

Add a resource asset skeleton test case

Check if Cloud Asset Inventory(CAI) supports the resource kind by seeing if it exists in this list of supported resources.

Whether the resource is supported or not, you will need to add an entry in pkg/resourceskeleton/testdata/asset-skeleton.yaml,

If the resource is not supported by CAI, add an entry that looks like this (replacing StorageNotification with your resource type):

# not supported by asset inventory
- resourceConfigId: StorageNotification

If the resource is supported by CAI, you will need to add a skeleton which will look similar to this

- asset:
    ancestors:
    - projects/1234567890
    - organizations/1234567890
    name: //spanner.googleapis.com/projects/kcc-test/instances/spannerinstance-3m5j7pbhtprbatjs96cg/databases/spannerdatabase-test
    asset_type: spanner.googleapis.com/Database
  expectedSkeleton:
    apiVersion: spanner.cnrm.cloud.google.com/v1beta1
    kind: SpannerDatabase
    metadata:
      annotations:
        cnrm.cloud.google.com/project-id: kcc-test
      name: spannerdatabase-test
    spec:
      instanceRef:
        external: spannerinstance-3m5j7pbhtprbatjs96cg
  resourceConfigId: SpannerDatabase

add a uri -> resource skeleton test case

The uri -> resource skeleton test cases ensure that a resource supports config-connector export. Modify pkg/resourceskeleton/testdata/uri-skeleton.yaml. Add a new entry as follows.

- ExpectedSkeleton: null
  ResourceConfigId: SpannerDatabase
  URI: ""
  1. Change the ResourceConfigId to match the resource config of the resource you have added.
  2. Determine if the resource can be fetched by URL. You can get a URI by looking at the wire logs from gcloud describing your resource. Example: gcloud spanner databases describe asdf --instance spannerdatabase-dep --log-http
  3. If the resource can fetched by URL:
    1. Modify the test case by filling out the URI field with a sample URL.
    2. Modify the test case by filling out the ExpectedSkeleton field which the expected Config Connector resource that contains all the fields necessary to identify the resource (i.e. name, location, Kind)
  4. If the resource cannot be fetched by URL then, in the service mapping resource config for your new resource, set the value for idTemplateCanBeUsedToMatchResourceName to false

Add Sample(s)

To enable users to get up and running quickly, samples are created for each resource. If a given resource has different important use cases then multiple samples are created for that resource.

  1. In config/samples/resources, create a new directory for the resource's samples: mkdir spannerinstance.
  2. Follow the sample guidelines here and create sample(s) for the resource.

Run/Disable Sample Tests

In addition to the integration tests run using pkg/test/testdata/resourcefixture, KCC has tests for resource samples. These tests can be run manually as follows (example given is for one of CloudSchedulerJob's samples):

go test -v -tags=integration ./config/tests/samples/create -test.run TestAll -run-tests scheduler-job-pubsub

Replace the environment variables to real values before running the tests.

The sample tests will be run periodically in a newly created project through internal CI/CD pipeline. If this is not feasible for your resource, for example, your resource requires a custom project with special setup, you can skip the test by adding it to the testDisabledList map in config/tests/samples/create/samples_test.go.

Cloud Code Snippets

We have a script that generates snippet files for Cloud Code using our samples. If you created multiple samples (i.e. multiple samples subdirectories), you must tell the script which sample to use for generating snippets. Update the preferredSampleForResource map in pkg/snippet/snippetgeneration/snippetgeneration.go.

Update the Public Documentation Templates

In order to keep resource reference doc update to date, we need to add the new CRDs and samples to existing doc.

  1. Copy one of existing files under scripts/generate-google3-docs/resource-reference/templates, and name it like spanner_spannerinstance.tmpl, (i.e., _.tmpl).
  2. Some resources need additional messages in the doc, for example, in BigQueryDataset there's a warning message. If you want to add message for your resource, similar to bigquery_bigquerydataset.tmpl, add the message into the tmpl file.
  3. Change the values for "Service name" (there are two, one for 'Google Cloud Service Name' and one for 'Config Connector Service Name'), "Service Documentation", "REST Resource Name", and "REST Resource Documentation" accordingly. Make sure that the URL in "REST Resource Documentation" points to a working document by adding the partial URL to https://cloud.google.com (i.e. https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instances). Add an additional paragraph/table if needed.
  4. Update scripts/generate-google3-docs/resource-reference/overview.md by adding a row for your resource to the resource table. Note you need to use a proper template variable for your new service. TODO: Provide guidance on how to find the variable name.
  5. Update scripts/generate-google3-docs/resource-reference/_toc.yaml by adding an entry and path to the corresponding API section. If there is no entry for the API, add one.

To generate the new resource doc, run make resource-docs. You should see your resource generated to the scripts/generate-google3-docs/resource-reference/generated/resource-docs folder.

Create a pull request

Run make ready-pr to make sure the local changes are ready to commit.

For more details see Contributing Guide.

After the PR is created:

  1. Wait for the presubmit tests to pass. Look into the failures if there are any.

  2. Wait for your review to get approved. Address the comments if there are any.

  3. Merge the PR!

Appendix

Using the Default ComputeNetwork/ComputeSubnetwork

When writing testdata for a resource, if at least one of its dependencies is a resource of kind ComputeNetwork or ComputeSubnetwork, then use the default network and a default subnet if your tests do not need to create their own networks/subnets. This is done to prevent test failures caused by tests exceeding GCP's networks quota.

Below is the proper configuration for the default network. Use this in dependencies.yaml if you want to be able to depend on the default network.

apiVersion: compute.cnrm.cloud.google.com/v1alpha3
kind: ComputeNetwork
metadata:
  name: default
  annotations:
    cnrm.cloud.google.com/deletion-policy: "abandon"
spec:
  description: Default network for the project

Below is the proper configuration for one of the default subnets (the default subnet for the us-central1 region in this case). Use this in dependencies.yaml if you want to be able to depend on a default subnet.

apiVersion: compute.cnrm.cloud.google.com/v1alpha3
kind: ComputeSubnetwork
metadata:
  name: default
  annotations:
    cnrm.cloud.google.com/deletion-policy: "abandon"
spec:
  ipCidrRange: 10.128.0.0/20
  region: us-central1
  networkRef:
    name: default

Note that if you want to create a custom subnet for your dependency, you will have to create a custom network as well. Do NOT create a custom subnet within the default network, since this will potentially lead to the creation of multiple subnets in the default network with overlapping ipCidrRanges, which will lead to tests failing because this action is prohibited by GCP.

Test Multiple Scenarios for the Same Resource Kind

If you want to add multiple test scenarios for the same resource kind, e.g. a common case is to have test suites for both global and regional compute addresses using the same ComputeAddress CRD, you can create multiple folders to represent different test scenarios where each of them contains necessary YAML files like create.yaml and update.yaml.

pkg/test/resourcefixture/testdata/basic
├── compute
|   ├── v1beta1
|       ├── computeaddress
|         ├── globalcomputeaddress
|             └── create.yaml
|             └── update.yaml
|         ├── regionalcomputeaddress
|             └── create.yaml
|             └── update.yaml

In case you need to customize the ResourceContext to disable ex: Update test, in pkg/test/resourcefixture/contexts/compute_context.go, you need to add the corresponding ResourceContext into ResourceContextMap and the key should be the same as the folder name. For the above example, add the following code snippet:

    ResourceContextMap["regionalcomputeaddress"] = ResourceContext{
        ResourceKind:   "ComputeAddress",
        SkipUpdate:     true,
    }

    ResourceContextMap["globalcomputeaddress"] = ResourceContext{
        ResourceKind:   "ComputeAddress",
        SkipUpdate:     true,
    }

Using the auto-generated service mappings as a reference

When you create a new service mappings file, or add a resource config, you can use the auto-generated service mappings as a reference.

Creating a new service mappings file using the auto-generated service mappings

  1. Locate the service mappings file of name $(lowercaseservicename).yaml under scripts/resource-autogen/generated/servicemappings.
  2. Open the file and copy the lines until resources:
  3. Paste the copied the lines into the new service mappings file you've created.
  4. Review the values to make sure they are correct. Please refer back to the instructions under Create the Service Mappings file for more details.

Adding a new resource config using the auto-generated service mappings

  1. Locate the service mappings file of name $(lowercaseservicename).yaml under scripts/resource-autogen/generated/servicemappings.
  2. Open the file and copy the resource config (items under resources:) with the name of the Terraform resource you plan to add.
  3. Paste the copied the lines to the end of the service mappings file.
  4. Review the values to make sure they are correct and comprehensive. Please refer back to the instructions under Add the Resource Config to Service Mappings for more details.

Common issues running tests locally

If you run into any errors that look like "API not enabled"/"quota exceeded" despite having the API enabled and everything set up correctly, you may have to reconfigure your Application Default Credentials.

Certain APIs require authentication through the environment variable GOOGLE_APPLICATION_CREDENTIALS. If this is not set correctly, your tests might run into an error that looks a bit like this: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the cloudidentity.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.

If the GOOGLE_APPLICATION_CREDENTIALS needs to be set to a service account's key (this is the case for CloudIdentity and AccessContextManager), you will need to download/create key from your service account and assign the path of that key to the GOOGLE_APPLICATION_CREDENTIALS variable. Follow these steps, replacing SERVICE_ACCOUNT_EMAIL variable with your service account.

SERVICE_ACCOUNT_EMAIL={{insert your service account email here}}
TMP_GOOGLE_APPLICATION_CREDENTIALS=$(mktemp)
gcloud iam service-accounts keys create --iam-account "${SERVICE_ACCOUNT_EMAIL}" ${TMP_GOOGLE_APPLICATION_CREDENTIALS}
export GOOGLE_APPLICATION_CREDENTIALS=${TMP_GOOGLE_APPLICATION_CREDENTIALS

To revert this change, simply run gcloud auth login and gcloud auth application-default login to return to your original service-account/default login configuration.

Naming conventions, special variables and constants in testdata

WARNING: Do NOT use any plain text resource IDs for the test organizations, projects, folders, and billing accounts.

  • All the resource names should be [lowercase(Kind)]-${uniqueId}, e.g. cloudschedulerjob-${uniqueId}, pubsubtopic-${uniqueId}.

    • ${uniqueId} is a 20-char unique ID generated here. We always want to use the -${uniqueId} as the suffix for the resource name in testdata.

    • If there are multiple resources of the same kind used in the same test cases, the resource names should be [lowercase(Kind)]-[index]-${uniqueId}, e.g. pubsubtopic-1-${uniqueId}, pubsubtopic-2-${uniqueId}.

    • If the resource name has length limitation, use a shorter prefix.

      • IAMServiceAccount should have the resource name gsa-${uniqueId}.

    • Use spec.resourceID to name the GCP resource if the GCP API has a different naming convention.

  • ${projectId}: The default project to create test resources under. When it comes to local testing, ${projectId} is the project ID of the current running project configured by gcloud config set project.

  • ${projectNumber}: Similar with ${projectId}, except that it's the generated numeric ID of the default project. We shouldn't use this variable unless the resource doesn't accept ${projectId} as the input.

  • Default test organization: organizations/${TEST_ORG_ID}

  • Default test billing account: ${TEST_BILLING_ACCOUNT_ID}

Legacy configs for locational resources

Config Connector used to handle locational resources, e.g. compute addresses, forwarding rules, disks differently than other non-locational resources. This special handling/configuration of resources is now discouraged as we want the KRM resources reflect underlying TF rsources as much as possible, which will also ease the learnings from contributors when adding new resources.

For posterity, below is the previous special configuration for locational resources:

Terraform google provider has separate resource types to represent global, regional and zonal resources respectively. For example, google_compute_global_address represents global addresses, google_compute_address represents regional addresses, google_compute_disk represents zonal disks and google_compute_region_disk represents regional disks. However given that the underlying protocol buffer schema is actually the same, Config Connector would like to use a single CRD to represent multiple locational resources where a required location filed is used to specify the region/zone name or 'global' for global resources.

If the new resource type that you want to add is about locational resources. In the associated ResourceConfig, make the following changes:

  1. have a generic kind name, e.g. name the kind of compute addresses as ComputeAddress rather than ComputeGlobalAddress or ComputeRegionAddress
  2. fill in locationality field with the proper value: 'global', 'zonal' or 'regional', e.g. google_compute_address terraform resource should specify 'global' for locationality field