DRY-ing mixed/multiple provider configs

[Gerrit-K]

I'm trying to set up a Terragrunt stack for Databricks on AWS. The resources are similar to this example but I planned on splitting this module into multiple ones. This means that some of these modules require the databricks and aws providers, whereas others simply need the aws or other basic providers (e.g. random).

To centralize the provider configuration, I'm following the pattern of having a root.hcl that generates the provider and is included in each unit:

root.hcl:

locals {
  region_vars      = read_terragrunt_config(find_in_parent_folders("region.hcl"))
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"

  contents = <<EOF
provider "aws" {
  region     = "${local.region_vars.locals.aws_region}"
  # ... shared config for AWS
}

provider "databricks" {
  alias "mws"
  # ... shared config for Databricks
}
EOF
}

some/unit/terragrunt.hcl:

include "root" {
  path   = find_in_parent_folders("root.hcl")
}

terraform {
  source = "..."
}

# ...

The issue I have with this approach is that this provider file is then unconditionally generated in all units, regardless of whether they actually need the provider or not. And if the Terraform module of a unit does not include the provider in the required_providers section in versions.tf, in the best case it would just download the latest provider (aws), but in the worst case (databricks) it would throw an error, because it defaults to hashicorp/<provider>, as it's missing the source = ... field.

I've come across a very similar question where the suggestion was to just use %{ if ... } around the provider code to conditionally template it. However, this didn't work for me, as I wanted the condition to be specified in the unit's terragrunt.hcl file (ideally somehow passing that down to the included root.hcl file). If I did a read_terragrunt_config("terragrunt.hcl") in the root.hcl I would get a cyclic dependency creating an infinite loop.

One other idea I currently have is to put the provider generation for each provider in a separate HCL file and then explicitly include that one, e.g.:
terragrunt.hcl:

include "root" {
  path   = find_in_parent_folders("root.hcl")
}

include "aws_provider" {
  path   = "${find_in_parent_folders("provider_configs")}/aws.hcl"
}

include "databricks_provider" {
  path   = "${find_in_parent_folders("provider_configs")}/databricks.hcl"
}

Does anyone else have a better idea or has come across the same or a similar problem?

[yhakbar]

Do you generally group together resources that need different providers hierarchically? Is the top-level aws directory responsible for AWS resources, and the top-level databricks directory responsible for Databricks resources?

If so, you could follow a pattern where you specify something like this in your root.hcl file:

# root.hcl

locals {
  providers_hcl_file = find_in_parent_folders("providers.hcl")
  contents           = local.providers_hcl_file.locals.contents
}

generate "providers" {
  path      = "providers.tf"
  if_exists = "overwrite"

  contents = local.contents
}

and a providers.hcl file in each directory like this:

# aws/providers.hcl

locals {
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  contents = <<EOF
provider "aws" {
  region = "${local.region_vars.locals.aws_region}"
  # ... shared config for AWS
}
EOF
}

# databricks/providers.hcl

locals {
  contents = <<EOF
provider "databricks" {
  alias "mws"
  # ... shared config for Databricks
}
EOF
}

If you aren't managing your resources in a hierarchy like that, you could also look to leverage Stacks, and have every unit define it's own generate block, but pass down the contents as a value.

EDIT: The example was wrong. Updated.

[Gerrit-K]

No, unfortunately it was not the plan to group units by provider. Moreover, there are certain patterns where it makes sense to have both providers used in a single module, e.g.:

data "databricks_aws_crossaccount_policy" "this" {}

resource "aws_iam_role_policy" "this" {
  # ...
  policy = data.databricks_aws_crossaccount_policy.this.json
}

The data block from the Databricks provider generates the json policy to be created in AWS. Of course I could split that into two units with a dependency from one to the other, but this feels overly complex.

This use-case, if used with a provider-based directory structure, would require me to have again some redundancy:

- terragrunt
  |- aws
  |  |- some_pure_aws_unit/terragrunt.hcl
  |- databricks
  |  |- some_pure_databricks_unit/terragrunt.hcl
  |- aws_databricks
     |- some_mixed_provider_unit/terragrunt.hcl

[yhakbar]

If you don't have a hierarchy for how your providers are organized, could you potentially do something like this?

# providers.hcl

locals {
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  aws_contents = <<EOF
provider "aws" {
  region = "${local.region_vars.locals.aws_region}"
  # ... shared config for AWS
}
EOF

  databricks_contents = ...
  databricks_and_aws_contents = ...
}

# some/unit/that/needs/aws-and-databricks/terragrunt.hcl

include "root" {
  path = find_in_parent_folders("root.hcl")
}

locals {
  contents = find_in_parent_folders("providers.hcl").locals.databricks_and_aws_contents
}

generate "providers" {
  path      = "providers.tf"
  if_exists = "overwrite"

  contents = local.contents
}

and switch out this line based on what you need in a given unit? You could also have a default provider generation in your root.hcl if you have one most common provider requirement.

contents = find_in_parent_folders("providers.hcl").locals.databricks_and_aws_contents

If not, then you might be better off doing multiple includes in each unit like you mentioned at the bottom of your initial description.

[Gerrit-K]

That's a good idea, but still not very DRY-ish. I would have to place the same generate "providers" {...} block into each unit instead of defining it once in the root.hcl. In that sense, it's somewhat similar to my very last suggestion, where I would just include the required providers for each unit manually.

If only Terragrunt would allow something like "parameterized includes" or some other way to access values of the including file (terragrunt.hcl) from within the included file (root.hcl) 🫤

Thanks for your quick replies and suggestions, by the way. Much appreciated!