Returning values at different layers of a HCL object, with many nested objects

Question

Before one asks why not change the output to be more friendly or uniform, I would if it came down to it but it will take a lot of effort to change across the board for our environment considering its dealing with terraform states

I'm returning state output from consul for various instances as defined by a variable with values set. The return is giving me this:

"instances" = [
  {
    "details" = {
      "controllers" = {
        "pri" = {
          "hostname" = "XXXXXXXXXdc001"
          "id" = "some_long_ocid_1"
          "ip" = "xxx.xxx.xxx.xxx"
        }
        "sec" = {
          "hostname" = "XXXXXXXXXdc002"
          "id" = "some_long_ocid_2"
          "ip" = "xxx.xxx.xxx.xxx"
        }
      }
      "domain" = "xxxxxxxxxxxxx.org"
      "smbdomain" = "XXXXX"
    }
  },
  {
    "xxxxxxxxxxxx000" = {
      "id" = "some_long_ocid_3"
      "ip" = "xxx.xxx.xxx.xxx"
    }
  },
  {
    "xxxxxxxxxxxx001" = {
      "id" = "some_long_ocid_4"
      "ip" = "xxx.xxx.xxx.xxx"
    }
  }
]

Each of the objects are individual terraform state outputs for instances. What I'm trying to do (without having to just grab everything from OCI and store all of that into my state file) is grab the attribute "id" from each state file, however whoever designed the domain controller state file output did not consider other sources using the state file for cohesion and so now I'm left to try and figure this out and be somewhat uniform across the board. Is there a way within terraform to pull this data in, and massage it in such a way to which I can store the output in a local variable like this:

"instances" = [
   "some_long_ocid_1",
   "some_long_ocid_2",
   "some_long_ocid_3",
   "some_long_ocid_4"
]

Any help on this would be greatly appreciated

Are you looking for assistance with the data transformation or a full end-to-end conversion? — Matthew Schuchard
– Matthew Schuchard, Commented Aug 25 at 16:29

Deprived.Writer · Accepted Answer · 2025-08-25 17:29:01Z

0

I figured out how to get the output that I needed. I'll post it here for others to see and comment on.

The way I did it was to also require jq as a provider, which then allowed me to run a jq_query data block. This is the full end to end conversion of the data sources:

locals {
  instances_json = jsonencode([ for value in data.terraform_remote_state.instances : value.outputs ])
}

data "jq_query" "all_ids" {
  data  = local.instances_json
  query = ".[] | .. | select(.id? != null) | .id"
}

locals {
  instances = split(",", replace(replace(data.jq_query.all_ids.result, "\n", "," ), "\"", "") )
}

The last locals block is needed because the jq_query block returns multiple values but the string is not in a standard json format. So we can't decode the string from json, we just simply have to work around it. So I replaced the "\n" characters with commas, and then replaced the \" with nothing so that the end result would give me something I could use the split function with to split up the values into a list.

answered Aug 25 at 17:29

Deprived.Writer

32 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Martin Atkins Sep 10 at 0:13

Is that using the massdriver-cloud/jq provider? (It might be helpful to include the relevant required_providers entry to make this example more complete in case anyone else wants to copy from it.)

Martin Atkins · Accepted Answer · 2025-08-25 17:34:58Z

The Terraform language itself does not include support for recursive analysis like this, where the data structure could be arbitrarily deep.

Therefore to solve this you will need to somehow rely on code written in another language. For example:

You could write a "function-only" Terraform provider which includes a function tailored directly to your goal, which takes a data structure like what you showed here and returns a set of id strings it discovered.
You could use the hashicorp/external provider to run an external program written in any language, passing it your input data as JSON and having it return a a JSON array of the ids.

(This means that you would need to preinstall the runtime for whichever language your external program is written in to the computer where Terraform is running. Also, the external data source only supports sending and recieving maps of strings, so annoyingly you would need to send your more complex data structures as strings containing nested JSON syntax.)
You could use the apparentlymart/javascript provider (disclaimer: I wrote it) to implement the transform you need in JavaScript executed through the javascript data source.

(This provider has a JavaScript interpreter embedded in it, so unlike the previous option it doesn't require preinstalling some other language runtime on to the system where Terraform is running.)

Whichever approach you take, the general idea is to implement the logic for finding the id values in a general-purpose programming language and then call that code from Terraform, rather than implementing the logic directly in the Terraform language.

Collectives™ on Stack Overflow

Returning values at different layers of a HCL object, with many nested objects

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related