elasticsearch watcher chain inputs with intermediate array

Question

Question

I am chaining different elasticsearch queries in a watcher. I short I'd like to do this:

find container_ids
run a query on all entries with any of those container_ids

the first query gives me an array. So I would like to pass that array as an input to the terms query. Trouble is that the ctx.payload... variables seem to expand only within strings.

"query": {
  "terms" : {
    "container_id" : ctx.payload.first_query._value,
  }
}

will give me errors like:

Unrecognized token 'ctx'

Would there be a way to insert an array without expanding to a string? If I use "ctx.payload.first_query._value" he will query for the string "[id1, id2]"...

Current workaround

I have currently implemented a workaround passing though a regex, but this seems a bit clunky:

"input": {
  "chain": {
    "inputs" : [ {
      "container_ids" : {
        ...
      }
    }, {
      "container_id_regex": {
        "transform" : {
          "script": """
            def container_id_rexexp = "";
            def regexp_separator = "";
            for(def hit : ctx.payload.container_ids.hits.hits){
              container_id_rexexp += regexp_separator + hit._source.container_id;
              regexp_separator = "|";
            }
            return container_id_rexexp;
          """
        }
      }
    }, {
      "container_details": {
        "search": {
          "request": {
            ...
            "body": {
              "query": {
                "regexp": {
                  "container_id": {
                    "value": "{{ '{{ctx.payload.container_id_regex._value}}' }}"
                  }
                }
              }
            }
          }
        }
      }
    } ]
  }
}

@Ahmed if you think this is a good question / workaround, plase upvote. — Chris Maes
– Chris Maes, Commented Sep 7, 2021 at 12:55

fbaligand · Accepted Answer · 2023-10-17 09:58:25Z

0

In your second chained input, you can use "template" option and "#toJson" special mustache tag. With "#toJson" tag, you can inject a JSON structure as is, like an array.

Here's an example:

  "input": {
    "chain" : {
      "inputs" : [ 
        {
          "first" : {
            "simple" : {
              "my_array" : ["VALUE1", "VALUE2"]
            }
          }
        },
        {
          "second" : {
            "search": {
              "request": {
                "template": {
                  "source": """{
                    "size": 0,
                    "query": {
                      "terms": {
                        "my_field": {{#toJson}}ctx.payload.first.my_array{{/toJson}}
                      }
                    }
                  }"""
                },
                "indices": [
                  "my_index"
                ]
              }
            }
          }
        }
      ]
    }
  }

answered Oct 17, 2023 at 9:58

fbaligand

212 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Chris Maes Over a year ago

that seems to work only when posting (creating) a new query. But my data (container_ids) are only available during executing, and then this {{#toJson}} is not accepted.

Chris Maes Over a year ago

I am trying to implement your proposal in dev-console (in a watcher), but it doesn't work.

fbaligand Over a year ago

Well, my first input is static, only to show a simple example, but it works also with a dynamic query as first input. {{#toJson}} works fine with Elasticsearch 7.17 for me. This is required to interpret a string as a real json structure (array or whatever)

fbaligand Over a year ago

As you say it doesn't work for you, could you precise what's your error, and what is your Elasticsearch version?

Chris Maes Over a year ago

I try to use what you propose in the "Dev tools/console" in the kibana web interface using the POST _watcher/watch/_execute method. This console doesn't even run the query because it says there is a Syntax Error at the "toJson" part.

|

Collectives™ on Stack Overflow

elasticsearch watcher chain inputs with intermediate array

Question

Current workaround

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Question

Current workaround

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related