Scripting in AWS elasticsearch

Question

I read online about scripting in aws elacticsearch service. It said that AWS ES doesn't support dynamic scripting so I am writing aggregations using scripts stored in my disk. I wrote the following query

{
    "query":{
        "match_all":{}
    },
    "aggs":{
        "inBoundRecieved":{
            "scripted_metric":{
                "init_script":{
                    "file": "init.groovy"
                },
                "map_script": {
                    "file": "map.groovy"
                },
                "combine_script": {
                    "file":"comb.groovy"
                },
                "params":{
                    "field":"call_direction"
                },
                "reduce_script": {
                    "file": "red.groovy"
                }
            }
        }
    }

}

But I keep getting this error.

Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]: [init_script]

I have searched a lot online but couldn't find a good solution. Full Error ->

{ 
 "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][0]:                 RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[plivoredshift][0]: query[ConstantScore(*:*)],  from[-1],size[-1]: Parse Failure [Failed to parse source [{    \"query\":{        \"match_all\":{}    },    \"aggs\":{        \"inBoundRecieved\":{                       \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                \"combine_script\": {                    \"file\":\"comb.groovy\"                },                \"params\":  {                    \"field\":\"call_direction\"                },                \"reduce_script\": {                    \"file\": \"red.groovy\"                       }            }        }    }}]]]; nested: SearchParseException[[plivoredshift][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a          START_OBJECT in [inBoundRecieved]: [init_script].]]; }{[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][1]: RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/ search[phase/query]]]; nested: SearchParseException[[plivoredshift][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{             \"query\":{        \"match_all\":{}    },    \"aggs\":{        \"inBoundRecieved\":{            \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                                \"combine_script\": {                    \"file\":\"comb.groovy\"                },                \"params\":{                    \"field\":                             \"call_direction\"                },                \"reduce_script\": {                    \"file\": \"red.groovy\"                }            }        }    }}]]];     nested: SearchParseException[[plivoredshift][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]:         [init_script].]]; }{[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][2]: RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested:      SearchParseException[[plivoredshift][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{    \"query\":{        \"match_all\":{}     },    \"aggs\":{        \"inBoundRecieved\":{            \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.                      groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                \"combine_script\":              {                    \"file\":\"comb.groovy\"                },                \"params\":{                    \"field\":\"call_direction\"                               },                \"reduce_script\": {                    \"file\": \"red.groovy\"                }            }        }    }}]]]; nested:                               SearchParseException[[plivoredshift][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]: [init_script].  ]]; }{[M-Sp4ZKmQCW0C4Ph2FIA1Q][plivoredshift][3]: RemoteTransportException[[Merlin][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested:                    SearchParseException[[plivoredshift][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{    \"query\":{        \"match_all\":{}     },    \"aggs\":{        \"inBoundRecieved\":{            \"scripted_metric\":{                \"init_script\":{                    \"file\": \"init.                      groovy\"                },                \"map_script\": {                    \"file\": \"map.groovy\"                },                \"combine_script\":              {                    \"file\":\"comb.groovy\"                },                \"params\":{                    \"field\":\"call_direction\"                               },                \"reduce_script\": {                    \"file\": \"red.groovy\"                }            }        }    }}]]]; nested:                               SearchParseException[[plivoredshift][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Unknown key for a START_OBJECT in [inBoundRecieved]: [init_script].
"status":400
}

Here are my scripts init.groovy

_agg['transactions'] = []

map.groovy

if (doc['call_direction']=="inbound" {_agg.transactions.add(1)} else {_agg.transactions.add(0)}

comb.groovy

inBoundRecieved=0; for( t in _agg.transactions) {inBoundRecieved+=t}; return inBoundRecieved

red.groovy

inBoundRecieved=0; for( a in _aggs) {inBoundRecieved += a}; return inBoundRecieved

I have been following this tutorial from the ElacticSearch website

Dynamic scripting is not available in the Amazon managed ES service, whether inline or via script files. see this — Val
– Val, Commented May 30, 2016 at 14:15
so you mean to say there is no way of adding scripts to aws search queries? — sidd607
– sidd607, Commented May 30, 2016 at 14:18
That's correct. As far as my experience goes, the AWS managed service is good enough for storing logs or for some very simple ES usage, but if you need some advanced ES features, you're better off rolling our own. — Val
– Val, Commented May 30, 2016 at 14:19
In that case can you help me. I have a field "CallDirection" that contains either INBOUND or OUTBOUND. I need to return the number of records that have callDirection=INBOUND and number of records where callDirection=OUTBOUND, can you help me create a query for this result? — sidd607
– sidd607, Commented May 30, 2016 at 14:23
Why not simply using a terms aggregation on the callDirection field? — Val
– Val, Commented May 30, 2016 at 14:24

user1383029 · Accepted Answer · 2017-03-24 15:36:56Z

4

Looks like scripting can be used in AWS elasticsearch service for version 5 now:

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-resources.html

https://forums.aws.amazon.com/thread.jspa?threadID=217896&start=25&tstart=0

answered Mar 24, 2017 at 15:36

user1383029

2,1552 gold badges24 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scripting in AWS elasticsearch

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related