logstash - map to json array with transformation

Question

I have the following json file, each line is a diferent json:

{"s":"some address","c":"some city"}
{"s":"some address1","c":"some city1"}
{"s":"some address2","c":"some city2"}

I have the following job:

input {
    file {
        start_position => "beginning"
        path => "/sources/someFile.txt"
    }
}

filter {
    json {
        source => "a"
        target => "addresses[0].street"
    }
    mutate {
        remove_field => ["message", "@timestamp", "host", "path", "@version"]
    }
}

output {
    elasticsearch {
        hosts => "http://elasticsearch:9200"
        index => "store"
    }
}

I want to write to to the index as the following (each address go to a different doc as the forst element in an array):

{
   "addresses": [{"street" : "some address", "city" : "some city"}]
}

{
   "addresses": [{"street" : "some address2", "city" : "some city1"}]
}

{
   "addresses": [{"street" : "some address3", "city" : "some city2"}]
}

The attached job is not working. no error and not doing anything.
Thanks

Badger · Accepted Answer · 2020-11-28 14:35:07Z

1

You cannot use that field reference in the target option of the json filter. In any version of logstash from the last couple of years I would expect that to result in a _jsonparsefailure tag and the error

Exception caught in json filter {:exception=>"Invalid FieldReference: `addresses[0].street`"

If you change the reference to be [addresses][0] then it will run without error, but the reference will be interpreted as the "0" entry in the "addresses" hash, not the first entry in the addresses array.

Your incoming JSON has the wrong field names, so you will have to rename the fields. I think it is easiest to do it in a ruby filter

json { source => "message" target => "[@metadata][json]" }
ruby {
    code => '
        json = event.get("[@metadata][json]")
        event.set("addresses", [ { "street" => json["s"], "city" => json["c"] } ] )
    '
}

which produces

 "addresses" => [
    [0] {
          "city" => "some city",
        "street" => "some address"
    }
],

The original JSON is placed inside the [@metadata] field so that is available but not indexed by the output.

answered Nov 28, 2020 at 14:35

Badger

4,1272 gold badges9 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SexyMF Over a year ago

Thanks. can you try and put it togehter with my original script. I can't make it work. thanks

Badger Over a year ago

The json and ruby filter I provided should just replace the json filter you have. A file input will put the text of the line of the file in the field [message], not [a]. If you had previously run logstash it will have recorded how much of the input file it read in the sincedb, so you will need to append new lines to the file to get logstash to read them.

Collectives™ on Stack Overflow

logstash - map to json array with transformation

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related