How to parse a fluent-bit json $log.<field> key?

Question

I have the following log to be parsed:

TID: [-1234] [] [2021-05-31 09:53:26,680] - Unique ID: Evento_Teste, Event: {"event":{"metaData":"blue"}}

And below the configuration files that I created to deal with it.

#td-agent-bit.conf
[SERVICE]
    log_level info
    parsers_file parsers.conf

[INPUT]
    Name tail
    Path /opt/wso2am/repository/logs/xtest-td.log
    Tag wso2.trace

[FILTER]
    name   parser
    match  wso2.*
    Key_Name log
    Parser wso2_trace

[OUTPUT]
    Name  stdout
    match *
    Format json_lines

# parsers.conf
[PARSER]
    Name   wso2_trace
    Format regex
    Regex  ^TID: \[(?<TID>[-\d]+)\] \[\] \[(?<time>[^\]]*)\].* Unique ID: (?<unique_id>[_\w\d]+), Event: (?<event>.*)$
    Time_Key time
    Time_Format %Y-%m-%d %H:%M:%S

And I'm getting the following in the output:

[2021/06/01 16:09:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=34902473 watch_fd=1 name=/opt/wso2am/repository/logs/xtest-td.log
{"date":1622454806.0,"TID":"-1234","unique_id":"Evento_Teste","event":"{\"event\":{\"metaData\":\"blue\"}}"}

This log will be sent to a ES instance, and I need to transform the value of the event key in an json object instead of a string as is.

I've tried to create some parsers using decoders but I couldn't find a way to transform the event in a json object.

How can I do it ?

Kleyson Rios · Accepted Answer · 2021-06-02 12:52:39Z

4

I figured out how to parse.

I've created a new parse:

[PARSER]
    Name         json_field
    Format       json
    # Command       | Decoder      | Field    | Optional Action   |
    # ==============|==============|==========|===================|
    Decode_Field_As   escaped_utf8   log        do_next
    Decode_Field_As   json           log

And I included it in the pipeline:

[SERVICE]
    log_level info
    parsers_file parsers.conf

[INPUT]
    Name tail
    Path /opt/wso2am/repository/logs/xtest-td.log
    Tag wso2.trace

[FILTER]
    name   parser
    match  wso2.*
    Key_Name log
    Parser wso2_trace

[FILTER]
    name   parser
    match  wso2.*
    Key_Name event
    Parser json_field
    Reserve_Data True

[OUTPUT]
    Name  stdout
    match *
    Format json_lines

answered Jun 2, 2021 at 12:52

Kleyson Rios

2,9378 gold badges45 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to parse a fluent-bit json $log.<field> key?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related