In NIFI how to convert from CSV to JSON without CSV header

Question

In NIFI how to convert from CSV to JSON without CSV header. For each CSV row searate json flow file sholud create and send to next processor. Below is the csv

861359032561480,1,15.480237,190506144035,a
861359032561480,3,16.480237,190506144035,b
861359032561480,2,17.480237,190506144035,c

The expected json is :

{"test":861359032561480,"test2":"1","test3":15.48,"test4":190506144035,"test5":A}

In the above JSON second json value 1 is string, in third json value the value is restricted to 2 digits after the decimal, in fourth json value is uppercased.

So, how can I convert and apply these transformations?

notNull · Accepted Answer · 2019-05-12 00:59:16Z

3

Use QueryRecord(to convert field to uppercase) processor and then Split the array of json records.

Configure Record Reader(CSVReader)/Writer(JsonRecordSetWriter).
In JsonSetWriter keep the matching avro schema like long,string,decimal.

In QueryRecord processor add the new property and keep the sql statement to convert "test5" to UPPER(test5) case, using ApacheCalcite sql parser.

Your sql statement would be something like below..

select test,test2,test3,test4,UPPER(test5)test5 from flowfile

Then use SplitRecord(prefered if json file is big) (or) SplitJson processors to split the array of json records to individual flowfiles.

Flow:

1.QueryRecord //to read csv and write in json format
2.SplitRecord (or) SplitJson //to Split array into individual flowfiles
3.other processors.

edited May 12, 2019 at 0:59

answered May 12, 2019 at 0:54

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ashok Over a year ago

How can I add csv header because the csv file is not having header

notNull Over a year ago

@vikram, We don't have to add csv header but while configuring CSVReader we need to configure Avro Schema with field names. then processor will read the csv file with the specified schema. in case if u want to add header then check out this link: community.hortonworks.com/questions/191116/…

ashok Over a year ago

If i am trying without header every field is coming as null

PragmaticProgrammer Over a year ago

@vikram You can use the InferAvroSchema processor and make it place the avro schema of the CSV into a flowfile attribute. Then in your CSV reader simply change the access strategy to schema text and use the inferred avro schema. In InferAvroSchema you can even hardcode the csv header line yourself

Collectives™ on Stack Overflow

In NIFI how to convert from CSV to JSON without CSV header

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related