update JSON file using terminal

Question

I want to write ubuntu terminal script/commands to update JSON file data. The current JSON file format is as per below.

{"_id":"dtpf7gfiay2dvt4","primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35219/optimized/annie-1.jpg","name_suggest":"Annie","name_suggest_lower":"annie","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"annie","model_number_suggest_lower":"annie","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"_id":"dt1f7gfiay2gvta","primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35220/optimized/arcade_a-1.jpg","name_suggest":"Arcade A","name_suggest_lower":"arcade a","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade a","model_number_suggest_lower":"arcade a","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"_id":"dt1f7yfiap2gvt4","primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35221/optimized/arcade_b-1.jpg","name_suggest":"Arcade B","name_suggest_lower":"arcade b","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade b","model_number_suggest_lower":"arcade b","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}

I want to remove _id field of JSON from the current line and want to add that value to above new line. Expected output:

{"index":{"_id":"dtpf7gfiay2dvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35219/optimized/annie-1.jpg","name_suggest":"Annie","name_suggest_lower":"annie","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"annie","model_number_suggest_lower":"annie","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"index":{"_id":"dtpf7gfiay2dvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35220/optimized/arcade_a-1.jpg","name_suggest":"Arcade A","name_suggest_lower":"arcade a","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade a","model_number_suggest_lower":"arcade a","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"index":{"_id":"dtpf7gfiay2dvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35221/optimized/arcade_b-1.jpg","name_suggest":"Arcade B","name_suggest_lower":"arcade b","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade b","model_number_suggest_lower":"arcade b","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}

I tried below commands

sed -e 's/$/\n{"index":{}}/' -i products_suggest_v4_singular_plural_v2-7.json #which is adding static new line. I want to make it dynamic
sed '/"_id":/s/"_id":[^,]*,//' > products_suggest_v4_singular_plural_v2.json #removing value of _id field

sed 's/"_id":"[^"]\+",//'

Serge
– Serge

2020-02-10 13:42:36 +00:00
Commented Feb 10, 2020 at 13:42 — Serge
– Serge, Commented Feb 10, 2020 at 13:42

Shawn · Accepted Answer · 2020-02-10 14:19:07Z

You want an actual JSON aware tool, like jq, the gold standard for command-line JSON manipulation:

 $ jq '{index: {_id: ._id }}, del(._id)' input.json
{"index":{"_id":"dtpf7gfiay2dvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35219/optimized/annie-1.jpg","name_suggest":"Annie","name_suggest_lower":"annie","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"annie","model_number_suggest_lower":"annie","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"index":{"_id":"dt1f7gfiay2gvta"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35220/optimized/arcade_a-1.jpg","name_suggest":"Arcade A","name_suggest_lower":"arcade a","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade a","model_number_suggest_lower":"arcade a","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"index":{"_id":"dt1f7yfiap2gvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35221/optimized/arcade_b-1.jpg","name_suggest":"Arcade B","name_suggest_lower":"arcade b","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade b","model_number_suggest_lower":"arcade b","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}

glenn jackman · Accepted Answer · 2020-02-10 14:14:49Z

I'd use a language with an actual JSON parser. Ruby is handy for this:

ruby -rjson -e '
  File.new(ARGV.shift).each do |line|
    data = JSON.parse(line)
    puts JSON.generate({index: {_id: data["_id"]}})
    data.delete("_id")
    puts JSON.generate(data)
  end
' file.json

{"index":{"_id":"dtpf7gfiay2dvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35219/optimized/annie-1.jpg","name_suggest":"Annie","name_suggest_lower":"annie","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"annie","model_number_suggest_lower":"annie","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"index":{"_id":"dt1f7gfiay2gvta"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35220/optimized/arcade_a-1.jpg","name_suggest":"Arcade A","name_suggest_lower":"arcade a","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade a","model_number_suggest_lower":"arcade a","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}
{"index":{"_id":"dt1f7yfiap2gvt4"}}
{"primary_image_url":"https://dt4f7ywfiapgvt.cloudfront.net/products_images/35221/optimized/arcade_b-1.jpg","name_suggest":"Arcade B","name_suggest_lower":"arcade b","brand_name_suggest":"Texturae","brand_name_suggest_lower":"texturae","model_number_suggest":"arcade b","model_number_suggest_lower":"arcade b","keywords_suggest":["wallpapers","wallpaper","wall-paper","wall-papers"],"keywords_suggest_lower":["wallpapers","wallpaper","wall-paper","wall-papers"],"category_suggest":["Wallpapers","Rugs & Decor"],"category_suggest_lower":["wallpapers","rugs & decor"]}

The hash.delete method does return the value at the index to be deleted, so you could write this, but at the cost of readability IMO

    puts JSON.generate({index: {_id: data.delete("_id")}})

Collectives™ on Stack Overflow

update JSON file using terminal

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related