Changing multiple keys on a JSON FILe using Python

Question

I am trying to update a new key into my JSON file if the conditions are met. The following is my python code attempting to make multiple updates in a JSON file.

 #!/usr/bin/env python
 # Usage: update json file 
import json
import os

json_dir="/opt/rdm/adggeth/ADGG-ETH-02/20181008/"
json_dir_processed="/opt/rdm/adggeth/ADGG-ETH-02/20181008updated/"
for json_file in os.listdir(json_dir):
    if json_file.endswith(".json"):
        processed_json = "%s%s" % (json_dir_processed, json_file)
        json_file = json_dir + json_file
        print "Processing %s -> %s" % (json_file, processed_json)
        with open(json_file, 'r') as f:
            json_data = json.load(f)
            # replacement mapping
            update_map = {"grp_farmerreg/farmerdetails/farmermobile":"grp_farmerdts/hh_id",
                          "grp_farmerdts/hh_region":"grp_farmerdts/region",
                          "grp_farmerdts/hh_district":"grp_farmerdts/district",
                          "grp_farmerdts/hh_ward":"grp_farmerdts/ward",
                          "grp_farmerdts/hh_village":"grp_farmerdts/village"}

            diff_keys = update_map.keys() - json_data.keys()
            if not diff_keys:
                print("No Update to JSON keys")
            else:
                for k in diff_keys:
                    json_data[k] = json_data[update_map[k]]
        with open(processed_json, 'w') as f:
            f.write(json.dumps(json_data, indent=4))
    else:
        print "%s not a JSON file" % json_file

The JSON file i am trying to make update to is as follows:

{
    ....
    "farmerregistrd": "1", 
    "grp_farmerdts/region": "5", 
    "datacollid": "0923678275", 
    "_status": "submitted_via_web", 
    "enumtype": "2", 
    "deviceid": "352948096845916", 
    "start_time": "2019-04-03T10:57:23.620+03", 
    "_uuid": "f1069eae-33f8-4850-a549-49fcde27f077", 
    "grp_farmerdts/village": "2852", 
    "_submitted_by": null, 
    "formhub/uuid": "42cb3fc351a74fd89702078160f849ca", 
    "grp_farmerdts/hh_id": "623", 
    "grp_farmerdts/ward": "136", 
    ...
    "_userform_id": "adggeth_ADGG-ETH-REG02-20181008", 
    "_id": 711097, 
    "grp_farmerdts/district": "31"
}

My expected output from running the following python file is as follows

 {
        ....
        "farmerregistrd": "1", 
        "grp_farmerdts/hh_region": "5", 
        "datacollid": "0923678275", 
        "_status": "submitted_via_web", 
        "enumtype": "2", 
        "deviceid": "352948096845916", 
        "start_time": "2019-04-03T10:57:23.620+03", 
        "_uuid": "f1069eae-33f8-4850-a549-49fcde27f077", 
        "grp_farmerdts/hh_village": "2852", 
        "_submitted_by": null, 
        "formhub/uuid": "42cb3fc351a74fd89702078160f849ca", 
        "grp_farmerdts/hh_id": "623", 
        "grp_farmerdts/hh_ward": "136", 
        ...
        "_userform_id": "adggeth_ADGG-ETH-REG02-20181008", 
        "_id": 711097, 
        "grp_farmerdts/hh_district": "31"
    }

Using my code i am unable to make my update my JSON file to fit the expected output as shown above. — Mirieri Mogaka
– Mirieri Mogaka, Commented Jul 23, 2019 at 8:42
Please take some time to read the help page, especially the sections named "What topics can I ask about here?" and "What types of questions should I avoid asking?". And more importantly, please read the Stack Overflow question checklist. You might also want to learn about Minimal, Complete, and Verifiable Examples. — sophros
– sophros, Commented Jul 23, 2019 at 8:46

Andrej Kesely · Accepted Answer · 2019-07-23 08:47:32Z

2

Using re module and json.loads() with object_hook= parameter (doc). This script will add hh_ prefix to every grp_farmerdts/* key where isn't:

json_str = '''{
    "farmerregistrd": "1",
    "grp_farmerdts/region": "5",
    "datacollid": "0923678275",
    "_status": "submitted_via_web",
    "enumtype": "2",
    "deviceid": "352948096845916",
    "start_time": "2019-04-03T10:57:23.620+03",
    "_uuid": "f1069eae-33f8-4850-a549-49fcde27f077",
    "grp_farmerdts/village": "2852",
    "_submitted_by": null,
    "formhub/uuid": "42cb3fc351a74fd89702078160f849ca",
    "grp_farmerdts/hh_id": "623",
    "grp_farmerdts/ward": "136",
    "_userform_id": "adggeth_ADGG-ETH-REG02-20181008",
    "_id": 711097,
    "grp_farmerdts/district": "31"
}'''

import re
import json

def change_keys(d):
    return {re.sub(r'grp_farmerdts/((?!hh_)(\w+))', r'grp_farmerdts/hh_\1', k): v for k, v in d.items()}

print(json.dumps(json.loads(json_str, object_hook=change_keys), indent=4))

Prints:

{
    "farmerregistrd": "1",
    "grp_farmerdts/hh_region": "5",
    "datacollid": "0923678275",
    "_status": "submitted_via_web",
    "enumtype": "2",
    "deviceid": "352948096845916",
    "start_time": "2019-04-03T10:57:23.620+03",
    "_uuid": "f1069eae-33f8-4850-a549-49fcde27f077",
    "grp_farmerdts/hh_village": "2852",
    "_submitted_by": null,
    "formhub/uuid": "42cb3fc351a74fd89702078160f849ca",
    "grp_farmerdts/hh_id": "623",
    "grp_farmerdts/hh_ward": "136",
    "_userform_id": "adggeth_ADGG-ETH-REG02-20181008",
    "_id": 711097,
    "grp_farmerdts/hh_district": "31"
}

edited Jul 23, 2019 at 8:47

answered Jul 23, 2019 at 8:42

Andrej Kesely

196k15 gold badges60 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mirieri Mogaka Over a year ago

How can apply this for several JSON files in a folder

Andrej Kesely Over a year ago

@MirieriMogaka Use json.load(<file object>, object_hook=change_keys) to parse json and change the keys. Them you can save the object with file.write and json.dumps, or simply json.dump(<file object)

RomanPerekhrest · Accepted Answer · 2019-07-23 09:37:26Z

1

According to your expected output all particular keys need to be checked (not one of them). Change your logic as shown below:

...
json_data = json.load(f)
# replacement mapping
update_map = {"grp_farmerreg/farmerdetails/farmermobile":"grp_farmerdts/hh_id",
              "grp_farmerdts/hh_region":"grp_farmerdts/region",
              "grp_farmerdts/hh_district":"grp_farmerdts/district",
              "grp_farmerdts/hh_ward":"grp_farmerdts/ward", "grp_farmerdts/hh_village":"grp_farmerdts/village"}

diff_keys = update_map.keys() - json_data.keys()
if not diff_keys:
    print("No Update to JSON keys")
else:
    for k in diff_keys:
        if update_map[k] in json_data:
            json_data[k] = json_data[update_map[k]]

edited Jul 23, 2019 at 9:37

answered Jul 23, 2019 at 8:52

RomanPerekhrest

93.1k4 gold badges75 silver badges112 bronze badges

11 Comments

Mirieri Mogaka Over a year ago

Traceback (most recent call last): File "/opt/rdm/adggeth/ADGG-ETH-02/addfidkey.sh", line 40, in <module> diff_keys = update_map.keys() - json_data.keys() TypeError: unsupported operand type(s) for -: 'list' and 'list' #this is the error i am getting after running your code @RomanPerekhrest

RomanPerekhrest Over a year ago

@MirieriMogaka. that means that you have used my code in wrong way. json_data and update_map are dictionaries. Use the proposed code attentively

RomanPerekhrest Over a year ago

@MirieriMogaka, what's your Python version?

Mirieri Mogaka Over a year ago

I have copied my changes

Mirieri Mogaka Over a year ago

python version 2.7.6

|

Collectives™ on Stack Overflow

Changing multiple keys on a JSON FILe using Python

2 Answers 2

2 Comments

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related