How to concatenate keys with nested levels in a json file in Python?

Question

I am working on a problem where i have a file of json objects given below:

{
    "id": "111",
    "name": {
        "firstname": "Tamara",
        "lastname": "Myers"
    },
    "address": {
        "street": "20722 Coleman Villages,East Rose",
        "zip": "71064-5894"
    }
}

I want to convert it into:

{
    "id": "111",
    "name_firstname": "Tamara",
    "name_lastname": "Myers",
    "address_street": "20722 Coleman Villages,East Rose",
    "address_zip": "71064-5894"
}

I am not able to do that because of the reason that we may have more fields in other json objects, which are not given in above example.

For example "Job":{"Engineer":"Junior","domain":"civil"}. And level of nesting is also irregular in all json objects.

Is there a particular reason you need to flatten the JSON? Otherwise, sounds like an XY Problem — OneCricketeer
– OneCricketeer, Commented Mar 3, 2017 at 22:04
Yes, After flattening it I want to change it to a pandas dataframe with keys as columns. — Prashant Sharma
– Prashant Sharma, Commented Mar 3, 2017 at 22:06

Mathieu Paturel · Accepted Answer · 2017-03-03 22:34:11Z

2

You need a recursive function: it doesn't care about the nesting level, unlike a while or for loop.

(a recursive function is just a function that calls itself)

The idea is to

create a new object (the target object)
in the recursive function
- loop over each key of the original object
  - if the value is a dict, call this function again
  - otherwise, add the formatted key (with the underscores) and value to the target object

def flat_keys(obj, new_obj={}, keys=[]):
    for key, value in obj.items():
        if isinstance(value, dict):
            # call the function again if the value is a dict
            # we go one step deeper: obj[key]
            # give the new_obj (by reference, so each call edit the *same* object)
            # give to used keys: keys + [key]
            flat_keys(obj[key], new_obj, keys + [key])
        else:
            new_obj['_'.join(keys + [key])] = value
    return new_obj

new_obj = flat_keys(json.JSONDecoder().decode("your object"))
print(new_obj)

edited Mar 3, 2017 at 22:34

answered Mar 3, 2017 at 22:11

Mathieu Paturel

4,53029 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Prashant Sharma Over a year ago

Hello @math2001 , Thanks a lot for replying.Your code helped me in solving this problem.

Mathieu Paturel Over a year ago

You're welcome! Is there something that you're confused about? If not, can you accept this answer then? 😋

2rs2ts · Accepted Answer · 2017-03-03 22:18:03Z

2

You need a recursive function. But this one is simpler than the other provided. It is also puts the base case first, which will help a little with stack size. I couldn't make it tail recursive though.

def merge_keys(d):
  to_return = {}
  for key, value in d.items():
    if not isinstance(value, dict):
      to_return[key] = value
    else:
      for merged_key, merged_value in merge_keys(value).items():
        to_return["_".join((key, merged_key))] = merged_value
  return to_return

answered Mar 3, 2017 at 22:18

2rs2ts

11.1k12 gold badges56 silver badges100 bronze badges

1 Comment

Prashant Sharma Over a year ago

Hello @2rs2ts, your code solved my problem. Thanks a lot for your help.

oshaiken · Accepted Answer · 2017-03-03 23:07:04Z

import sys
sys.path.insert(0, '.')
from sys import stderr


def pare(data, key, is_verbose=False):

   parts = key.split('.')

   i = 0
   ptr = data

   for part in parts:
       if is_verbose is True:
          if i > 0:
              stderr.write(' -> ')

          stderr.write(part)

       try:
          if issubclass(ptr.__class__, list) is True:
              ptr = ptr[int(part)]
          else:
              ptr = ptr[part]
       except:
          if is_verbose is True:
              stderr.write("\n")
          raise ValueError("Could not descend to child node: %s" % (part))
       i += 1
    if is_verbose is True:
       stderr.write("\n")
    return ptr
def path_list(dictionary, path):

    key_path_list = []
    if dictionary.__class__.__name__ == 'dict':
        if len(dictionary.keys())>0:
          i = 0
          n = len(dictionary.keys())
          while i< n:
              new_path  = dictionary.keys()[i]
              i += 1
              key_path = path + '.' + new_path
              key_path_list.append(key_path)
    else:
        pass
return key_path_list

def rec_data(data, key_path):
    pared = pare(data, key_path)
    value = []
    nd = {}
    if pared.__class__.__name__ == 'dict':
        paths = path_list(pared, key_path)
        for p in paths:
           if p in paths:
                   sl = pare(data, p)
                   nd[p] =sl
                   value.append(sl)
           else:
              pass
           rec_data(data, p)
    else:
        nd[key_path] = pared
    return nd
def main():
    json = {
    "id": "111",
    "name": {
       "firstname": "Tamara",
       "lastname": "Myers"
    },
    "address": {
       "street": "20722 Coleman Villages,East Rose",
       "zip": "71064-5894"
     }
     }

    dic = {}
    for k,v in json.items():
        dic.update(rec_data(json,k))
    print dic
if __name__ == "__main__":

    main()

sharez · Accepted Answer · 2017-05-29 07:44:53Z

1

I believe a recursive generator would be nice:

def nested_to_flat(data):                     
     for k, v in data.items():
         if isinstance(v, dict):
             for x, y in nested_to_flat(v):
                 yield ('%s_%s' % (k, x), y)
             continue
         yield (k, v)

and use it as below:

result = {k: v for k, v in nested_to_flat(data)}

edited May 29, 2017 at 7:44

answered Mar 3, 2017 at 22:38

sharez

1,54716 silver badges16 bronze badges

Comments

Ismael Martinez · Accepted Answer · 2017-03-03 22:09:41Z

0

In Python you can create the keys at fly

dic = {}
dic["key"] = value
dic["key"] = "value"
print dic
{'key': 'value'}

And you can do something like this with your object:

myNewDic = {}
for key, value in dic.items():
    if isinstance(value,dict):
        for k, v in value.items():
            myNewDic[key+"_"+k] = v
    else:
       myNewDic[key] = value

Check if works, i think im on the way.

Happy codding

answered Mar 3, 2017 at 22:09

Ismael Martinez

235 bronze badges

3 Comments

Mathieu Paturel Over a year ago

This won't work if there is more than more than 2 nesting level

Prashant Sharma Over a year ago

Hello Ismael, Thanks for jumping in. I think you have assumed that there is only two level of nesting. In some objects , I have keys with 7 to 8 level of nesting. In that case , I am not sure it would work.

Ismael Martinez Over a year ago

Hello, Yes, i assume that only are two levels of nesting, and functions posted by @math2001 or 2rs2ts solves your problem

Collectives™ on Stack Overflow

How to concatenate keys with nested levels in a json file in Python?

5 Answers 5

2 Comments

1 Comment

Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

1 Comment

Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related