splitting JSON file using python

Question

I have below big json file

    {
        "sections": [
            {
                "facts": [
                    {
                        "name": "Server",
                        "value": "<https://xxxxxxx:18443/collector/pipeline/v1_allagents>"
                    },
                    {
                        "name": "Environment",
                        "value": "dev"
                    },
                    {
                        "name": "Issue",
                        "value": "Server is [EDITED]"
                    }
                ]
            },
            {
                "facts": [
                    {
                        "name": "Server",
                        "value": "<https://xxxxx:18443/collector/pipeline/customer-characterstics-v1>"
                    },
                    {
                        "name": "Environment",
                        "value": "dev"
                    },
                    {
                        "name": "Issue",
                        "value": "Server is [STOPPED]"
                    }
                ]
            }
{'facts': 
    [
    {'name': 'Server', 'value': u'<https://xxxxxx:18443/collector/pipeline/soap-post-v1_relations>'}, 
    {'name': 'Environment', 'value': u'dev'}, {'name': 'Issue', 'value': u' status is [STOPPED]'}
    ]
}, 
{'facts': 
    [
    {'name': 'Server', 'value': u'<https://xxxxxxx.134:18443/collector/pipeline/characterstics-v1_allagents>'}, 
    {'name': 'Environment', 'value': u'dev'}, {'name': 'Issue', 'value': u' status is [EDITED]'}
    ]
}, 
{'facts': 
    [
    {'name': 'Server', 'value': u'<https://xxxxxxx:18443/collector/pipeline/ab23-8128b7c9fcf2>'}, 
    {'name': 'Environment', 'value': u'dev'}, {'name': 'Issue', 'value': u'status is [EDITED]'}
    ]
}
     ]
    }
    ....

now I'm struggling to split above file as below and dump into another new files:

{
    "text": "Status",
    "themeColor": "#FF0000",
    {
        "sections": [
            {
                "facts": [
                    {
                        "name": "Server",
                        "value": "<https://xxxxxxx:18443/collector/pipeline/v1_allagents>"
                    },
                    {
                        "name": "Environment",
                        "value": "dev"
                    },
                    {
                        "name": "Issue",
                        "value": "Server is [EDITED]"
                    }
                ]
            }
     ]
    }
}

what I could able to achieve so far is print each tags under facts, but not the way I expect as above.

so, I'm having trouble adding those extra lines prior the final ones and then dump it to another file. How should I approach this? not using JQ. each splitted file should have same header and then exactly same pattern for key sections and facts .

edit:

As per Andrej's solution it works perfectly alright for one split at a time. But how to split the file based on n size, let's say I want to split my original big file where 5 facts exists 2 facts per file.n = 2 so, it should create 3 json files , where first 2 contains 2 blocks of facts and last one should be only one since that's left.

Then final output should be:

 {'text': ' Status', 'themeColor': '#FF0000', 'sections': 
    [
    {'facts': 
        [
        {'name': 'Server', 'value': u'<https://xxxxxx:18443/collector/pipeline/soap-post-v1>'},
        {'name': 'Environment', 'value': u'dev'}, 
        {'name': 'Issue', 'value': u' status is [STOPPED]'}
        ]
    }, 
    {'facts': 
        [
        {'name': 'Server', 'value': u'<https://xxxxx:18443/collector/pipeline/be9694085a70>'}, 
        {'name': 'Environment', 'value': u'dev'},
        {'name': 'Issue', 'value': u' status is [STOPPED]'}
        ]
    }
    ]
    }

and

  {'text': ' Status', 'themeColor': '#FF0000', 'sections': 
    [
    {'facts': 
        [
        {'name': 'Server', 'value': u'<https://xxxxxx:18443/collector/pipeline/soap-post-v1_relations>'}, 
        {'name': 'Environment', 'value': u'dev'}, {'name': 'Issue', 'value': u' status is [STOPPED]'}
        ]
    }, 
    {'facts': 
        [
        {'name': 'Server', 'value': u'<https://xxxxxxx.134:18443/collector/pipeline/characterstics-v1_allagents>'}, 
        {'name': 'Environment', 'value': u'dev'}, {'name': 'Issue', 'value': u' status is [EDITED]'}
        ]
    }
    ]}

as per above one block of fact from original file, hence it will create it's own json

 {'text': ' Status', 'themeColor': '#FF0000', 'sections': 
    [
    {'facts': 
        [
        {'name': 'Server', 'value': u'<https://xxxxxxx:18443/collector/pipeline/ab23-8128b7c9fcf2>'}, 
        {'name': 'Environment', 'value': u'dev'}, {'name': 'Issue', 'value': u'status is [EDITED]'}
        ]
    }
    ]}

Andrej Kesely · Accepted Answer · 2019-12-03 16:33:14Z

2

You can load the big file json into dictionary using json module. Then treat the loaded data as classical Python dict.

If your file contains the string in question, then this example:

import json

with open('YOUR_JSON_FILE.json', 'r') as f_in:
    data = json.load(f_in)

    for i, fact in enumerate(data['sections'], 1):

        with open('data_out_{}.json'.format(i), 'w') as f_out:
            d = {}
            d['text'] = 'Status'
            d['themeColor'] = '#FF0000'
            d['sections'] = fact
            json.dump(d, f_out, indent=4)

This creates two files data_out_1.json and data_out_2.json containing:

{
    "text": "Status",
    "themeColor": "#FF0000",
    "sections": {
        "facts": [
            {
                "name": "Server",
                "value": "<https://xxxxxxx:18443/collector/pipeline/v1_allagents>"
            },
            {
                "name": "Environment",
                "value": "dev"
            },
            {
                "name": "Issue",
                "value": "Server is [EDITED]"
            }
        ]
    }
}

and

{
    "text": "Status",
    "themeColor": "#FF0000",
    "sections": {
        "facts": [
            {
                "name": "Server",
                "value": "<https://xxxxx:18443/collector/pipeline/customer-characterstics-v1>"
            },
            {
                "name": "Environment",
                "value": "dev"
            },
            {
                "name": "Issue",
                "value": "Server is [STOPPED]"
            }
        ]
    }
}

EDIT:

To chunk the JSON file, you can use this example:

import json

def chunk(lst, n):
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

with open('YOUR_JSON_FILE.json', 'r') as f_in:
    data = json.load(f_in)


for i, fact in enumerate(chunk(data['sections'], 2), 1):  # <-- change 2 to your chunk size
    with open('data_out_{}.json'.format(i), 'w') as f_out:
        d = {}
        d['text'] = 'Status'
        d['themeColor'] = '#FF0000'
        d['sections'] = fact
        json.dump(d, f_out, indent=4)

edited Dec 3, 2019 at 16:33

answered Dec 3, 2019 at 14:47

Andrej Kesely

196k15 gold badges60 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

change198 Over a year ago

thanks @andrej, but if I want final json with n blocks facts inside, from the large original json. like below, will it be possible? like if I say split the original based on 2 sets, then it it should have final json as combination of your suggested 2 files data_out_1.json and data_out_2.json but with one text, themeColor and section?

Andrej Kesely Over a year ago

@ManasSamantaray Can you edit your question and put the input and desired output there?

tharndt · Accepted Answer · 2019-12-03 14:55:12Z

1

import json

with open('/tmp/json_response_output.json') as datafile:
    datastore = json.load(datafile)

for n, details in enumerate(datastore['sections']):
    split_json = datastore.copy()
    split_json['sections'] = [details]
    with open(f'json_response_output_part{n}.json', 'w') as f:
        json.dump(split_json, f, indent=4, ensure_ascii=False)

answered Dec 3, 2019 at 14:55

tharndt

1373 silver badges10 bronze badges

Collectives™ on Stack Overflow

splitting JSON file using python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related