Writing JSON File Containing Array Continuously

Question

I want to create a JSON file with Python. JSON structure looks like :

{
    "index_pattern" : "all_packets"
    "packets" : [
        {
            "packet_type" : "TCP"
            "source_ip" : "192.168.0.2",
            "destination_ip" : "192.168.0.114"
        },
        {
            "packet_type" : "ICMP"
            "source_ip" : "192.168.0.2",
            "destination_ip" : "192.168.0.114"
        }
    ]
}

Whenever I catch a packet with wireshark module, I want to add that packet to say myoutput.json file. The problem is, using something like

f = open("myoutput.json", "a")
f.write({
            "packet_type" : "ICMP"
            "source_ip" : "192.168.0.2",
            "destination_ip" : "192.168.0.114"
        })

is not suitable, because I do not have a graceful shutdown and I can not add necessary closing paranthesis to finish up the JSON. So I need to write in BETWEEN the packet array. What is the best way to achieve this.

Could you store this in a NoSQL db instead, and just send the packets to that? — doctorlove
– doctorlove, Commented Mar 22, 2019 at 12:04
Hi, thank you for attention. My project requirement is creating a JSON file as an output that another program (say X) can read. And program X already uses ElasticSearch so we may not involve another storing tool into project. That's why I we can't guess.@doctorlove — I.K.
– I.K., Commented Mar 22, 2019 at 12:14

Vlad · Accepted Answer · 2019-03-22 10:49:58Z

1

When you writing the file:

import json
with open('myoutput.json', 'a') as fo:
    json.dump({
        "packet_type" : "ICMP",
        "source_ip" : "192.168.0.2",
        "destination_ip" : "192.168.0.114"
          }, fo)

When reading the file, do:

with open('myoutput.json') as fo:
    data = fo.read()
    data = data.replace('}{', '},{')
    data = '[' + data + ']'
    data = json.loads(data)
    result = {"index_pattern" : "all_packets",
              "packets": data}

Edit

You can also create a helper class to do it:

import json

class WriteObject:
    def __init__(self, fname='myoutput.json'):
        self.fname = fname

    def push(self, data):
        #    data_example={
        #    "packet_type" : "ICMP",
        #    "source_ip" : "192.168.0.2",
        #    "destination_ip" : "192.168.0.114"
        #    }
        with open(self.fname, 'a') as fo:
            json.dump(data, fo)
    def get_all(self):
        with open(self.fname) as fo:
            data = fo.read()
            data = data.replace('}{', '},{')
            data = '[' + data + ']'
            data = json.loads(data)
            result = {"index_pattern" : "all_packets",
                      "packets": data}
        return result

# usage
obj = WriteObject()

obj.push({
        "packet_type" : "ICMP",
        "source_ip" : "192.168.0.2",
        "destination_ip" : "192.168.0.114"
          })

edited Mar 22, 2019 at 10:49

answered Mar 22, 2019 at 10:31

Vlad

8,6255 gold badges38 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

taras Over a year ago

But did you actually check if your code is working?

Vlad Over a year ago

@taras, Yes, in Jupyter!

I.K. Over a year ago

Hi, thank you for the answer. To be more clear, I want make a push operation in packets field that lies in a file. Just to demonstrate its like, myoutput.json["packets"].push({ "packet_type" : "ICMP", "source_ip" : "192.168.0.2", "destination_ip" : "192.168.0.114" })

I.K. Over a year ago

Solution with the class implementation solved it. Thanks to all of you <3

taras · Accepted Answer · 2019-03-22 10:43:09Z

0

One possible solution is json.loading the entire file into python object, say data and appending the packet to data["packets"]. Then you need to write the updated data back to your json file.
The drawback of this approach is the need to constantly read and write data to json file.

It can be improved by aggregating a list of packets (so you temporary save your packets to packets list) and updating the json file only when it reaches some predefined size (say, when len(packets) == 20). After that you can empty packets and start gathering data into it again.

answered Mar 22, 2019 at 10:43

taras

6,93510 gold badges45 silver badges54 bronze badges

Collectives™ on Stack Overflow

Writing JSON File Containing Array Continuously

2 Answers 2

Edit

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Edit

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related