Detect duplicate elements python

Question

I get continuously data from a server and can receive the data via the following line of code:

id, type, value = getId(payload)

After that I would like to write them into a file via:

out.write(str(id) + ";" + str(type) + ";" + str(value) + "\n")

Now the case is, that the same id can appear multiple times, but the value will be a different one. Therefore I would like to extend the out.write into the following way that the different values are added at the right side but still being referred to the same id:

out.write(str(id) + ";" + str(type) + ";" + str(value) + ";" + str(value1) + ";" + str(value2) + "\n")

Does anyone has an idea how to do this in python?

make a dict of dicts where id is key and value is another dict with type and value field and value should be array ? — rasjani
– rasjani, Commented May 8, 2020 at 14:02
I would use a defaultdict(set), mapping id to the unique set of values. — 0x5453
– 0x5453, Commented May 8, 2020 at 14:02

asheidan · Accepted Answer · 2020-08-31 09:51:34Z

2

Using the hints that were already added as comments you can create something similar to this:

from collections import defaultdict

values = defaultdict(set)
types = dict()

for payload in input_stream:
    id, type, value = get(payload)
    values[id].add(value)
    types[id] = type

for id in types.keys():
    out.write(";".join(map(str, [id, types[id]] + list(values[id]))) + "\n")

If the values is more of a time series (order is important), then replace set with list.

edited Aug 31, 2020 at 9:51

answered May 8, 2020 at 14:09

asheidan

1495 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sedavidw · Accepted Answer · 2020-05-08 14:04:24Z

2

You want to keep a dictionary where the key in the dictionary is your id and the value is a set and you can add all the values

from collections import defaultdict
results = defaultdict(set)

id, type, value = getId(payload)
results[id].add(value)

This dictionary will then keep all the values per id.

It should be noted that using a set will remove the count of duplicates, so if you want to keep every value, including duplicates use a list instead

answered May 8, 2020 at 14:04

sedavidw

11.8k18 gold badges70 silver badges103 bronze badges

Collectives™ on Stack Overflow

Detect duplicate elements python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related