TypeError when importing json data with pymongo

Question

I am trying to import json data from a link containing valid json data to MongoDB. When I run the script I get the following error:

TypeError: document must be an instance of dict, bson.son.SON, bson.raw_bson.RawBSONDocument, or a type that inherits from collections.MutableMapping

What am I missing here or doing wrong?

import pymongo
import urllib.parse
import requests

replay_url = "http://live.ksmobile.net/live/getreplayvideos?"

userid = 769630584166547456

url2 = replay_url + urllib.parse.urlencode({'userid': userid}) + '&page_size=1000'

print(f"Replay url: {url2}")

raw_replay_data = requests.get(url2).json()

uri = 'mongodb://testuser:[email protected]:45687/liveme'

client = pymongo.MongoClient(uri)

db = client.get_default_database()

replays = db['replays']

replays.insert_many(raw_replay_data)

client.close()

check your raw_replay_data try to convert in it dictionary then try to save it — Abdullah Ahmed Ghaznavi
– Abdullah Ahmed Ghaznavi, Commented Mar 6, 2018 at 13:46
@AbdullahAhmedGhaznavi Can you give me a hint how to do that? :) — aquatic7
– aquatic7, Commented Mar 6, 2018 at 13:50
insert_many accepts a list of dictionaries. I guess your raw_replay_data is a dictionary. use insert_one and check the results — Alireza
– Alireza, Commented Mar 6, 2018 at 14:04

Rohit Shende · Accepted Answer · 2018-03-06 14:12:12Z

1

I saw that you are getting the video information data for 22 videos.

You can use :

replays.insert_many(raw_replay_data['data']['video_info'])

for saving them

answered Mar 6, 2018 at 14:12

Rohit Shende

564 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

aquatic7 Over a year ago

When I run it twice I get duplicate entries. Can you help me with how I don't insert already existing videos, but only add new videos. Make it skip already inserted entires.

Rohit Shende · Accepted Answer · 2018-03-07 10:42:39Z

1

You can make one field as _id for mongodb document

use the following line before insert_many

for i in raw_replay_data['data']['video_info']:
    i['_id'] = i['vid']

this will make the 'vid' field as your '_id'. Just make sure that the 'vid' is unique for all videos.

answered Mar 7, 2018 at 10:42

Rohit Shende

564 bronze badges

Collectives™ on Stack Overflow

TypeError when importing json data with pymongo

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related