importing JSON to mongoDB using pymongo

Question

i am trying to import a JSON file i pull from a URL and send it to mongoDB as is, using the pymongo module.

I have the following code

#!/usr/bin/env python
import sys, urllib2, json, pymongo
from pymongo import MongoClient
myurl = "https://gist.githubusercontent.com/border/775526/raw/b921df18ba00262ab5bba8cadb3c178e1f7748f7/config.json"
response = urllib2.urlopen(myurl)
data = response.read()
connection = MongoClient('mongodb://user:[email protected]:27017/database')
connection.database_names()
db = connection.database
posts = db.posts
post_id = posts.insert_many(data).inserted_id

upon executing this, i get this error raise TypeError("documents must be a non-empty list") TypeError: documents must be a non-empty list

ideally, i want to just be able to pull the json from the url and update the mongoDB as this json file will be updated every week. Thanks

Well, have you debugged your code to check at every stage what each variable holds, both data and type? — Vince Bowdren
– Vince Bowdren, Commented Mar 29, 2017 at 14:08
To import local file, is the urllib2 required? Or I can do this in other way? — mcgtrt
– mcgtrt, Commented Sep 4, 2018 at 18:34

A. Jesse Jiryu Davis · Accepted Answer · 2017-03-29 15:11:35Z

22

You need to convert JSON to Python objects, which PyMongo will then convert to BSON for sending to MongoDB. To convert JSON to Python objects use the "bson.json_util" module included with PyMongo:

from bson import json_util
data = json_util.loads(response.read())

The standard Python json.loads() function works, too, but PyMongo's json_util.loads() handles some MongoDB-specific details better.

answered Mar 29, 2017 at 15:11

A. Jesse Jiryu Davis

24.1k5 gold badges60 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Yannis · Accepted Answer · 2018-06-12 11:48:54Z

-1

You can also use the "requests" library:

response = requests.get(myurl)
data = response.json()

and post that data to the database.

edited Jun 12, 2018 at 11:48

Yannis

1,7198 gold badges30 silver badges51 bronze badges

answered Jun 12, 2018 at 10:31

Srujan Kumar

193 bronze badges

Collectives™ on Stack Overflow

importing JSON to mongoDB using pymongo

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related