How to make record insert operation faster in django with postgres backend

Question

I build a django based geolocation service which determines user's location by their IP address. The first thing I need to do is to insert ip data into my database.

I used following code(simplified) to insert records into my db:

 for ipLoc in ipSeeker.ipLocationList:
     placeName =ipLoc.country + ipLoc.area
     IPLog.objects.create(
                startIP = int_to_dqn(ipLoc.startIP), 
                endIP = int_to_dqn(ipLoc.endIP), 
                place = placeName
     ).save()

The ipLocationList has approx 400k ip records. And my script only insert 20k records in 20 minutes. It is tooooooo slow and could not be accepted.

So my question is: where is the bottleneck and how could I make it faster?

Thanks in advance!

See my answer on this page: stackoverflow.com/questions/7019831/…. — atrain
– atrain, Commented Aug 18, 2011 at 13:07

Community · Accepted Answer · 2017-05-23 12:26:54Z

1

Use raw sql and transactions for django-side: https://docs.djangoproject.com/en/dev/topics/db/sql/ along with the Copy command from Postgres!

Whats the fastest way to do a bulk insert into Postgres?

edited May 23, 2017 at 12:26

CommunityBot

11 silver badge

answered Aug 18, 2011 at 12:59

StefanNch

2,62926 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

xiao 啸 · Accepted Answer · 2011-08-22 04:15:53Z

1

I found a complete tutorial about how to load GEOIP database into postgresSQL in the siafoo. It is well written and covers most detail. The link is http://www.siafoo.net/article/53

answered Aug 22, 2011 at 4:15

xiao 啸

6,61010 gold badges43 silver badges52 bronze badges

Collectives™ on Stack Overflow

How to make record insert operation faster in django with postgres backend

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related