Why does not Django release fetched db objects?

Question

I have very simple django app:

models.py:

class Product(models.Model):
        name = models.CharField(max_length=1000, default="")
        desc = models.TextField(default="")

views.py:

from django.http import HttpResponse
from models import Product

def fetch(request):
        for p in Product.objects.all()[:300000]:
                pass
        return HttpResponse("done")

I've loaded 300k sample records in MySQL database, turned debug off in settings.py and tried executed fetch view - after it completes, django still sits on 700Mb of RAM

I understand that it needs memory to fetch all these 300k objects, but why on earth it keeps them after view functions exits?

Again, I'm with DEBUG=False, tried this with django dev web server and also with uwsgi and its the same weird behavior.

P.S. Verified with Django 1.4 and 1.5.4 on both python2.6/2.7 Linux 64-bit

Daniel Roseman · Accepted Answer · 2013-10-23 12:05:25Z

3

This is not really anything to do with Django. Generally, Python does not return memory to the operating system until it needs to.

See the effbot's explanation for more detail.

answered Oct 23, 2013 at 12:05

Daniel Roseman

602k68 gold badges910 silver badges923 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Zaar Hai Over a year ago

But somehow if I just allocate array of strings instead of Django QuerySet it gets freed after I exit a view function. There must be something different about querysets. Following your thinking, I can suggest that using Django ORM for anything but small data sets is absolutely impractical.

Exelian Over a year ago

I wouldn't be surprised if strings were handled differently then other (non-primitive) objects. But I haven't tested it.

Zaar Hai Over a year ago

Well, Exelian, you are right - creating a bunch of dummy class instances instead of model objects produces the same behavior. So its not a django, its a Python's memory management.

Henrik Andersson · Accepted Answer · 2013-10-23 11:56:58Z

0

This line is the culprit

for p in Product.objects.all()[:300000]:

The slicing forces the QuerySet to evaluate, ie. hit the database, and it then returns a list() containing the slice of objects, which you then can iterate over. And Django's QuerySet cache keeps it in memory since you might want to iterate over the same "QuerySet" again.

You could optimize this by using an iterator.

It's in the docs here

edited Oct 23, 2013 at 11:56

answered Oct 23, 2013 at 11:50

Henrik Andersson

47.5k16 gold badges102 silver badges94 bronze badges

1 Comment

Zaar Hai Over a year ago

I have no problem with returning 300k objects from DB. But it expect them to be freed when view function finishes - this is what my question is about.

Collectives™ on Stack Overflow

Why does not Django release fetched db objects?

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related