Caching issues with django or postgresql

Question

I have an issue where my production server displays incorrect results for a query (create and deletes were properly accounted for but not updates to existing fields which impacted the results). I assumed it was a date comparison issue and spent several hours debugging as I could not reproduce the error on dev. Eventually, I restarted gunicorn and the issue disappeared. It seems either postgresql or django was caching the queryset results. I have not set up caching so do not understand why this is happening.

I have 2 questions please:

Who is doing the caching?
How can I disable it? - I need to queryset to be evaluated each time it is called

Setup

Python 2.7

Django 1.7

PostgreSQL

Ubuntu

views.py Extract

class EventList(ListView, FilterMixin):
    model = Listing
    queryset = Listing.events.order_by('-from_date', )

models.py Extracts

class Listing(models.Model):
    title = models.CharField(max_length=200, null=True)
    to_date = models.DateTimeField(null=True, blank=True)
    from_date = models.DateTimeField(null=True, blank=True)
    events = EventManager()
    event = models.ForeignKey('Event', related_name='listings', null=True, blank=True, on_delete=models.SET_NULL)

class EventManager(models.Manager):
    def get_queryset(self):
        now = arrow.now()
        # Only listings which have events ending after today
        qry = super(EventManager, self).get_queryset().filter(event__isnull=False).select_related('event').filter(to_date__gte=now.datetime)
        return qry

This is unlikely to be caching or anything to do with the database; more likely to be a queryset evaluated at class or module level. But we can't help without seeing code. — Daniel Roseman
– Daniel Roseman, Commented Oct 8, 2015 at 13:33
@DanielRoseman Please see updated question. The issue I am facing is that when a listing is withdrawn, the to_date is set to 1 day before today, but the queryset (the one in EventManager) continues returning it until gunicorn is restarted. Thanks for helping — RunLoop
– RunLoop, Commented Oct 8, 2015 at 14:20
PostgreSQL does not cache result sets. Ever. So it's not PostgreSQL, unless perhaps you're running in a SERIALIZABLE transaction and seeing an old snapshot, in which case that's by design and your app asked for it. — Craig Ringer
– Craig Ringer, Commented Oct 9, 2015 at 4:41

ppetrid · Accepted Answer · 2015-10-15 22:09:19Z

7

+50

This seems like an import-time evaluation problem (which in a way is a date comparison issue as you said).

What happens is once your views.py module is imported, the queryset member of the EventList class is evaluated. The result is a QuerySet instance, which has been calculated at import time. So the .filter(to_date__gte=now.datetime) part is actually executed right then and therefore the date is actually always the same (the date at import time).

When the view is called, all django does behind the scenes is to append a .all() to the queryset in an effort to force a new evaluation of the query. But since the datetime filter was actually evaluated upon loading the module, it will always filter by the same date. For example, if you insert new listings dated before the server was restarted, I would expect you to get these new results!

For cases like these where dynamic filtering is required, you should override the ListPageView.get_queryset method instead, to enforce run-time evaluation:

class EventList(ListView, FilterMixin):
   def get_queryset(self):
      return Listing.events.order_by('-from_date')

edited Oct 15, 2015 at 22:09

answered Oct 15, 2015 at 22:04

ppetrid

3,83529 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

iankit Over a year ago

Have deleted my answer. This answer makes perfect sense.

ppetrid Over a year ago

Thanks @iankit. I wouldn't delete the answer though, your debugging/test flow could help someone else with similar problems - especially since unlike the title suggests this does not seem to be an actual caching problem!

iankit Over a year ago

Yeah, thats a generic answer. Fear I might get downvotes on that. !

Community · Accepted Answer · 2017-05-23 11:44:46Z

0

If you havent added any cache layer, it is impossible that the Django has caching. It doesnt come out of the box in Django. You have to look elsewhere for the root cause of the problem. With the partial code you have given, it isnt possible to speculate the possible causes.

Try to find differences between your dev server and production server, only that can give you initial cues about pinpointing the problem.

Is the production server located in different timezone?
What are the differences in prod and dev's environment and environmental vars?
What changes on gunicorn restart?
If the bug surfaces again on production, how long did it take to come back. Can you add log events on production using tools such as logstash and kibana.
Python language has some non intuitive aspects of date time handling. You can read more about it, one article is here. - https://opensourcehacker.com/2008/06/30/relativity-of-time-shortcomings-in-python-datetime-and-workaround/, Python date function bugs

These are the generic strategies you can use to figure out root cause.

edited May 23, 2017 at 11:44

CommunityBot

11 silver badge

answered Oct 13, 2015 at 18:00

iankit

9,44210 gold badges53 silver badges57 bronze badges

1 Comment

iankit Over a year ago

This isnt an accurate answer. Please see the other answer. I am not deleting this answer because this might help people in debugging such issues in general.

Collectives™ on Stack Overflow

Caching issues with django or postgresql

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related