Closing database connection from pipeline and middleware in Scrapy

Question

I have a Scrapy project that uses custom middleware and a custom pipeline to check and store entries in a Postgres DB. The middleware looks a bit like this:

class ExistingLinkCheckMiddleware(object):

    def __init__(self):

        ... open connection to database

    def process_request(self, request, spider):

        ... before each request check in the DB
        that the page hasn't been scraped before

The pipeline looks similar:

class MachinelearningPipeline(object):

    def __init__(self):

        ... open connection to database

    def process_item(self, item, spider):

        ... save the item to the database

It works fine, but I can't find a way to cleanly close these database connections when the spider finishes, which irks me.

Does anyone know how to do that?

Community · Accepted Answer · 2017-05-23 10:32:29Z

7

I think the best way to do it is to use scrapy's signal spider_closed, e.g.:

from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher

class ExistingLinkCheckMiddleware(object):

    def __init__(self):
        # open connection to database

        dispatcher.connect(self.spider_closed, signals.spider_closed)

    def spider_closed(self, spider, reason):
        # close db connection

    def process_request(self, request, spider):
        # before each request check in the DB
        # that the page hasn't been scraped before

1 Comment

Jamie Brown Over a year ago

I wasn't aware of the spider_closed signal. That's perfect - thank you!

Collectives™ on Stack Overflow

Closing database connection from pipeline and middleware in Scrapy

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related