How to give delay between each requests in scrapy?

class S(Spider):
    rate = 1

    def __init__(self):
        self.download_delay = 1/float(self.rate)

rate sets a maximum amount of pages could be downloaded in one second.

There is a setting for that:

DOWNLOAD_DELAY

Default: 0

The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard.

DOWNLOAD_DELAY = 0.25    # 250 ms of delay

Read the docs: https://doc.scrapy.org/en/latest/index.html

Beside DOWNLOAD_DELAY, you can also use AUTOTHROTTLE feature of scrapy, https://doc.scrapy.org/en/latest/topics/autothrottle.html

It changes delay amount between requests depending on settings file. If you set 1 for both start and max delay, it will wait 1 second in each request.

It's original purpose is to vary delay time so detection of your bot will be harder.

You just need to set it in settings.py as follows:

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 3

You can also set 'download_delay' attribute on spider if you don't want a global download delay. See http://doc.scrapy.org/en/latest/faq.html#what-does-the-response-status-code-999-means