pass user agent in scrapy code example
Example 1: scrapy user agent
#add this lines into settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}
Example 2: How to pass a user defined argument in scrapy spider
class VisualScrapySpider(scrapy.Spider):
name = 'visualscrapy'
def __init__(self, category='', **kwargs):
self.start_urls = [f'http://www.visualscrapy.com/search?query={category}']
super().__init__(**kwargs)
#scrapy crawl visualscrapy -a category=<search_parameter>