Debugging Scrapy Project in Visual Studio Code

In order to execute the typical scrapy runspider <PYTHON_FILE> command you must to set the following config into your launch.json:

{
    "version": "0.1.0",
    "configurations": [
        {
            "name": "Python: Launch Scrapy Spider",
            "type": "python",
            "request": "launch",
            "module": "scrapy",
            "args": [
                "runspider",
                "${file}"
            ],
            "console": "integratedTerminal"
        }
    ]
}

Set the breakpoints wherever you want and then debug.

1. Inside your scrapy project folder create a runner.py module with the following:

   import os
   from scrapy.cmdline import execute
   
   os.chdir(os.path.dirname(os.path.realpath(__file__)))
   
   try:
       execute(
           [
               'scrapy',
               'crawl',
               'SPIDER NAME',
               '-o',
               'out.json',
           ]
       )
   except SystemExit:
       pass
   

2. Place a breakpoint in the line you wish to debug

3. Run runner.py with vscode debugger