Debugging a Python Scrapy Project in VSCode

Debugging, Development, IDE, Programming, Python

I’ve been working on a small web scraping project using Scrapy in Visual Studio Code using the Python Extension. I’m using a virtualenv with Scrapy installed and my launch.json file looks like this:

    ...
    {
        "name": "Scrapy with Integrated Terminal/Console",
        "type": "python",
        "request": "launch",
        "stopOnEntry": true,
        "pythonPath": "${config.python.pythonPath}",
        "program": "/Users/strefethen/.virtualenvs/scrapy/bin/scrapy",
        "cwd": "${workspaceRoot}",
        "args": [
            "crawl",
            "specs",
            "-o",
            "bikes.json"
        ],
        "console": "integratedTerminal",
        "env": null,
        "envFile": "${workspaceRoot}/.env",
        "debugOptions": [
            "WaitOnAbnormalExit",
            "WaitOnNormalExit"
        ]
    },
    ...
The key parts are the “cwd” and “args” for calling scrapy to start the crawl. My Workspace setting overrides look like this:
// Place your settings in this file to overwrite default and user settings.
{
    "python.pythonPath": "/Users/strefethen/.virtualenvs/scrapy/bin/python",
    "python.venvPath": "~/.virtualenv"
}