Release notes

Scrapy 2.4.1 (2020-11-17)

• Fixed feed exports overwrite support (issue 4845, issue 4857, issue 4859)
• Fixed the AsyncIO event loop handling, which could make code hang (issue 4855, issue 4872)
• Fixed the IPv6-capable DNS resolver CachingHostnameResolver for download handlers that call reactor.resolve (issue 4802, issue 4803)
• Fixed the output of the genspider command showing placeholders instead of the import part of the generated spider module (issue 4874)
• Migrated Windows CI from Azure Pipelines to GitHub Actions (issue 4869, issue 4876)

Scrapy 2.4.0 (2020-10-11)

Highlights:

• Python 3.5 support has been dropped.

• The file_path method of media pipelines can now access the source item.

  This allows you to set a download file path based on item data.

• The new item_export_kwargs key of the FEEDS setting allows to define keyword parameters to pass to item exporter classes

• You can now choose whether feed exports overwrite or append to the output file.

  For example, when using the crawl or runspider commands, you can use the -O option instead of -o to overwrite the output file.

• Zstd-compressed responses are now supported if zstandard is installed.

• In settings, where the import path of a class is required, it is now possible to pass a class object instead.

Modified requirements

• Python 3.6 or greater is now required; support for Python 3.5 has been dropped

  As a result:

  • When using PyPy, PyPy 7.2.0 or greater is now required
  • For Amazon S3 storage support in feed exports or media pipelines, botocore 1.4.87 or greater is now required
  • To use the images pipeline, Pillow 4.0.0 or greater is now required

  (issue 4718, issue 4732, issue 4733, issue 4742, issue 4743, issue 4764)

Backward-incompatible changes

• CookiesMiddleware once again discards cookies defined in Request.headers.

  We decided to revert this bug fix, introduced in Scrapy 2.2.0, because it was reported that the current implementation could break existing code.

  If you need to set cookies for a request, use the Request.cookies parameter.

  A future version of Scrapy will include a new, better implementation of the reverted bug fix.

  (issue 4717, issue 4823)

Deprecation removals

• scrapy.extensions.feedexport.S3FeedStorage no longer reads the values of access_key and secret_key from the running project settings when they are not passed to its __init__ method; you must either pass those parameters to its __init__ method or use S3FeedStorage.from_crawler (issue 4356, issue 4411, issue 4688)
• Rule.process_request no longer admits callables which expect a single request parameter, rather than both request and response (issue 4818)

Deprecations

• In custom media pipelines, signatures that do not accept a keyword-only item parameter in any of the methods that now support this parameter are now deprecated (issue 4628, issue 4686)
• In custom feed storage backend classes, __init__ method signatures that do not accept a keyword-only feed_options parameter are now deprecated (issue 547, issue 716, issue 4512)
• The scrapy.utils.python.WeakKeyCache class is now deprecated (issue 4684, issue 4701)
• The scrapy.utils.boto.is_botocore() function is now deprecated, use scrapy.utils.boto.is_botocore_available() instead (issue 4734, issue 4776)

New features

• The following methods of media pipelines now accept an item keyword-only parameter containing the source item:

  • In scrapy.pipelines.files.FilesPipeline:
    • file_downloaded()
    • file_path()
    • media_downloaded()
    • media_to_download()
  • In scrapy.pipelines.images.ImagesPipeline:
    • file_downloaded()
    • file_path()
    • get_images()
    • image_downloaded()
    • media_downloaded()
    • media_to_download()

  (issue 4628, issue 4686)

• The new item_export_kwargs key of the FEEDS setting allows to define keyword parameters to pass to item exporter classes (issue 4606, issue 4768)

• Feed exports gained overwrite support:

  • When using the crawl or runspider commands, you can use the -O option instead of -o to overwrite the output file
  • You can use the overwrite key in the FEEDS setting to configure whether to overwrite the output file (True) or append to its content (False)
  • The __init__ and from_crawler methods of feed storage backend classes now receive a new keyword-only parameter, feed_options, which is a dictionary of feed options

  (issue 547, issue 716, issue 4512)

• Zstd-compressed responses are now supported if zstandard is installed (issue 4831)

• In settings, where the import path of a class is required, it is now possible to pass a class object instead (issue 3870, issue 3873).

  This includes also settings where only part of its value is made of an import path, such as DOWNLOADER_MIDDLEWARES or DOWNLOAD_HANDLERS.

• Downloader middlewares can now override response.request.

  If a downloader middleware returns a Response object from process_response() or process_exception() with a custom Request object assigned to response.request:

  • The response is handled by the callback of that custom Request object, instead of being handled by the callback of the original Request object
  • That custom Request object is now sent as the request argument to the response_received signal, instead of the original Request object

  (issue 4529, issue 4632)

• When using the FTP feed storage backend:

  • It is now possible to set the new overwrite feed option to False to append to an existing file instead of overwriting it
  • The FTP password can now be omitted if it is not necessary

  (issue 547, issue 716, issue 4512)

• The __init__ method of CsvItemExporter now supports an errors parameter to indicate how to handle encoding errors (issue 4755)

• When using asyncio, it is now possible to set a custom asyncio loop (issue 4306, issue 4414)

• Serialized requests (see Jobs: pausing and resuming crawls) now support callbacks that are spider methods that delegate on other callable (issue 4756)

• When a response is larger than DOWNLOAD_MAXSIZE, the logged message is now a warning, instead of an error (issue 3874, issue 3886, issue 4752)

Bug fixes

• The genspider command no longer overwrites existing files unless the --force option is used (issue 4561, issue 4616, issue 4623)
• Cookies with an empty value are no longer considered invalid cookies (issue 4772)
• The runspider command now supports files with the .pyw file extension (issue 4643, issue 4646)
• The