Situation 1: Set USER_AGENT to default, and don’t open Downloader_middlewares
Results with user_agent=”mozilla/5.0(windows;u;windows nt 5.1;en-us;rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1″ Proxy
Situation 2: Open Downloader_middleware and use 1000 custom randomware.
Results with user_agent=”mozilla/5.0(windows;u;windows nt 5.1;en-us;rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1″ Proxy
Reason: Downloader_middlewares_base and Downloader_middlewares, the lower the value, who gives priority to the execution of the priority execution
(Use user_agent=”mozilla/5.0(windows;u;windows nt 5.1;en-us;rv:1.8.1.2pre) Gecko/2007021. Ninja/2.1.1. The value to use, the reason is unknown and will be studied later)
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 400
Приоритетный порядок выполнения больше, чем
'ImgCrawler.middlewares.RandomUserAgentMiddleware': 1000,
correct position:
settings settings:
Mid level settings:
Spider test: