Using multiple proxies with arun_many() #626
Unanswered
Apapora
asked this question in
Forums - Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
In issue #246 there is this example to bypass 429 "too many requests" errors:
This code will crawl many urls using only one of the proxy addresses in PROXY_LIST, which quickly triggers 429 responses in my testing. Is there a way for the crawler to fully utilize all available proxies intelligently when using arun_many()? Say I have a list of 5 proxies and 'urls' is a list of 10 URLs, ideally the crawler would know to send 2 requests per each proxy. Or instead of using arun_many(), should I keep calling arun() with my separate rotating proxy logic? Thanks for any help with this.
Beta Was this translation helpful? Give feedback.
All reactions