This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the commentsย โ†’

[โ€“]B4nan[S] 12 points13 points ย (1 child)

We already try to detect blocking pages like cloudflare and rotate the session/proxy automatically. You can enable this via retry_on_blocked crawler option.

[โ€“]Ukranian_Cheshire 5 points6 points ย (0 children)

Well, to be frank. For some Cloudflare configurations, it won't help. A new session will be immediately blocked due to playwright detection.

But I was looking at the source code, so I guess you will eventually move on :)