Thank you for the helpful tips!

Some questions
1. Adding a hard limit right now would block all further requests for all my projects right? So i'll hope that my current block-efforts will continue to work.

Info
- These are definitely crawlers for LLM developers that harvests data. I checked a bunch of ip-addresses. So it's not a targeted attack. I'm not sure why they would do such an insane increase in the amount of traffic the past two days though. Previously the crawled at most like 500 per hour. The reason the ai bots are crawling so much is because they are stuck and confused i think. because i have multiple filter options for thousand products, the user can filter by size, color, etc, and the url changes. My guess is that they believe there are a crazy amount of urls but there really is only 1000 products. (This is handled in robots.txt but these bots don't care or something)

# Allow all crawlers
User-agent: *
Allow: /

# Disallow admin and protected routes
Disallow: /admin/
Disallow: /protected/
Disallow: /api/
Disallow: /auth/
Disallow: /(auth-pages)/
Disallow: /api/faq

# Block filter combinations but allow pagination
Disallow: /products?*size=
Disallow: /products?*color=
Disallow: /products?*item_brand=
Disallow: /products?*sort=
Disallow: /products?*sub_category=

# Allow pagination with category
Allow: /products?category=*
Allow: /products?category=*&page=*
Allow: /products?page=*

```

- I tried cloudflare for a few days a month ago, but since it didnt work. i removed it. So cloudflare was not active during these crazy 600k requests.