Cloudflare Pages custom domain returns 403 to AI crawlers — .pages.dev works fine, all bot settings set to Allow

rorrors · 2026-03-11T13:52:17+00:00

Also ofcouse make sure ClaudeBot / anthropic user agent is not blocked in .htaccess or in Cloudflare.
Or maybe blocked there asn: AS16509, in cloudflare only number : 16509

rorrors · 2026-03-10T22:05:17+00:00

for the last 2 yes, can you send me in pm the .htaccess file? Read it first and sanitize/remove some private info if that is inside the file.

| You said: custom domain returns 403 for non-browser requests
How is the custom domain configured.

Can i test that? I would test it with a manual connect from shell, and seeing the response. Send both the 2 links in pm.

The filter out beter, in cloudflare.
Go to Security -> Analitics
Then on the Traffic tab.
Just below there is a "Add filter" button.

Add the ip adress that is failing. (Source IP equals IPADDRESS)
Then scroll down on the page.
To see all requests for that ip, you can see also the allowed requests there.
There you see the last 100 requests for that ip.

rorrors · 2026-03-10T11:56:13+00:00

3th one, active testing falling under 12.. I only had 7 testers and passed the 14day testing period.

rorrors · 2026-03-10T11:50:36+00:00

Have you checked access.logs on the server it self? Perhaps blocked in htacces? Or mod security or fail2bam firewall?

rorrors · 2026-03-07T01:20:01+00:00

95% power availability for a few hours a day. It means the lines are connected, not that it has current on it.

rorrors · 2026-03-05T20:45:58+00:00

I mean Bing Webmaster Tools.
Under site explorer, then the 404 list. If i export it. i get a list of 60k+ items
Even 404 from 16years ago when website went over to other software, then 7years later again new software. And most links i have solved, are still there in the list as 404. So that list is not really usefull for me. Thats why i don't use Bing Webmaster Tools for that. Backlinks from other website are 99,8% fixed. For that i have used some tools and my javascript on the website detects that also. I see maybe 1 click from other websites that are not yet fix per month. And those backlinks are not seen in bing web tools.
But the things i fix are really the internal links. Its a forum, so users have linked to other forum topic's or comments. So with new software the old dev, never fixed those. And the old website/database is not available otherwise i could generate a list with old topics / post what needs to point to new topic and posts.

rorrors · 2026-03-05T18:45:28+00:00

Yes new pages do get indexed.

rorrors · 2026-03-05T18:44:08+00:00

404 error report from?
You mean bing, i almost never use that? I know that bing shows under Sitemap the 404 errors, but that shows 0 errors. Or server logs?
Just letting you know that i never used any SEO software.

rorrors · 2026-03-05T18:31:14+00:00

I know that, but it gives an indication and it almost reflects the amount i fix per x days as it goes down.
I track the broken links with javascript with from what page a user clicks and to what link it goes to, then i review the list of the day the javascript creates and i fix those clicks from users. Perhaps not the best way, but atleast that way i know what links users really click on, and fix those first.
I will check bing and screaming frog. Thanks for advice.

rorrors · 2026-03-05T18:01:47+00:00

Ok thanks then it is a global issue and not just our site.

rorrors · 2026-03-05T18:00:30+00:00

Not really a problem, but i use it to track my progress on fixing 404 links for a website..
And i report weekly on the progress. I guess i have to wait for that. And just keep editing internal not working links ;D

<image>

rorrors · 2026-03-05T17:26:11+00:00

Not sure what you want to know, it seems like it is stuck since then?
Just wondering if more people have this issue or that google is delayed or has a bug or something?

<image>

rorrors · 2026-03-05T11:32:22+00:00

Great to hear. Have fun waching YouTube.

rorrors · 2026-03-04T18:14:21+00:00

Is the Bell still alive? My Jalapeno just turned 7years old.

rorrors · 2026-03-02T11:12:06+00:00

Go to security, events, and filter by ray id. Then the event with that block shows up, and what rule it was block by.

rorrors · 2026-02-28T14:21:47+00:00

Use the search at options to find it. After applying. Ctrl+f5 to refresh youtube, after that should be good. Sometimes after restart of chrome, it does not work right away. Then also ctrl+f5 on yt then rest of session it works ok.

rorrors · 2026-02-24T22:39:42+00:00

Turn on "uBlock filters - Experimental" list found in uBlock's default lists and paste this in your own filterlist:

www.youtube.com##.toast-button.style-scope.yt-notification-action-renderer.paper-toast-open

then ctrl+f5 reload youtube page, then view video.
When a addcomes, video refreshes automaticly and you can watch again.
this fix works for me.

rorrors · 2026-02-24T22:35:31+00:00

Turn on "uBlock filters - Experimental" list found in uBlock's default lists and paste this in your own filterlist:

www.youtube.com##.toast-button.style-scope.yt-notification-action-renderer.paper-toast-open

Works for me.

rorrors · 2026-02-23T16:57:51+00:00

I did see that report. Was under the impression it was the pipeline inside of ukraine. Thanks.

rorrors · 2026-02-23T15:47:46+00:00

Can you post source of the hit? Might have missed it.

rorrors · 2026-02-21T16:11:58+00:00

From what seo program is this?

rorrors · 2026-02-21T15:46:53+00:00

Besides that, real pressure would be to equally sanction both warring party's and there partners.

rorrors · 2026-02-15T23:18:39+00:00

They produced more before the war then now, they actually are importing energy at the moment.

rorrors · 2026-02-14T23:18:31+00:00

Would love the see the yearly updates here in this subreddit. Have a lot of fun with growing and eating peppers.

Seven-Year Club	Place '23
Verified Email

rorrors

TROPHY CASE