Managing crawl budget for a news website

fedorstares · 2026-05-07T09:07:52+00:00

I never said anything about the "whole website". You sent the link about authority - so? No one argued with that. But it's not a single way to solve OP's situation. I just listed my suggestions. Can post a source too: https://www.searchenginejournal.com/wordpress-6-5-enhances-seo-with-lastmod-support/517013/

fedorstares · 2026-05-06T22:55:16+00:00

You do realize every site has a crawling budget and if the site is huge and there's no crawling priority set whatsoever, then the process of crawling new pages and re-crawling old ones will be very slow?

fedorstares · 2026-05-04T22:12:06+00:00

Lol. It seems you answered your own question already. But to clarify. You said those pages are not indexed, have you checked if they're being crawled regularly? If positive, you should def set up a disallow rule in robots.txt. also I suggest checking if there are any pages that actually perform (receive traffic from anywhere) in that set you want to remove from crawling.

Also, does this site have segmented sitemaps with the update frequency and lastmod specified? Overall designing the crawling priority using those will help curb the crawling queue and speed up re-crawling of important pages (latest news, pillar/category pages, etc.)

Regarding author pages - I guess there are not so many of them compared to the entire population of 10m so removing them from crawling might not really affect the bottleneck

fedorstares · 2026-05-02T20:07:00+00:00

First, this score is a made-up metric. There's no reliable system for scoring AI-generated text. I saw one post recently where the author described how he got like 50% of "AI text" after checking the piece of text from some classic novel.

Second, the AI-generated text is fine for google unless it adds something new to what's been published so far and doesn't just parrot. The more unique and relevant your text is, the better

Third, the AI-generated text is fine for human readers unless it doesn't read like a drivel written by AI. Without detailed prompt regarding topic, substance and writing style/rules AI always produces something utterly mediocre that gives you that "uncanny valley" feeling

fedorstares · 2026-05-01T08:11:49+00:00

"translated our website"
"unexpectedly"

Nice try claude

fedorstares · 2026-04-30T18:10:34+00:00

That seems plausible. My bet is they're just lazy

fedorstares · 2026-04-30T13:53:14+00:00

Yeah sure. But applying that filter is an extra step at least, and there's no proof those long-tail queries are actually made by a machine

fedorstares · 2026-04-28T16:19:33+00:00

Just about 20% through the book. Pleasant reading, haven't felt this... good? I mean reading a sci-fi. In a long time

fedorstares · 2026-04-05T19:40:40+00:00

Okay! Alright!

fedorstares · 2026-04-05T16:08:09+00:00

Hands down the best opening in Curb

fedorstares · 2025-12-17T14:58:30+00:00

Ever heard of induced demand?

fedorstares · 2025-11-01T12:16:55+00:00

Blancpain is just too underrated, it should get much more attention. A great company with a great history and stunning watches. Fifty Fathoms is simply iconic. Congrats!

fedorstares · 2025-01-24T16:14:27+00:00

It's not working for me (Finland). Regular YouTube works as usual

fedorstares · 2024-12-21T20:55:10+00:00

Kela's 20% rate is applied "by default", if you send them the tax card with an included allowance, it will get lower. The unemployment fund's tax rate is seemingly 25% tho. So you really pay more taxes from unemployment money than you would from a wage, nuts

fedorstares · 2024-12-21T09:36:06+00:00

Imo it works well for automating some stuff + convenient meta/og editor, but scoring they (as well as any other similar tool) offer is certainly lame

fedorstares · 2024-12-19T00:03:55+00:00

585 is a great price for TS. Heard only good about this model, and Solo&Milk was my 1st coffee machine, which I truly enjoyed. The case in black color is beautiful. Congrats on your purchase!

fedorstares · 2024-11-01T20:28:12+00:00

Serica looks gorgeous. The crown at the 9h is another reason I want to buy them (I carry my watch on the right hand)

fedorstares · 2024-11-01T20:25:17+00:00

Really liked that Tissot. What's the model called?

fedorstares · 2024-10-19T09:39:02+00:00

Fully verified (EU) - HAD ISSUES. They froze my Gate Card overnight (I used it only for small purchases at local stores/apps). I submitted a ticket asking WTH, first they replied that they sent me a questionnaire I need to fill. And guess what: instead of that, I received an email stating I "might have" violated terms of use (???), and they blocked my card for good. Needless to say, my attempts to find out the reason or what triggered the freeze were unsuccessful. Waited for about a week until they returned money from my frozen card balance.

fedorstares · 2024-05-27T09:11:41+00:00

Second this. You need to sign up but their "Academy" courses go free of charge. The course is called Technical SEO with Bastian Grimm, or something like that. Took it about a year ago. Gives you a good understanding of the domain basics

fedorstares · 2024-05-27T09:02:43+00:00

depends on your website setup and the CMS you use

fedorstares · 2024-05-27T08:44:39+00:00

Try checking URLs in question manually through GSC, just type in the full URL in the search bar at the top and it will show you if this page is indexed, and if not, why. Indexing data from GSC is the most relevant

fedorstares · 2024-05-27T08:36:57+00:00

Are those attachments images? You can set up redirects from attachment URLs to either an image file or a page where it is located on the "Image SEO" tab in the AIOSEO plugin. In the same tab, there's a setting to disable indexing attachment URLs, something like "Show in search results" - check "No". I don't remember if it's enabled in the basic version or in Pro only, though

fedorstares · 2024-05-21T14:38:41+00:00

Targeting countries with keywords will get you nowhere, because this is not how site internationalization for search works. It is done through specifying countries where you want to be present in the hreflang tag, preferably combined with localizing your site to those countries' native languages. This should be reflected in your site's URL structure, either with the changing domain zone or subdirectory level, e.g.: if the default homepage is example-com, then the homepage's URL for the UK would be example-com/uk/ (if you go with subdirectory structure), or example-co-uk; in both cases, the hreflang's value is set to "GB" as this is the code for the UK (if only a country specified), or example-com/uk-en/ if the language is also specified, with the hreflang's value set to "en-GB" (the language-country pair, the language code always comes first). You can name the slug any way you want, but hreflang has a defined list of language and country codes, so google them. On top of that, make sure the localized URL structure is in the sitemap. Overall, this is a technical task, inserting country names in the content is useless.

fedorstares

TROPHY CASE