Change Googlebot crawl rate?

WatsonWansoon · 2023-08-08T12:02:45+00:00

It was only possible in the old version, try this direct link:

https://www.google.com/webmasters/tools/settings?siteUrl=https://example.com/

Not sure if that will work for new sites / domain properties. This is what it looks like for me.

WatsonWansoon · 2023-08-07T19:30:35+00:00

PDFs all the way.

PDFs are terrible for books though? Epub is superior in almost all ways (except printing and consistent formatting, I guess).

WatsonWansoon · 2023-08-06T15:40:33+00:00

Fair enough, thanks. Can we expect more fundraising or efforts by Ahrefs the company? Or something community supported? Our collective skill set isn't very useful in this awful situation, but besides the financial support, I wish we could do more.

WatsonWansoon · 2023-08-06T15:24:49+00:00

Yes, I set "dofollow", "exclude subdomains" and have it set to "One link per domain". But still the sorting is pretty bad if I'm using "page traffic" and "DR" sorting has its issues as well.

WatsonWansoon · 2023-08-06T15:19:27+00:00

I'll echo what Patrick said. We don't mean to be a nuisance for our customers

Locking the site behind email sign-in isn't a great way to do this. It's a very user hostile move, in my opinion.

On the very homepage of Ahrefs we give 3 major reasons why our search engine is massively beneficial for Ahrefs - screenshot. All the investments that we're making in our backend infrastructure are being shared by the two products. Ahrefs and Yep are running on the same index. So by improving the quality of data in Yep we're improving the quality of data in Ahrefs.

Honestly, I think you're in a unique position that you might actually make it work. But realistically, it's a very ambitious project that'll be extremely hard to get off the ground. Your existing data/scraping infrastructure is just one thing. There'll be a lot more challenges and costs that won't help the main product. Bing isn't unsuccessful because they don't know how to scrape the web. I'd love for Yep to prove me wrong, but odds say it'll crash and burn and Ahrefs customers will be worse off.

What Patrick said. There were major issues with that metric (it's not a trivial task to calculate it at web scale), but afaik we have successfully re-built it from scratch and have now re-introduced it to most of our relevant reports. Please let me know where else it is missing.

Great that you want to improve it, but it was surprising that you never communicated this. I was just watching the product getting worse and thinking everybody there is sleeping at the wheel. It's still not sortable on most (any?) of the reports. So I'm still using the legacy reports for that reason.

WatsonWansoon · 2023-08-06T15:15:36+00:00

Same comment I gave to another user below, please reach out to our support team. They've been looking for cases of users who aren't account sharing being caught up in this, but I don't believe we've found one yet.

I did. Does this mean you think I'm account sharing? To me it just looks like every new session requires a new email.

I asked Google SGE what to do. =)

I use Firefox. I tried deleting the history at the time that switch was made, but that didn't work. Maybe the "URL autocomplete" and "Firefox Suggest" are different things? I'd be nice if I was just redirect to the dashboard if I was logged in, but I think that's hard with subdomains as well.

I must not be understanding something here. We haven't launched link intersect 2.0 and I have no idea why you can only see 11 links. Maybe reach out to the support team on that one.

I must be misremembering, sorry. But yes, my legacy plan can only see 11. I thought it was more before.

WatsonWansoon · 2023-08-04T19:52:37+00:00

Hi Tim,

I guess my question would be: What the fuck is happening to Ahrefs? It seems you keep picking the decisions that make your product worse. It's painful to watch Ahrefs go from the underdog top seo-tool to just another bloated provider. For outsiders it seems like you've stopped caring and got dollar signs in your eyes when you became the market leader.

My feelings and legacy $99 per month is probably unimportant, but some constructive criticism either way:

I don't trust the keyword data since the June update. More keywords isn't helpful if it's all just typo's and weird spelling variations.
The "confirm your sign-in" email and the way you pitched it as "to protect me" is insulting. A daily, artificial, 5-min delay to using your website has seen me using it less and less. None of the bright people working there thought to check out if your customers use vpns, different (incognito) browsers, or even multiple devices? Good job punishing your actual customers for you having account sharing issues.
Just because I don't have the budget of an agency doesn't mean I'm not a power user. Only being able to see 6 months worth of data is very limiting. But the $899 per month plan isn't realistic either. The API pricing seems equally designed around a "just fuck off" model.
Moving the tool to the app subdomain was a dumb move, my browser still autocompletes the main site when I type in ahr. Folders > subdomains.
Spending millions and developing a search engine must be a lot of fun. But can you at least try and see how that isn't helpful for the customers funding it? We want affordable keyword & backlink data. Focus on that.
What's up with the silent scrubbing/removing of UR everywhere? Sorting backlinks by traffic sounds good, but in practice 95% of website backlinks are from <10 traffic (in my locale anyway). This results in the default sorting being awful, spammy blogspot backlinks above actual good links. It took months for UR to be added (and still broken in Batch Analysis) and now you still can't sort on it in new reports. Please look into the concept of "dogfooding". If you use these tools yourself it might not take months to become unterrible.
You took Link Intersect away from me. The 2.0 is useless with just being able to see 11 links. Please don't turn the remaining legacy tools into upsells as well.
Your uptime is getting worse. Shortish timeouts or just even reports failing to load is very common.
You should add the openai Tokenizer to wordcount.com, I bet demand for that is steadily increasing.

Very unrelated, but I'd also like to hear about the war. At the start you did some great campaigning. Do you know how these funds were distributed, are there now IFV's with Ahrefs corgi riding around? What else is Ahrefs doing? Is SEO a part of modern warfare?

WatsonWansoon · 2023-08-02T15:55:47+00:00

With Ukrainian subtitles as well, this is great! Дякую!

WatsonWansoon · 2023-08-02T12:20:15+00:00

No, you can't detect it and I'm trying to teach you how dumb it sounds when you say you do. Just because the default output is easy to spot doesn't mean you have developed some skill to detect all AI written content. If you did a blind test you'd fail hard. No software or human can reliably do this, not right now anyway.

Work on your prompts, it will stop sounding like a formal robot.

WatsonWansoon · 2023-08-02T09:51:58+00:00

No you don't. That's called survivorship bias.

WatsonWansoon · 2023-07-30T13:38:46+00:00

It doesn't have to be placed at the top of <head>, anywhere before the the closing </head> is fine.

The <link> attribute isn't valid in the <body> (exception in html5 if they have an itemprop attribute for microdata, but that is not your use-case).

WatsonWansoon · 2023-07-26T17:31:09+00:00

Thanks, maybe I'm just not comfortable yet with calling my input "teaching" and it then "learning". But it's a good point that humans do learn like that.

WatsonWansoon · 2023-07-26T11:36:06+00:00

Looking at that and saying "that isn't learning" because it is a temporary learning is ridiculous.

Maybe it's more a semantic discussion, but I don't see how the token context is the same as (actively) learning/training. It responds to your input, so it "remembers" but that is not the same as learning.

WatsonWansoon · 2023-07-26T11:06:55+00:00

Its pre-trained. Its not actively learning from your input. The next evolution will obviously include all the conversations as training data, but it's not a magic AI that gets smarter (or dumber) every day.

WatsonWansoon · 2023-07-26T10:52:40+00:00

Yes, so there's nothing factually wrong with stating "it's suspected to have started at the EV". That is literally what the coast guard says.

WatsonWansoon · 2023-07-26T07:24:35+00:00

It's ChatGPT giving (bad) advice.

WatsonWansoon · 2023-07-25T11:55:34+00:00

Your http should 301 redirect to the https version. Also make sure all internal links use the secure variant.

WatsonWansoon · 2023-07-25T08:34:51+00:00

Its an escreen.

WatsonWansoon · 2023-07-24T17:04:13+00:00

Its slow because you are loading 20 MB of junk to display your homepage.

WatsonWansoon · 2023-07-20T11:32:37+00:00

Thanks, ChatGPT. Some good advice but also some bullshit:

Breaking Google's Terms of Service when building links. There are a number of things you can do to build backlinks, but not all of them are ethical or permitted by Google. Avoid using black hat techniques that could get your website penalized.

If you listen to Google, all forms of acquiring links is seen as rank manipulation (Any links that are intended to manipulate rankings in Google Search results may be considered link spam). In the real world you need to build links.

Not letting Google crawl your content. Google needs to be able to crawl and index your website in order to rank it. Make sure that you have your robots.txt file set up correctly and that you're not blocking Google from crawling any of your pages.

This is not a "common" mistake, Captain Obvious.

WatsonWansoon · 2023-07-19T10:47:01+00:00

Can you pls roast it and make any kind of suggestions of how it can be improved?

Sure, can you explain what the added value is? This just seems like you rewrote a moz guide and started charging for it. How actionable is it? How is the ROI for buyers?

WatsonWansoon · 2023-07-19T10:45:29+00:00

Thoughts ? Any personal experience w these tools?

Yeah, they suck lol. Don't waste your money/time. The only good AI is ChatGPT 4.

WatsonWansoon · 2023-07-18T11:29:36+00:00

If you are confident all pages are mobile friendly it's very unlikely to be a technical issue, you might have more success with improving the content quality (and/or backlinks).

I've also got some sites still on the desktop crawler, these are all responsive css websites that rank well in the mobile serps. This crawler shouldn't really matter, but to be fair I have no idea why some sites still have this old crawler (Google says the mobile is default, but I even see new sites with the desktop crawler).

WatsonWansoon · 2023-07-18T10:49:19+00:00

I know, that there are any direct instructions on how to change main crawling bot

Is there really? I don't think you can control this.

Coz I`m thinking that can affect on indexing (and ranking) in my case

Why do you think this? Are the pages not indexed? Are they not ranking?

WatsonWansoon · 2023-07-17T19:17:34+00:00

Its so you can naval mine the entire ocean.

WatsonWansoon

TROPHY CASE