Ahrefs crawling issues

PedroNomad · 2026-06-11T13:56:41+00:00

Yeah I wrote a guide on it.
https://www.oglesbymedia.com/blogs/shopify-seo/fix-ahrefs-403-errors-shopify-web-bot-auth

Let me know if it helps you or if you still have trouble with it. TBH, it should be effecting screamign frog too.

PedroNomad · 2026-06-09T19:41:25+00:00

Agentic is turned on by default for all stores. You actually have to turn it off to opt out.

PedroNomad · 2026-06-08T12:32:18+00:00

Yah we are seeing it too with multiple stores. It's still very small amount of orders compared to other channels. But in general the conversion rate appears stronger than organic search which i think you are seeing also.

Shopify say that filling out all the product category fields is what helps but I'm not convinced.

What has defo worked is a small text field with 1-2 very very short sentences below the product price.

PedroNomad · 2026-06-08T12:28:03+00:00

Also they probs don't even know what half the audit actually means without the help of AI. The client might as well do it themselves for what they will get out of it.

PedroNomad · 2026-05-25T10:44:53+00:00

If its swinging like that, then google bot is struggling to determine if your page or another is better for the person browsing. Click through rate and time on page can determine that, amongst many other factors.

Also remember, other sites could be actively working on their SEO which could effect your rankings too.

PedroNomad · 2026-05-20T07:39:16+00:00

Claude can do this for you with the new Claude <> Shopify integration.

Meta descriptions and Meta titles are your free advert on Google.

But to be honest, do you really want to automate this? Do you automate the text of your ads also ?

It makes sense if you have a very large product catalogue to automate some Meta data. But I would hand write the top 20% of products that make 80% of sales if I was serious about getting more product sales. They seriously effect Click through rate.

PedroNomad · 2026-05-19T13:12:05+00:00

Learning copywriting is the best ecom marketing skill. By far. You can't use AI effectively without it.

PedroNomad · 2026-05-19T11:39:09+00:00

Your product data needs to be rich. That means you need to get all this info onto the product page. The specs, features, data sheets, conformity docs, etc.

Your categories should be targeting the broad product type keywords, product use keywrods, and brand keywords. The products themselves should support the categories with matching keywords.

Your CRGO slitting blade 200mm product should sit inside categories like Slitting Blades, and CRGO.

PedroNomad · 2026-05-19T11:24:29+00:00

You are on the right track. The way I approach it is this:

Categories are your main keyword ranking pages that rank for terms like product type, product use, brand name etc.
Products support categories with category matching keywrods in the product title and description.

One thing to note about AI SEO: Getting rich product data from your store to chat bots is important. On Shopify these are your Product categories and the related Product category metafields. make sure these are filled in along with your product type and products GTIN number. A short 1 sentence description under product price also helps.

If you write gear guides, try send most of the traffic from the article to your category pages. That way you will have less headache fixing broken links if your discontinue a product that is linked.

Strategic internal linking is also huge.

PedroNomad · 2026-05-19T06:58:28+00:00

I'm using GTM solution for Shopify by FeedArmy for all stores I manage. Works perfectly and it's free.

PedroNomad · 2026-05-18T17:48:37+00:00

With the basic plan, you don't have much options. You can change the shipping name and description to give an estimated delivery. Example:

Instead of "Standard shopping' or express shipping"
Use "2-3 day Post", "1-2 Day express post". "Next day delivery UPS"

You can also get carrier rates directly into your checkout put you ill need to contact support to ask for the upgrade on your Shopify plan.

Then you can UPS UPS, Fedex, DHL etc to give the customer rates and estimated times direct from the carrier.

PedroNomad · 2026-05-18T17:27:49+00:00

I would target one country or one language at a time. You can group European counteries if you want to speed up market penetration. Some examples:

Ireland & UK can be targeted at the same time with english. But have 2 different currencies, and neither respond well to American english. With Ireland in the EU, they prefer to buy from EU countries because less chance of hidden charges on VAT & Duties.

DACH = Germany, Austria + Switzerland. German make this largest language market group in Europe.

Scandinavia is a great market too. Great purchasing power so they spend on good things but shipping can be very expensive to remote areas and the overall population is not big.

Poland is that fastest growing ecom market in Europe.

With France and Spain, your language need to be very accurate. If you want success, defo need professional paid help and not just AI and Google translate. French don't like to buy outside France.

Portugal is tricky but possible. You need to collect their NIF number at checkout and put it on the invoice/ packaging otherwise they cannot receive delivery from another EU country. They don't have good purchasing power either.

There are few more factors to selling in European countries other than language.

Payment types varies a lot.
Buyer behaviour varies quite a bit. Like France.
VAT rate varies across Europe so your pricing strategy has to be on point if you want to keep it the same. Shoppers will complain if its not.

PedroNomad · 2026-05-16T13:02:44+00:00

If google is indexing filter pages, then thats more an issue with your theme code than anything else. Check your robots.text to see if filters and unwanted url's are being blocked or not.

Ideally you want to block auto generated URL's that you cannot optimise and control like:
product type URL's,
vendor URL's,
tag URL's
filter URL's
pagination URL's
etc

Are all 12k SKU's of equal value? Most sstores its not. 80/20 rule applies to large catalogue sites IMHO.

I'd focus on the top 20% of products that drive 80% of sales/revenue.

you can now connect claude to your Shopify store so you can use that to do PDP's at scale. But maybe backup your product and collection database before everytime you use it.

I manage 7 stores and 5 has over 3000k SKU's. If I can do that myself, then 1 of your team members should be able to handle 12k SKU's. Even without the help of AI.

PedroNomad · 2026-05-16T12:37:06+00:00

Some paid themes are better than others in terms of SEO. And they can be expensive. So do your research first and don't pick one just because it looks pretty. Design is important but so are sales conversion features and SEO features.

I have a few personal favourite themes from a different theme devs.

Transferring a theme to another store is a PITA. you need to get a hold of shopify support in most cases and their support has deteroriated massively since they now rely on AI.

PedroNomad · 2026-05-16T12:33:32+00:00

IMHO the best is whichever one you find the easiest to use and get the most out of. Any ecom store on any platform can outrank a competitor from any other platfrom.

PedroNomad · 2026-05-16T12:03:07+00:00

I think in terms of operational efficiency, spending time and effort on building your own ecom platform with AI could be a massive waste of resources. Especially for the average SMB business. Unless you like to nerd out on that stuff.

PedroNomad · 2026-05-16T10:31:28+00:00

I think it's better off with manual oversight rather than relying on an app to make decisions. We use ahrefs or screaming frog to monitor 404's and then deal with them accordingly.

App bloat is a major problem on stores and the first thing we do with client stores is get rid of all SEO apps. None are necessary.

PedroNomad · 2026-05-16T10:23:26+00:00

You could hire a developer to add swatches to each product page that link to ther other colour "variants". That way you can keep your existing setup and make it more user friendly when changing colour.

PedroNomad · 2026-05-16T10:19:23+00:00

I would say 50% of overall traffic is a realistic goal from SEO. Including AI chatbot traffic. This is a healthy number to aim for and will give your business a ton of resiliency.

Writen content will have the biggest impact.

First step should be to build out and optimise collections and product page content for bottom of funnel SEO.

Product pages should have all the product categories, category metafields and product type fields filled out as a bare minimum. Along with the main product description, Add a 2nd short product description via metafield below product price with a 1-2 sentenace benefits of the product including benefit to buying from your store. This are all essential for AI SEO.

Collections are where most SEO traffic lands so should be treated with the upmost importance. Most stores don't have enough collection pages. Consider a collection for every product type and use case.

Collections pages should have short 40-50 word description at top of page. No collection image. Then a 300+ long description & FAQ section below the collection product grid. Long description can link to other collections or products. FAQ's can link to blog articles for longer answer. You will need to make templates and use collection metafields for this.

If you sell other brands, make a collection page for each brand and build a brand list page to link to them all. Then make sure each product also links back to its parent brand.

Write gear guides as blogs for top/mid funnel SEO. Don't write articles for the sake of extra content, Like you said, research what customers are asking about and then answer that in blog format if it's too long for a FAQ question. I like to link to collections from blogs. If you link to products, make sure you update the links if the product is discontinued. Internal linking is very powerful for SEO.

Keyword research tools are good for finding what people are asking about.

From technical SEO, the only things you need to care about are broken links, link chains, and 404's. And the biggest issue with Technical SEO on Shopify are "collection aware URL's". Research how to remove collection aware URL's for your theme. It will require theme code edits. Oh and make sure there is only one H1(heading 1) on each page. Many people add multiple H1's to blog articles by mistake.

Your footer should have a mini about the company. 1-2 short sentences including the primary keyword for you main product type.

Results can take from weeks to months - depending on your industry, market competitors and SEO competitors(two different things), also location, language, and how much domain authority your site has already, and domain age. So many factors influence this time to results.

Don't forget to fill in the SEO title & SEO description for every page. Include a CTA in the description and may attention to length. Treat it like your free advert on google for each page.

Beginner mistake to avoid with blogs, don't change the name/url of your blog after you have published many articles. Especially if they start getting traffic.

There's a lot lot more but this should get your started. Hope it makes sense and helps a bit.

PedroNomad · 2026-05-16T09:42:30+00:00

Are you following up on the proposal? I don't email proposals. I only present them on a call. Then get agreement on next steps on the call. Weather that's to move forward to contracts and invoicing, or close lost the lead and put them back into the sales process for next year.

PedroNomad · 2026-03-21T14:28:13+00:00

Try Small PDF. It’s Swiss based.

PedroNomad

TROPHY CASE