Thread by @darth_na

1/?
:: Crawling Large, Huge and Mega sites ::
:: Partial vs Full crawls ::

@JetOctopus has done a piece looking at some of the issues that can arise from not doing a Full Crawl,
or, as I'd phrase it, doing a really shoddy Partial Crawl.

#SEO #Crawling

>>>

https://twitter.com/JetOctopus/status/1579752117891633152

2/?
>>>

Personally, I hold a slightly different opinion.

Yes - you should do a Full Crawl,
but you shouldn't need to do one every single time.

Instead, you should do an initial,
and run one after any significant change (such as platform etc. (preferably dev ver. first)

>>>

3/?
>>>

But most of the time, Partial Crawls should be sufficient,
if done properly!

So, here's a quick guide for Partial Crawling ...

> Priorities
You should know what pages are Vital to the site/business

> Updates
You should know URLs of new and altered content

>>>

4/?
>>>

> Changes (2)
You should know what URLs represent which files/templates (for code changes)

> Site knowledge
You should know if the "site" is made up of different platforms, the PageTypes and templates, the sections and depths of them etc.

>>>

5/?
>>>

> Sampling
You should pick a reasonable representative figure per Platform/PageType/Section
(as well as Priority/Updated URLs)

(No, 1K out of 85K (<1%) is Not good - and anyone sampling less than 10% should be beaten with a Haddock :D)

>>>

6/?
>>>

Now - I know that this sort of thing can seem a lot of work … but if you get a little Dev help,
it's actually really easy.

Have Meta added that includes some additional information (such as Template, Content Type etc.),
with Edit Dates.

And use the Sitemap!

>>>

7/?
>>>

This will give you a % of the site,
with a fair number of URLs per platform, section and each set of templates/content types,
and cover all the New content,
and any recently Updated content.

In most cases, it will still be less than 25% of the site.

>>>

8/?
>>>

Are Partial Crawls perfect?
No.

But they are far faster (and typically cheaper).
And due to that, they permit you to find issues sooner,
and more responsive to issues.

You don't need to wait for hundreds of thousands/millions/tens of millions of pages!

>>>

9/?
>>>

You can also make it part of your normal Full Scan,
but do it in 2 parts (or more!).
Run the Priority crawl,
then the Secondary whilst going through the Priority results.

Much more efficient!

The key is knowing what to crawl!

10/?
>>>

And, if you haven't read it yet,
here's the article by @JNesterets,
more than worth a read!

jetoctopus.com/partial-crawli…