I've been analyzing crawling by the major search engines and link research tools on @yoast together with @jonoalderson.
We store all our logs in Elastic, through Logstash, and analyze them in Kibana.
We have to talk about this. Some of these results are shocking.
We store all our logs in Elastic, through Logstash, and analyze them in Kibana.
We have to talk about this. Some of these results are shocking.
Our logs show @bing consistently crawls more 404s than all the other engines, and seems to keep doing so. 404s are often not cached. This is costing lots and lots of server time, electricity etc.
On our servers, it consumed 10 GB of data in those 30 days.
On our servers, it consumed 10 GB of data in those 30 days.
But: @Yoast is only one website. And while it's dear to me, it's not THAT big. We *have* to talk about how all of these engines all maintaining their own index is ludicrous. Entire server farms are online to crawl the same things over and over again.
If you believe this Forbes article: forbes.com/sites/neilyeoh…
then 10k views per month is the equivalent of driving a car for over 5,000 miles. We do *double* that, on yoast.com, on search engine bots alone. Every month. And so do tons of other sites.
then 10k views per month is the equivalent of driving a car for over 5,000 miles. We do *double* that, on yoast.com, on search engine bots alone. Every month. And so do tons of other sites.
Even *if* @Google is making sure all the electricity *they* use is green, they're not buying green electricity for all the sites they hit. Nor are all those other bandwidth consuming bots.
All these bots put a tax on every website on the internet, by inflating hosting cost. Only @Google has a positive ROI on that for most sites.
There are solutions to this. Everyone using and feeding into @CommonCrawl would be one of them. Better notification systems, so engines only crawl what's truly changed, would be another. I bet there's more. Hit me with your ideas :)
