Profile picture
, 4 tweets, 1 min read
My Authors
Read all threads
I stopped complaining about the challenges with understanding how Google parses robots.txt and made (a version of) their open source parser available on the web instead: distilled.net/resources/free…
Stopped complaining *for now* I should say
My tool does have differences compared to the old search console one (because the SC one is wrong) and compared to the open source tool (because that doesn't capture all google crawler subtleties). I explain all in the post
Here's the tool itself: realrobotstxt.com

Let me know if you spot any issues!
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Will Critchlow

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

More from @willcritchlow see all

Profile picture
Will Critchlow
@willcritchlow
OK. Here we go - thread of answers to questions that came up during my #FOS19 presentation in Amsterdam today - about SEO / CRO / full funnel testing (read more here: distilled.net/resources/anno… ) cc @basvandenbeld
Q: how do you test the homepage of a website?

A: although you can run CRO tests on a homepage, SEO (and hence full funnel) tests require a site section with multiple pages with similar template. You can only really do before/after tests. [contd]
The techniques I described are mainly applicable to large websites with large site sections (e.g. ecommerce, real estate, travel, jobs, large brick+mortar chains etc). In these cases, most organic traffic is not to the homepage
Read 14 tweets

Related threads

Profile picture
Christian Oliveira
@christian_wilde
I recently had to deal with some robots.txt behaviour that "didn't make sense" and made me feel confused, until I discovered some things I didn't know, so I will share them here in case they're useful for anybody #SEO
The robots.txt file was something similar to this:
- A disallow for /directory/
- Two lines allowing Googlebot to crawl any .css or .js file

But, when testing it with the robots.txt tester, it allowed Googlebot to crawl /directory/
After several minutes trying to understand, I tried deleting the second set of rules (the ones allowing Googlebot to crawl css/js files), and after that, the first rule worked as I expected:
Read 15 tweets
Profile picture
Ruptly VU
@RuptlyVU
Turkish military began attacking Kurdish positions in north-east #Syria this week, days after US troops pulled back from the area.

Here's a detailed insight into the verification & validation work behind some of the footage we've published for our clients. #OSINT
On Monday [Oct 7] our cameraman filmed exclusive footage of US forces withdrawing from their position close to Tell Abyad near the Turkey-Syria border. Our footage shows the convoy moving through Tell Abyad.

Ruptly footage:
ruptly.tv/en/videos/2019…
Before filming, our cameraman sent us this image of him at the position where he filmed US forces assembled along the 712 highway, approx 6 kilometers to the east of Tell Abyad.
Read 80 tweets
Profile picture
The Epoch Times
@EpochTimes
What if I told you that #BigTech and #SocialMedia platforms are manipulating you, steering you toward #Health information that they think is right, rather than letting you evaluate content for yourself?

Commentary by @MaryamHenein (Thread 👇) theepochtimes.com/google-censors…
Accredited professionals, meanwhile, who stand for #Health freedom and who criticize Big Anything, are losing posting privileges, getting banned, being buried, finding themselves deranked, and getting digitally assassinated.
Content is literally disappearing from the Internet along with our #Health choices.

It sounds conspiratorial because it is. Independent #Journalists and health experts are blowing the whistle. But it’s become a silent one.
Read 94 tweets
Profile picture
T. Benjamin
@1FreeInhabitant
Building an open-source app that will handle:

Email, voice, video, streaming, blog, social, automation, subscriptions, publishing, chat, forums, contracts, store, inventory, accounting.

Virtualized, decentralized, encrypted, & automated.

Price to use = computer & electricity.
2) The differences this time around is that:

- I don't have an employer
- There is no profit motive
- No deadline
- An enormous (potential) user base

You.
3) Because... I'm willing to bet that YOU want pretty much the same thing I want.

- Control over my privacy
- Control over my communication
- Control over my content
- Control over business affairs

Pretty much the same thing that your current providers want, I suspect.
Read 28 tweets
Profile picture
Manuela Battaglini
@manuelabat
Sábado, sabadete, por lo que hoy toca hilo sobre #privacidad y #protecciondedatos o (des)protección de datos. Tema central de hoy: ¿Por qué estamos actualmente en una situación en la que la privacidad y la falta de transparencia se han convertido en cuestiones legales centrales?
Obviamente, se debe a un rápido desarrollo tecnológico, pero quizás sea útil para nuestra discusión sobre #transparencia, #privacidad y #perfiles.
para cavar un poco más profundo en el por qué, y no en el cómo. ¡Adelante hilo!
Dos hechos tecnológicos hicieron posible esta enorme transformación: 1) el desarrollo de hardware informático y 2) el desarrollo de un software que permitió que muchos ordenadores funcionaran como uno solo. ¿Qué supusieron estos grandes hitos para las organizaciones y empresas?
Read 35 tweets
Profile picture
Brian Grant
@bgrant0607
Kubernetes Borg/Omega history topic 10: In honor of #KubeConEU and the 5th anniversary of open-sourcing Kubernetes, I’ll add more perspective from the Borg and Omega teams to the origin story
Internally, Google puts a lot of emphasis on both resource efficiency and engineering efficiency. For both reasons, back in June 2013, a few months before GCE was ready to GA (cloudplatform.googleblog.com/2013/12/google…), the Borg and GCE teams started to work more closely to improve both
The initial focus gravitated towards directly supporting features needed by Cloud in Borg so that Cloud wouldn't need to work around the the lack of those features (see my previous comment on 2-level scheduling: )
Read 17 tweets

Trending hashtags

#Google #Patriots #team21con #datospersonales #Honeybees #CQRS #SecureDrop #Communities #NoSQL #tech #protocols #Technology #HouseIntelligenceCommittee #tool #FakeNews #Traffic #HAMMER #Fascism #transparencia #Internet #CFR #ditto #HealthAndWellness #GoogleGDC2019 #AlternativeMedicine
Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!