Squeezing the juices out of robots.txt.
A fully automated workflow that you've never seen before.
(thread)
A fully automated workflow that you've never seen before.
(thread)
1. This script scrapes the disallowed paths from the robots.txt files of a list of domains and saves them to a single file. It also removes any unwanted entries and sorts the file in a particular way.
Can you write it yourself? Here’s how the script should look like.
Can you write it yourself? Here’s how the script should look like.
2. Create a directory called "massrobots" in the pwd. This is where you'll save all the robots.txt files for later processing.
3. Read a file passed as an argument (referred to as "$1" in the script) which contains a list of domains/subdomains.
4. For each domain in the file, run another script "getrobo.py" passing the current domain as an argument.
5. The script "getrobo.py" makes a request to the robots.txt file for the passed domain and writes the response to a local file with the same name as the domain.
6. Filter the contents of the downloaded robots.txt file to only show the "Disallow" entries and remove the "Disallow" keyword.
7. Sort the filtered contents and write them to a file in the "massrobots" directory using the same name as the domain.
8. Remove the original downloaded robots.txt file.
9. For each file in the "massrobots" directory, append the domain name to each line of the file.
10. Concatenate all the files in the "massrobots" directory into a file called "waybackbots.txt". Remove the "massrobots" directory.
11. Remove any lines starting with ".", "/.", "?." "","/","?" from the "waybackbots.txt" file and sort the entries.
12. Finally, concatenate the original file passed as an argument to the script and rename the file to "waybackbots.txt". This step is optional and does not affect the results.
13. Comment below if you were able to build the script. If not and if this tweet gets 836+ likes, I’ll share my version on my Github github.com/cristivlad25
14. If you enjoyed this thread, there's much more to come! So, stay tuned.
Motivate me to continue posting by liking, retweeting, and following me @cristivlad25.
#pentesting #appsec #infosec #cybersecurity #hacking #bugbountytips
Motivate me to continue posting by liking, retweeting, and following me @cristivlad25.
#pentesting #appsec #infosec #cybersecurity #hacking #bugbountytips
• • •
Missing some Tweet in this thread? You can try to
force a refresh
