A fully automated workflow that you've never seen before.
(thread)
1. This script scrapes the disallowed paths from the robots.txt files of a list of domains and saves them to a single file. It also removes any unwanted entries and sorts the file in a particular way.
Can you write it yourself? Here’s how the script should look like.
2. Create a directory called "massrobots" in the pwd. This is where you'll save all the robots.txt files for later processing.
3. Read a file passed as an argument (referred to as "$1" in the script) which contains a list of domains/subdomains.
4. For each domain in the file, run another script "getrobo.py" passing the current domain as an argument.
5. The script "getrobo.py" makes a request to the robots.txt file for the passed domain and writes the response to a local file with the same name as the domain.
6. Filter the contents of the downloaded robots.txt file to only show the "Disallow" entries and remove the "Disallow" keyword.
7. Sort the filtered contents and write them to a file in the "massrobots" directory using the same name as the domain.
8. Remove the original downloaded robots.txt file.
9. For each file in the "massrobots" directory, append the domain name to each line of the file.
10. Concatenate all the files in the "massrobots" directory into a file called "waybackbots.txt". Remove the "massrobots" directory.
11. Remove any lines starting with ".", "/.", "?." "","/","?" from the "waybackbots.txt" file and sort the entries.
12. Finally, concatenate the original file passed as an argument to the script and rename the file to "waybackbots.txt". This step is optional and does not affect the results.
13. Comment below if you were able to build the script. If not and if this tweet gets 836+ likes, I’ll share my version on my Github github.com/cristivlad25
14. If you enjoyed this thread, there's much more to come! So, stay tuned.
Motivate me to continue posting by liking, retweeting, and following me @cristivlad25.
2023 Hacker's Guide: How to Break into Pentesting and AppSec.
(thread)
1. Hands down, one of the best practical resources is @RealTryHackMe. I would take the "Complete Beginner" learning path, then the "Jr. Penetration Tester" path. It will get you off the ground in no time.
The most frequent vulns I found in 80+ pentests in 2022.
(thread)
1. Rate Limit Bypass
In more than half of the pentests I conducted, I found that there were no rate limits imposed on login functions or authentication mechanisms. This lack of rate limiting, combined with weak password policies, can lead to accounts being easily compromised.
2. Session Token not Invalidated upon Logout
This vuln is caused when the session token is not invalidated at user logout. If the validity of the token or the time to expiry is long, this can lead to the token being leaked, potentially resulting in a data breach or other vulns.