Creating a Workspace
The workspace is an area that will help keep your reconnaissance organized. Each workspace has it’s own directory inside the hidden .recon-ng directory in the home directory.
First we will find an organization to recon and build our workspace around this company. We will use HackerOne to get our company.
This is how Wikipedia describes HackerOne:
“HackerOne is a vulnerability coordination and bug bounty platform that connects businesses with cybersecurity researchers (aka, hackers). It is one of the first companies to embrace and utilize crowd-sourced security and hackers as linchpins of its business model, and is the largest cybersecurity firm of its kind.[1]”
Even though we are only performing reconnaissance in a non-intrusive manner, we will use a company from HackerOne’s Directory. Under the right conditions, this company has agreed to recon and scanning. We will only be using recon-ng. Figure 1 shows the company we will use in the tutorial but feel free to select a different company from HackerOne or use any one that you are authorized to test against.
Figure 1: HackerOne Company
Figures 2 and 3 show the scope that is authorized for testing including eligible submissions and domains.
Figure 2: Eligible Items
Figure 3: Allowed Domains
workspaces -h shows us the different option we have a available to use (Figure 4).
Figure 4: Workspace -h
Next we will add our workspace using the following command (Figure 5)
workspaces add
Figure 5: Adding a Workspace
After this command you are automatically placed into your new workspace. workspaces list will show you the status of your workspaces.
Figure 6: List of workspaces
Next, we will add our company and our domain. This will add information to the SQLite database. To add information into the database, we need to understand the schema, the layout of the tables. To look at the schema of the database run the following command (Figure 7)
show schema
Figure 7: Show Schema
There are thirteen different tables, we will view the schema of the tables we use in this tutorial.
add companiesRunning the add companies command will make the other columns available. Press enter if you want to leave that column blank.
Figure 8: Add Company
Add the domain using the following command (Figure 9).
add domains
Figure 9: Add Domains
To verify that the domain was added successfully run the command shown in Figure 10.
show domains
Figure 10: List Domains
A simple way of thinking about adding to the tables is shown in the next Figure 11.
Figure 11: Table Visualization
Now that we’ve added data to the database and know how to ensure that data was manually inserted correctly lets move on to importing and exporting data.
Importing Data into the Database
We will uses theHarvester to gather information about United.com and import this into recon-ng’s database.
From Edge Security http://www.edge-security.com/theharvester.php
“The objective of this program is to gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN computer database.
This tool is intended to help Penetration Testers in the early stages of the penetration test in order to understand the customer footprint on the Internet. It is also useful for anyone that wants to know what an attacker can see about their organization.”
If theHarvester isn’t already installed, i.e. you aren’t using Kali Linux, you can clone it from here: https://github.com/laramies/theHarvester
Figure 12: theHarvester
We called theHarvester to gather data on domain united.com using all the data sources listed in the help screen. We directed the output to my recon-ng folder using the ‘>’ operator. The sample command we used follows:
./theHarvester.py -d united.com -b all > ~/recon-ng/harvester.txtThe file name is harvester.txt. This is an ugly file that well will parse through using a few linux utilities. Sample results are shown in the next figure.
Figure 13: Sample Results
The next step is to make this snippet and clean it up a bit with some Linux utilities. We will use grep and AWK to trim the tree.
Grep and AWK
grep is a command-line utility for searching plain-text data sets for lines matching a regular expression.
This is by no means the perfect way. This is just one of many to get the results you need. Using grep, we will create a list of email addresses from harvester.txt file. (Figure 14)
grep @united.com harvester.txt > united_emails.txt
Figure 14: grep command
If you are interested in the file contents use the cat command to view file the contents in the terminal
cat united_emails.txt
Figure 15: cat results
Next, we will create a list of hosts for import from theHarvester results. (Figure 16)
grep ":" harvester.txt
Figure 16: grep host
Grep will also help create the virtual host list. Also take note that since “united.com” is the only domain in scope, it becomes part of the command.
grep ":" harvester.txt | grep united.comThe pattern that we wanted to match was “=” and I didn’t want to count the lines after the pattern so I chose to use 200 as my line count after the pattern, as shown in Figure 17.
Figure 17 grep for Virtual Hosts
This command was a little harder to figure out. The pattern that we wanted to match was “=” and I didn’t want to count the lines after the pattern so I chose to use 200 as my line count after the pattern.
grep -A200 "=" harvester.txt | grep united.com > virtual_hosts.txtIt is time to import our information into recon-ng.
Using the show modules command, we get a list of modules broken down by categories. We will use import/list module from the Import category.
show modules
Figure 18: Import Modules
The “show info” command shows the options to use and the table and columns that will be needed for the import.
show info
Figure 19: Show Info
To find the column and table, we will use the “show schema” command. This will give use a list of the Tables and the different columns in each.
show schema
Figure 20: Show Schema
To import email addresses, we will the “contacts” table and the email column. Our file name will be the united_email.txt file we created using theHarvester. The “set” statement, sets the variables for the import. The “run” command executes the module.
set TABLE contacts
set COLUMN email
set FILENAME united_emails.txt
run
Figure 21: Email Import
The “show contacts” command show the data inside the “Contacts Table”. This is a second verification that the data imported correctly.
Figure 22: Show Contacts