Inventory

Let’s take inventory of the information we now have and decide where we will go from here.

Figure 1 – Information Inventory

Using Modules

The three commands we used (show domains, show contacts, and show companies) will help us to decide which modules to use. The show modules command will display a list of modules to choose from.

show modules

Figure 2 – show modules

As a quick note for looking at the modules, the “-” delimiter divides the module into, “what you have and what you want”. So your command would look something like this: use I have recon/domains I want hosts/shodan_hostname

use recon/domains - hosts/shodan_hostname

Figure 3 – recon-ng to shodan module

The red text indicates that an error occurred when running the module. The green text indicates the new elements added to the database.

Figure 4 – shodan summary

The module added hosts so using the show hosts command will show the additions.  Notice that we also have ports as well.

show hosts

Figure 5 – show hosts results

Notice this command displays the row id, the host, the ip address, and the module that was used.

show ports

Figure 6 – show ports results

Remove Unwanted Entries

If we wanted to stay in the .com domain, we need a way to remove the .hk and other domains.

help delete

Figure 7 – help delete results

Remember show ports was the last command we ran so ports was the table we viewed. Running the show ports command again shows that the selected rows were removed ONLY for the ports table. To validate the command worked we will check the table again.

show ports

Figure 8 – Cleaned ports table

The .hk domains are still present in the hosts table.  You will need to remove them from each table.

show hosts

Figure 9 – show hosts results

Exporting Data and Report Generation

Now that we’ve imported data from an outside source, ran several modules inside recon-ng, and we’ve even deleted data from the database, it’s time to create our report.  There are lots of options to choose from. The search reporting command gives us our choices.

search reporting

Figure 10 – search reporting results

The show dashboard command allows us to look at the modules used and the number of times they’ve been ran.  We can also see the amount of information inside the database.

show dashboard

Figure 11 – show dashboard results

Some of the modules I ran were not in this tutorial.  From Figure 11 you can see all the modules used. Figure 12 is a continuation of the show dashboard command.  Here you can see the information that is captured in the database.  This also makes it easier for creating a report or exporting information.

Figure 12 – show dashboard summary

Exporting Data

We will use the reporting/list module to create a list of IP addresses to use in nmap.  This will tie in several things we’ve already covered.

• Search for modules
• Show options
• Schema command
• Set command

We will also use Nmap to scan for port 80.

search reporting

Figure 13 – search reporting

use report/list
show options

Figure 14 – report/list options

We will run the show schema and only show the truncated results so we can get the table schema.

show schema

Figure 15 – show schema

Next, use the set command to give recon-ng the file location.

set FILNAME /location/on/file/system

Figure 16 – set file location

Finally, run and let recon-ng generate the results. The screenshot is truncated so you can get an idea of what it looks like, your mileage may vary.

run

Figure 17 – Report Results

<<Truncation Occurs>>>

Figure 18 – Report Summary

Using export_iplist.txt as input for our Nmap scan.

• -iL input list filename
• -p 80 port to scan
• -Pn No Ping

nmap -iL export_iplist.txt -Pn -p 80

Figure 19 – Nmap port 80 scan

Create Report

This section will show you how to create an HTML report using the same data set.

use reporting/html
show options
set CREATOR Pentester
set COMPANY United Airlines

Figure 20 – report/html

Figure 21 – set options for report

We used the set command to add the creator and the customer properties for our report. Use the run command to execute the module.

run

Figure 22 – generate report

Not too exciting but we have our report waiting for us in the .recon-ng folder.

Figure 23 – Report location

Lets look at that file using a browser.

Figure 24 – File Browser

Figure 25 – HTML Report Example

The next set of figures will show the expanded results for the Summary, Domains, and Locations sections.

Figure 26 – Summary Section

Figure 27 – Domains Section

Figure 28 – Locations Section

The Contacts section we could have done a more with the information here.  One thing I like to do is us with this information is expand using the  https://pipl.com website. Using Pipl we could really dig into who any of the individuals are to create more effective spear phishing attacks or sales calls. Who are we kidding? We don’t do sales calls.

Figure 29 – Contacts Section

Look through the Vulnerabilities section. We haven’t even started a technical vulnerability assessment and we already have a place to start. OSINT for the win!

Figure 30 – Vulnerabilities Section

Figure 31 – Vulnerabilities Section 2

Conclusion

In this tutorial we covered Recon-ng.  It can be found at https://bitbucket.org/LaNMaSteR53/recon-ng.  I really enjoy working with this tool.  Just playing with it can give you a better understanding of other ways to gather information about your target.  It really becomes about bread crumbs. How deep can you dig into a company, email address, or person?

Areas we covered:

• Installation
• Adding API Keys
• Creating a Workspace
• Importing information into the database “ Grep and Awk commands”
• Using Modules
• Removing unwanted entries
• Exporting Data “ to use with nmap”
• Creating Reports

Creating a Workspace

The workspace is an area that will help keep your reconnaissance organized.  Each workspace has it’s own directory inside the hidden .recon-ng directory in the home directory.

First we will find an organization to recon and build our workspace around this company.  We will use HackerOne to get our company.

This is how Wikipedia describes HackerOne:

“HackerOne is a vulnerability coordination and bug bounty platform that connects businesses with cybersecurity researchers (aka, hackers). It is one of the first companies to embrace and utilize crowd-sourced security and hackers as linchpins of its business model, and is the largest cybersecurity firm of its kind.^[1]”

Even though we are only performing reconnaissance in a non-intrusive manner, we will use a company from HackerOne’s Directory.  Under the right conditions, this company has agreed to recon and scanning.  We will only be using recon-ng. Figure 1 shows the company we will use in the tutorial but feel free to select a different company from HackerOne or use any one that you are authorized to test against.

Figure 1: HackerOne Company

Figures 2 and 3 show the scope that is authorized for testing including eligible submissions and domains.

Figure 2: Eligible Items

Figure 3: Allowed Domains

workspaces -h shows us the different option we have a available to use (Figure 4).

Figure 4: Workspace -h

Next we will add our workspace using the following command (Figure 5)

workspaces add

Figure 5: Adding a Workspace

After this command you are automatically placed into your new workspace. workspaces list will show you the status of your workspaces.

Figure 6: List of workspaces

Next, we will add our company and our domain.  This will add information to the SQLite database. To add information into the database, we need to understand the schema, the layout of the tables. To look at the schema of the database run the following command (Figure 7)

show schema

Figure 7: Show Schema

There are thirteen different tables, we will view the schema of the tables we use in this tutorial.

 add companies

Running the add companies command will make the other columns available.   Press enter if you want to leave that column blank.

Figure 8: Add Company

Add the domain using the following command (Figure 9).

add domains

Figure 9: Add Domains

To verify that the domain was added successfully run the command shown in Figure 10.

show domains

Figure 10: List Domains

A simple way of thinking about adding to the tables is shown in the next Figure 11.

Figure 11: Table Visualization

Now that we’ve added data to the database and know how to ensure that data was manually inserted correctly lets move on to importing and exporting data.

Importing Data into the Database

We will uses theHarvester to gather information about United.com and import this into recon-ng’s database.

From Edge Security  http://www.edge-security.com/theharvester.php

“The objective of this program is to gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN computer database.

This tool is intended to help Penetration Testers in the early stages of the penetration test in order to understand the customer footprint on the Internet. It is also useful for anyone that wants to know what an attacker can see about their organization.”

If theHarvester isn’t already installed, i.e. you aren’t using Kali Linux, you can clone it from here: https://github.com/laramies/theHarvester

Figure 12: theHarvester

We called theHarvester to gather data on domain united.com using all the data sources listed in the help screen.  We directed the output to my recon-ng folder using the ‘>’ operator. The sample command we used follows:

./theHarvester.py -d united.com -b all > ~/recon-ng/harvester.txt

The file name is harvester.txt. This is an ugly file that well will parse through using a few linux utilities. Sample results are shown in the next figure.

Figure 13: Sample Results

The next step is to make this snippet and clean it up a bit with some Linux utilities. We will use grep and AWK to trim the tree.

Grep and AWK

grep is a command-line utility for searching plain-text data sets for lines matching a regular expression.

This is by no means the perfect way.  This is just one of many to get the results you need. Using grep, we will create a list of email addresses from harvester.txt file. (Figure 14)

grep @united.com harvester.txt > united_emails.txt

Figure 14: grep command

If you are interested in the file contents use the cat command to view file the contents in the terminal

cat united_emails.txt

Figure 15: cat results

Next, we will create a list of hosts for import from theHarvester results. (Figure 16)

grep ":" harvester.txt

Figure 16: grep host

Grep will also help create the virtual host list.  Also take note that since “united.com” is the only domain in scope, it becomes part of the command.

grep ":" harvester.txt | grep united.com

The pattern that we wanted to match was “=” and I didn’t want to count the lines after the pattern so I chose to use 200 as my line count after the pattern, as shown in Figure 17.

Figure 17 grep for Virtual Hosts

This command was a little harder to figure out. The pattern that we wanted to match was “=” and I didn’t want to count the lines after the pattern so I chose to use 200 as my line count after the pattern.

grep -A200 "=" harvester.txt | grep united.com > virtual_hosts.txt

It is time to import our information into recon-ng.

Using the show modules command, we get a list of modules broken down by categories. We will use import/list module from the Import category.

show modules

Figure 18: Import Modules

The “show info” command shows the options to use and the table and columns that will be needed for the import.

show info

Figure 19: Show Info

To find the column and table, we will use the “show schema” command.  This will give use a list of the Tables and the different columns in each.

show schema

Figure 20: Show Schema

To import email addresses, we will the  “contacts” table and the email column. Our file name will be the united_email.txt file we created using theHarvester. The “set” statement, sets the variables for the import. The “run” command executes the module.

set TABLE contacts
set COLUMN email
set FILENAME united_emails.txt
run

Figure 21: Email Import

The “show contacts” command show the data inside the “Contacts Table”. This is a second verification that the data imported correctly.

Figure 22: Show Contacts

Part 3: Usage and Reporting

Intro

Recon-ng is a Open Source Reconnaissance framework written in Python.  This SQLite database driven tool incorporates Python modules and API Keys to allows itself to be a conduit for many tools ranging from The Harvester to Metasploit.  It is an awesome standalone reconnaissance tool in its own right. As a side note we all totally have a geeky nerd crush on LaNMaSterR53.

This part of the series will take a look at installation, adding API Keys. Later we will show you how to create a Workspace, importing data into the database, and export data for the use with other tools.

For our targets of reconnaissance, we will use HackerOne’s directory of companies.  This is not our way of saying, “Go out and hack these companies” but our way of doing safe recon and provide continuous screenshots.  That will be easy to follow.  This is also our way of introducing you to HackerOne and the Bug Bounty community if you are not already familiar with it.

Getting Started

While most penetration testers will be running this out of Kali Linux the prerequisites (git and pip) may need to be installed before you start. Fortunately, this is easy on most linux flavors and requires just a few simple commands:

sudo apt-get update
sudo apt-get install git

sudo apt-get install python-pip python-dev build-essential
sudo pip install --upgrade pip
sudo pip install --upgrade virtualenv

Next clone Recon-ng from bitbucket (Figure 1). In this tutorial we clone to the Home directory but feel free to use whatever directory structure works for you.

git clone https://LaNMaSteR53@bitbucket.org/LaNMaSteR53/recon-ng.git

Figure 1: git install

Next, change directory into the newly created recon-ng and list the contents (Figure 2).

cd recon-ng
ls

Figure 2: recon-ng contents

We will use the REQUIREMENTS file to finish installing the dependencies for recon-ng.

pip install -r REQUIREMENTS

At this point the installation is almost ready to use, we will go over a little bit of information now while you’re still paying attention and then get recon-ng running and the API keys loaded.

The installation of recon-ng also created a .recon-ng a hidden directory inside your home directory.  This directory is empty.  This is where your key.db and your workspaces will be created. After logging into recon-ng for the first time, a directory and the keys.db is entered in the hidden .recon-ng directory (Figure 3).

Figure 3: .recon-ng directory

To run recon-ng, go to the folder where you ran the “git clone” command. This is where the magic happens.

cd recon-ng 
./recon-ng

Don’t worry if you get the “_api key not set error” (Figure 4).  We have not added any API keys yet.

Figure 4: Initial Start

From our screen, we can see that there are 76 Recon modules, 8 Reporting modules, 2 Import modules, 2 Exploitation modules, and 2 Discovery modules.  We are also using the “default” workspace. (Figure 5)

Figure 5: Recon-ng start screen

Close recon-ng and lets look at the modules and the underlying code. (Figure 6)

cd modules
cd recon
ls

Figure 6: Module Directory

If we go inside the module directory and inside a module, we can see the Python script that does all the magic. (Figure 7)

Figure 7: Module Content

Adding API Keys

As I said in the introduction, this is a database driven tool.  Now it’s time to add information into the database.

The API keys are used by the modules to gather information for the SQLite database.  Some of the API keys are free but some can be expensive.  I will keep this tutorial to the free API keys that are available.

After going back into the recon-ng directory and typing “./recon-ng”, you will be inside the recon-ng console. (Figure 8)

keys list

Figure 8: Keys List

The following command is an example of adding the shodan_api key. (Bottom of Figure 8, Look close it is there)

keys add shodan_api <paste key here>

API Keys Signup URLs

Signing up for the API keys is the least fun and most time consuming part of the setup. Showing each signup would be lethally boring so here are the list of URLs. All links open in a new window because we are thoughtful like that.

Google API – https://console.developers.google.com/apis/library
Bing API – https://msdn.microsoft.com/en-us/library/bing-ads-getting-started.aspx
Facebook API – https://developers.facebook.com/docs/apis-and-sdks
Instragram API – https://www.programmableweb.com/api/instagram
Linkedin API – https://developer.linkedin.com/docs/rest-api
Shodan API – https://developer.shodan.io/
Twitter API – https://apps.twitter.com/

Part 2: Workspaces and Importing Data