Gathers All Legitimate IP Addresses From All Textual content Information From A Listing, And Checks Them In opposition to Whois Database, TOR Relays And Location

This script scans each file from a given folder recursively, extracts each IPv4 and IPv6 tackle, filters out the general public IP addresses after which queries these IPs in opposition to TOR relays, WhoIs service and Location service. It outputs 4 information: a CSV file and a JSON file with all particulars of the IP addresses, a CSV file containing an index of each discovered IP tackle and a KML file with all of the places gathered.

• Parses any variety of information in a single run
• Will also be run for a single enter file
• Extracts all distinctive legitimate IPv4 and IPv6 addresses (appropriately evaluating two addresses, even when they’re written otherwise)
• Collects publicly out there Who Is data for all public IP addresses
• Queries all public IP addresses in opposition to TOR relays
• Collects geographical location data for all public IP addresses
• Could be run with command line parameters in non-interactive mode for simple integration in different scripts
• Could be run with out parameters and the enter values might be supplied interactively
• Permits consumer to pick out what steps to carry out or exclude
• Verbosely outputs data to console at each step that it performs
• Exports all IP addresses particulars in each a CSV file and a JSON file
• Exports a KML file that may be simply loaded on prime of a map
• Retains an index of all discovered IP addresses in a separate CSV file for simple traceback

Pre-requisites:

Python 3.9.x

Set up

To put in dependencies, run:

pip set up -r necessities.txt

Compile to executable

pip set up pyinstaller

pyinstaller --onefile osips.py

Utilization

python osips.py [-h] [-F INPUTFOLDER] [-f INPUTFILE] [-p INPUTFILESPATTERN] [-o OUTPUTFILESNAME] [-t {y,n,Y,N}] [-w {y,n,Y,N}] [-l {y,n,Y,N}] [-locationEndpoint LOCATIONENDPOINT]

Interactively

python osips.py

Non-interactively

python osips.py --checkTor Y --checkWhoIs Y --checkLocations Y --inputFolder testFolder/ --inputFilesPattern * --outputFilesName check

Assist

python osips.py -h

Choices

• -h, --help Present the assistance message and exit
• -F INPUTFOLDER, --inputFolder INPUTFOLDER A folder containing information with IPs (log information, electronic mail information, textual content information and many others). The IPs might be IPv4 and/or IPv6 and might be positioned wherever within the information. The script will parse each file within the folder and can scan for each IP
• -f INPUTFILE, --inputFile INPUTFILE Use this feature to scan just one file as an alternative of a folder with information. The file might be any textual content file that incorporates IP addresses (log information, electronic mail information, textual content information and many others.)
• -p INPUTFILESPATTERN, --inputFilesPattern INPUTFILESPATTERN Enter the sample to make use of for the information within the enter folder. Instance: * for all information, *.txt for textual content information.
• -o OUTPUTFILESNAME, --outputFilesName OUTPUTFILESNAME The identify for the output information, WITHOUT EXTENSION. 4 completely different information can be created with this identify: a CSV file storing the IP particulars, an index file storing the IP indexes, a JSON file storing the main points and a KML file storing the places.
• -t {y,n,Y,N}, --checkTor {y,n,Y,N} Specify if you wish to examine each public IP tackle in opposition to the TOR relays IP addresses
• -w {y,n,Y,N}, --checkWhoIs {y,n,Y,N} Specify if you wish to extract WhoIs data for each public IP tackle
• -l {y,n,Y,N}, --checkLocations {y,n,Y,N} Specify if you wish to geocode each public IP tackle and extract location information utilizing https://reallyfreegeoip.org
• -locationEndpoint LOCATIONENDPOINT Default REST endpoint for location is “https://reallyfreegeoip.org/json/“. You may specify a diffrent finish level, includding placeholder. The end result should be in JSON and should embody the next fields: country_name, region_code, region_name, metropolis, zip_code, time_zone, latitude, longitude, metro_code

Remarks

• The software extracts IPv4 addresses solely in dot decimal notation (xxx.xxx.xxx.xxx). It doesn’t extract IPv4 addresses in decimal, octal or hexadecimal notation due to the excessive price of false positives.
• Relying on the variety of public IP addresses discovered, querying the web providers might take a very long time. If you’re coping with plenty of IP addresses, it is suggested to first run the software with out querying any of the providers and simply to extract the IP addresses. After seeing the extraction end result, you possibly can calculate an estimate of the overall run time of the software if the providers are queried, and finally select to question just one or two of the providers.
• TOR relays are checked for the time being of operating the software. As a result of these IPs are continually altering, this data is probably not very correct. If a date will also be recognized for an IP, then it needs to be checked in opposition to ExoneraTor at this tackle: https://metrics.torproject.org/exonerator.html
• Who Is data in addition to the placement data is gathered dwell from publicly out there providers, so the IP addresses are despatched to those providers. There are downloadable databases that may let you accumulate this data regionally, however these downloads are usually not free and would wish fixed updates. This software isn’t constructed to work with native databases.
• The default service used for location gathering is https://reallyfreegeoip.org. This can be a free service with no limitations, nevertheless it is probably not very correct. You may change this service with one other, by utilizing the -locationEndpoint parameter of the script. You may present one other service that takes an IP tackle as a parameter and returns a JSON response. Use the as a placemark for the IP tackle location. Examples: https://ipapi.co//json, https://api.ipstack.com/?access_key=ACCESS_KEY, https://freeapi.robtex.com/ipquery/. Please be aware that a few of these different REST endpoints might not return the required fields and thus will break the script’s logic. On this case, the tactic named RequestInfoFromRestEndpoint needs to be modified accordingly.

Acknowledgements