Jack B.

In this blog, I will be providing you a free and open source solution for researching and monitoring threats and or threat actors. The opportunity here is potentially a cost savings of between $500,000 and upwards of $1.5 million on “Intelligence” gathering platforms. The savings can be invested in people/talent as opposed to commercial software.

Note, the blog example herein, is for educational purposes. If you would like to know more about my personal experience on this subject, reach out on LinkedIn. Before implementing mass collection of data, I recommend consulting with legal counsel. Also, when crawling sites, it is possible to come across illegal media. Be sure to read over Yacy's ability to blacklist sites and prevent the downloading of media content.

Yacy is a free search engine that anyone can use. It is based on a decentralized peer to peer environment with privacy and "security" in mind. The tool is purposed for freeing the Internet from censorship. The tool can be used to research and monitor threats and threat agents. I will not be covering how to install Yacy as it is well documented on the Yacy.net site.

Let's begin with a scenario. Imagine being an organization which received a report that a potential “threat actor” on 4chan.org has disclosed malicious intent toward the business. Wouldn’t it be nice to monitor 4chan.org without having to go through each page manually? Sure, let me show you how it can be done.

I am using Ubuntu Linux 16.04 LTS. The first thing I do is go to 4chan.org and get an understanding of the web application’s domains and sub-domain structure. I am interested in http://www.4chan.org. Notice, how the site is broken down? I need to acquire a list of the top domain and sub-domains. From the terminal do:

# Install required tool is xclip $ sudo apt-get install xclip

# Create a list of domains and sub-domains and copy to clipboard $ curl -s -L http://boards.4chan.org | egrep -o "\bboards.4chan.org/[a-zA-Z0-9]{1,3}\b" | sort -u | sed 's/[[:blank:]]\+/\n/g' | xclip -selection clipboard Next, initiate Yacy from terminal

Then visit: http://localhost:8090

Select “Administration” on the top right hand corner

From “Production” select “Advanced Crawler”

Push “Ctrl+V” within the Crawl Job Start Point, be sure the last line is not an empty space

Notice the other options as you scroll down

Select, “ Restrict to Start Domains ”

I recommend keeping “ No Doubles ”

Please explore the “ Document Cache ” information button

Remove check-mark “ Index Media ”

It is critical when scanning sites such as 4chan to remove the check-mark for "index media." The reason is some media on sites like 4chan.org could be illegal to download.

Select “ Start New Crawl Job ”

The Yacy Engine will commence crawling the domains as requested by the user. It will use the power of a Peer to Peer environment. It is possible to automate or schedule these crawls periodically. Allow the crawl to occur for a few hours.

Select “Search Interface” button, top-right corner

Now you should be able to search your current crawled text from 4chan.org.

Select the “More Options” button for advanced search options.

An example search:

boards.4chan.org nfl /near /date inurl: site:boards.4chan.org

Select “Sort by Date”

Yacy is a very powerful tool! Scaling and automating Yacy for a Cyber Threat Intelligence program is possible. One can save RSS feeds and alert on keywords. For example, I use it to monitor Cybersecurity news and when an article contains something in my company inventory, I receive an alert. My objective was to expose you to this tool and to help you save money. Remember, consult with legal before meeting an objective that requires the mass collection of data. Reach out if you have any questions!

Yacy, Open Source Search Engine - http://yacy.net/en/Tutorials.html

Jack B.

United States

500+ connections

Experience

Trace3

-

-

-

-

Volunteer Experience

Advisor

The National Child Protection Task Force

Team member

https://twimcf.org/

Cyber and Threat Intelligence Advisor

Flanders Fields

View Jack’s full profile

Explore collaborative articles

Others named Jack B. in United States

Jack B

Jack B

Jack B

Jack O'Neill

Jack B.

United States 500+ connections

See your mutual connections View mutual connections with Jack Sign in Welcome back Email or phone Password Show Forgot password? Sign in or New to LinkedIn? Join now or New to LinkedIn? Join now

Experience

Trace3

-

-

-

-

Volunteer Experience

Advisor

The National Child Protection Task Force

Team member

https://twimcf.org/

Cyber and Threat Intelligence Advisor

Flanders Fields

View Jack’s full profile

Explore collaborative articles

Others named Jack B. in United States

Jack B

Jack B

Jack B

Jack O'Neill

United States

500+ connections