Toolsmith #126: Adversary hunting with SOF-ELK

As we celebrate Independence Day, I'm reminded that we honor what was, of course, an armed conflict. Today's realities, when we think about conflict, are quite different than the days of lining troops up across the field from each other, loading muskets, and flinging balls of lead into the fray.
We live in a world of asymmetrical battles, often conflicts that aren't always obvious in purpose and intent, and likely fought on multiple fronts. For one of the best reads on the topic, take the well spent time to read TJ O'Connor's The Jester Dynamic: A Lesson in Asymmetric Unmanaged Cyber Warfare. If you're reading this post, it's highly likely that your front is that of 1s and 0s, either as a blue team defender, or as a red team attacker. I live in this world every day of my life as a blue teamer at Microsoft, and as a joint forces cyber network operator. We are faced, each day, with overwhelming, excessive amounts of data, of varying quality, where the answers to questions are likely hidden, but available to those who can dig deeply enough.
New platforms continue to emerge to help us in this cause. At Microsoft we have a variety of platforms that make the process easier for us, but no less arduous, to dig through the data, and the commercial sector continues to expand its offerings. For those with limited budgets and resources, but a strong drive for discovery, that have been outstanding offerings as well. Security Onion has been forefront for years, and is under constant development and improvement in the care of Doug Burks.
Another emerging platform, to be discussed here, is SOF-ELK, part of the SANS Forensics community, created by SANS FOR572, Advanced Network Forensics and Analysis author and instructor Phil Hagen. Count SOF-ELK in the NFAT family for sure, a strong player in the Network Forensic Analysis Tool category.
SOF-ELK has a great README, don't be that person, read it. It's everything you need to get started, in one place. What!? :-)
Better yet, you can download a fully realized VM with almost no configuration requirements, so you can hit the ground running. I ran my SOF-ELK instance with VMWare Workstation 12 Pro and no issues other than needing to temporarily disable Device Guard and Credential Guard on Windows 10.
SOF-ELK offers some good test data to get you started with right out of the gate, in /home/elk_user/exercise_source_logs, including Syslog from a firewall, router, converted Windows events, a Squid proxy, and a server named muse. You can drop these on your SOF-ELK server in the /logstash/syslog/ ingestion point for syslog-formatted data. Additionally, utilize /logstash/nfarch/ for archived NetFlow output, /logstash/httpd/ for Apache logs, /logstash/passivedns/ for logs from the passivedns utility, /logstash/plaso/ for log2timeline, and  /logstash/bro/ for, yeah, you guessed it.
I mixed things up a bit and added my own Apache logs for the month of May to /logstash/httpd/. The muse log set in the exercise offering also included a DNS log (named_log), for grins I threw that in the /logstash/syslog/ as well just to see how it would play.
Run down a few data rabbit holes with me, I swear I can linger for hours on end once I latch on to something to chase. We'll begin with a couple of highlights from my Apache logs. The SOF-ELK VM comes with three pre-configured dashboards including Syslog, NetFlow, and HTTPD. You can learn more in the start page for the SOF-ELK UI, my instance is http://192.168.50.110:5601/app/kibana. There are three panels, or blocks, for each dashboard's details, at the bottom of the UI. I drilled through to the HTTPD Log Dashboard for this experiment, and immediately reset the time period for analysis (click the time marker in the upper right hand part of the UI). It defaults to the last 15 minutes, if you're reviewing older data it won't show until you adjust to match your time stamps. My data is from the month of May so I selected an absolute window from the beginning of May to its end. You can also select quick or relative time options, it's great to get comfortable here quickly and early. The resulting opening visualizations for me made me very happy, as seen in Figure 1.

Figure 1: HTTPD Log Dashboard

Nice! An event count summary, source ASNs by count (you can immediately see where I scanned myself from work), a fantastic Access Source map, a records graph by HTTP verbs, and one by response codes.
The beauty of these SOF-ELK dashboards is that they're immediately interactive and allow you to drill right in to interesting data points. The holisticinfosec.org website is intentionally flat and includes no active PHP or dynamic content. As a result, my favorite response code as a web application security tester, the 500 error, is notably missing. But, in both the timeline graphs we note a big traffic spike on 8 MAY 2017, which correlates nicely with my above mention scan from work, as noted in the ASN hit count, and seen here in Figure 2.

Figure 2: Traffic spike from scan

This visualizes well but isn't really all that interesting or uncommon, particularly given that I know I personally ran the scan, and scans from the Intarwebs are dime a dozen. What did jump out for me though, as seen back in Figure 1, was the presence of four PUT requests. That's usually a "bad thing" where some @$$h@t is trying to drop something on my server. Let's drill in a bit, shall we? After clicking the graph line with the four PUT requests, I quickly learned that two requests came from 204.12.194.234 AS32097: WholeSale Internet in Kansas City, MO and two came from 119.23.233.9 AS37963: Hangzhou Alibaba Advertising in Hangzhou, China. This is well represented in the HTTPD Access Source panel map (Figure 3).

Figure 3: Access Source

The PUT request from each included a txt file attempt, specifically dbhvf99151.txt and htjfx99555.txt, both were rejected, redirected (302), and sent to my landing page (200).
Research on the IPs found that 119.23.233.9 was on the "real time suspected malware list as detected by InterServer's intrusion systems" as seen 22 MAY, and 204.12.194.234 was found twice in the AbuseIPDB, flagged on 18 MAY 2017 for Cknife Webshell Detected. Now we're talking. It's common to attempt a remote file include attack or a PUT, with what is a web shell. I opened up SOF-ELK on that IP address and found eight total hits in my logs, all looking for common PHP opportunities with the likes of GET and POST for /plus/mytag_js.php, noted in PHP injection attack attempts.
SOF-ELK made it incredibly easy to hunt down these details, as seen in Figure 4 from the HTTPD Discovery panel.

Figure 4: Discovery

That's a groovy little hunting trip through HTTPD logs, but how about a bit of Syslog? I spotted I likely oddity that could be correlated across a number of the exercise logs, we'll see if the correlation is real. You'll notice tabs at the top of your SOF-ELK UI, we'll use Discover for this experiment. I started from the Syslog Dashboard with my time range set broadly on the last two months. 7606 records presented themselves, sliced neatly by hosts and programs, as seen in Figure 5.

Figure 5: Syslog Dashboard

Squid proxy logs showed the predominance of host entries (6778 or 57.95% of 11,696 to be specific), so I started there. Don' laugh, but I'll often do keyword queries just to see what comes up, sometimes you land a pointer to a good rabbit hole. Within the body of 6778 proxy events, I searched malware. Two hits came back for GET request via a JS redirector to bleepingcomputer.com for your basic how-to based on "random websites opening in Chrome". Ruh-roh.

Figure 6: Malware keyword

More importantly, we have an IP address to pivot on: 10.3.59.53. A search of that IP across the same 6778 Squid logs yielded 3896 entries specific to this IP, and lots to be curious about:

• datingukrainewomen.com 
• anastasiadate.com
• YouTube videos for hair loss
• crowdscience.com for "random pop-ups driving me nuts"

Do I need to build this user profile out for you, or are you with me? Proxy logs tell us so much, and are deeply worthy of your blue team efforts to collect and review.
I jumped over to the named_log from the muse host to see what else might reveal itself. Here's where I jumped to Discover, the Splunk-like query functionality inherent to SOF-ELK (and ELK implemetations). I did reductive query to see what other oddities might surface: 10.3.59.53 AND dns_query: (*.co.uk OR *.de OR *.eu OR *.info OR *.cc OR *.online OR *.website). I used these TLDs based on the premise that bots using Domain Generation Algorithms (DGA) will often use the TLDs. See The DGA of PadCrypt to learn more, as well as ISC Diary handler John Bambanek's OSINT logic. The query results were quite satisfying, 29 hits, including a number of clearly randomly generated domains. Those that were most interesting all included the .cc TLD, so I zoomed in further. Down to five hits with 10.3.59.53 AND dns_query: *.cc, as seen in Figure 7.

Figure 7:. CC TLD hits

Oh man, not good. I had a hunch now, and went back to the proxy logs with 10.3.59.53 AND squid_request:*.exe. And there you have it, ladies and gentlemen, hunch rewarded (Figure 8).

Figure 8: taxdocs.exe

It taxdocs.exe isn't malware, I'm a monkey's uncle. Unfortunately, I could find no online references to these .cc domains or the .exe sample or URL, but you get the point. Given that it's exercise data, Phil may have generated it to entice to dig deeper.
When we think about the IOC patterns for Petya, a hunt like this is pretty revealing. Petya's "initial infection appears to involve a software supply-chain threat involving the Ukrainian company M.E.Doc, which develops tax accounting software, MEDoc". This is not Petya (as far as I know) specifically but we see pattern similarities for sure, one can learn a great deal about the sheep and the wolves. Be the sheepdog!
Few tools better in the free and open source arsenal to help you train and enhance your inner digital sheepdog than SOF-ELK. "I'm a sheepdog. I live to protect the flock and confront the wolf." ~ LTC Dave Grossman, from On Combat.

Believe it or not, there's a ton more you can do with SOF-ELK, consider this a primer and a motivator.
I LOVE SOF-ELK. Phil, well done, thank you. Readers rejoice, this is really one of my favorites for toolsmith, hands down, out of the now 126 unique tools discussed over more than ten years. Download the VM, and get to work herding. :-)
Cheers...until next time.

Toolsmith - GSE Edition: Scapy vs CozyDuke

In continuation of observations from my GIAC Security Expert re-certification process, I'll focus here on a GCIA-centric topic: Scapy. Scapy is essential to the packet analyst skill set on so many levels. For your convenience, the Packetrix VM comes preconfigured with Scapy and Snort, so you're ready to go out of the gate if you'd like to follow along for a quick introduction.
Scapy is "a powerful interactive packet manipulation program. It is able to forge or decode packets of a wide number of protocols, send them on the wire, capture them, match requests and replies, and much more." This includes the ability to handle most tasks such as scanning, tracerouting, probing, unit tests, attacks or network discovery, thus replacing functionality expected from hping, 85% of nmap, arpspoof, tcpdump, and others.
If you'd really like to dig in, grab TJ O'Connor's  Violent Python: A Cookbook for Hackers, Forensic Analysts, Penetration Testers and Security Engineers (you should already have it), as first discussed here in January 2013. TJ loves him some Scapy: Detecting and Responding to Data Link Layer Attacks is another reference. :-)
You can also familiarize yourself with Scapy's syntax in short order with the SANS Scapy Cheat Sheet as well.
Judy Novak's SANS GIAC Certified Intrusion Analyst Day 5 content offers a nice set of walk-throughs using Scapy, and given that it is copyrighted and private material, I won't share them here, but will follow a similar path so you have something to play along with at home. We'll use a real-world APT scenario given recent and unprecedented Russian meddling in American politics. According to SC Magazine, "Russian government hackers apparently broke into the Democratic National Committee (DNC) computer systems" in infiltrations believed to be the work of two different Russian groups, namely Cozy Bear/ CozyDuke/APT 29 and Fancy Bear/Sofacy/APT 28, working separately. As is often the case, ironically and consistently, one the best overviews of CozyDuke behaviors comes via Kaspersky's Securelist. This article is cited as the reference in a number of Emerging Threats Snort/Suricata rules for CozyDuke. Among them, 2020962 - ET TROJAN CozyDuke APT HTTP Checkin, found in the trojan.rules file, serves as a fine exemplar.
I took serious liberties with the principles of these rules and oversimplified things significantly with a rule as added to my local.rules file on my Packetrix VM. I then took a few quick steps with Scapy to ensure that the rule would fire as expected. Of the IOCs derived from the Securelist article, we know a few things that, if built into a PCAP with Scapy, should cause the rule to fire when the PCAP is read via Snort.

1. CozyDuke client to C2 calls were over HTTP
2. Requests for C2 often included a .php reference, URLs included the likes of /ajax/index.php
3. 209.200.83.43 was one of the C2 IPs, can be used as an example destination IP address

The resulting simpleton Snort rule appears in Figure 1.

Figure 1: Simple rule

To quickly craft a PCAP to trigger this rule, at a bash prompt, I ran scapy, followed by syn = IP(src="10.0.2.15", dst="209.200.83.43")/TCP(sport=1337, dport=80, flags="S")/"GET /ajax/index.php HTTP/1.1", then wrote the results out with wrpcap("/tmp/CozyDukeC2GET.pcap", syn), as seen in Figure 2.

Figure 2: Simple Scapy

Then a quick run of the resulting file through Snort with snort -A console -q -K none -r /tmp/CozyDukeC2GET.pcap -c ../etc/snort.conf, and we have a hit as seen in Figure 3.

Figure 3: Simple result

Scapy is ridiculously powerful and is given no justice here, hopefully just enough information to entice you to explore further. With just the principles established here, you can see the likes of options to craft and manipulate with ls(TCP) and ls(IP).
 

Figure 4: ls()

If you're studying for the likes of GCIA or just looking to improve your understanding of TCP/IP and NSM, no better way to do so than with Scapy.
Cheers...until next time.

Tool review: NetworkMiner Professional 1.2

I've been slow in undertaking this review as NetworkMiner's Erik Hjelmvik sent me NetworkMiner Professional 1.1 when it was released and 1.2 is now available.

Seeing Richard Bejtlich's discussion of Pro 1.2 has served to get me off the schnide and is helpful as I will point you to his post as an ideal primer while I go into to a bit deeper detail as to some of NetworkMiner's power as well as what distinguishes Professional from the free edition.

I covered NetworkMiner in toolsmith in August 2008 back when it was version 0.84. Erik has accomplished all of his goals for improvement as identified in the article including reporting, faster parsing of large PCAP files (.735 MB/s at the command-line),  more protocols implemented, and PIPI (Port Independent Protocol Identification). NetworkMiner Professional 1.2 incorporates all of the above.

To exemplify NetworkMiner Professional's PIPI capabilities, I changed my lab web server port to 6667, then set NetworkMiner to grab a live capture while browsing to the reconfigured server.

Note: you need to Run as Administrator to grab the interface on Windows 7.

Sure, it's more likely that someone would be more likely to hide evil traffic over port 80 but you get the point. As Richard said, "PIPI has many security implications for discovery and (preferably) denial of covert channels, back doors, and other policy-violating channels."

Note as seen in Figure 1 that NetworkMiner Professional clearly differentiates HTTP traffic regardless of the fact that it traversed port 6667.

Figure 1

I was a bit surprised to note that the Hosts view as seen in Figure 1 did not identify that any data was pushed as cleartext although it unequivocally identified the admin/password combination I sent in both the Cleartext view and the Credentials view.

I used an 18.8MB PCAP from the Xplico sample set as it includes a plethora of protocols and carve-able content with which to test NetworkMiner Professional.

Exporting results to CSV for reporting is as easy as File --> Export to CSV and selecting output of your choosing. As seen in Figure 2 I opted for Messages as NetworkMiner Professional cleanly carved out an MSN to Yahoo email session (HTTPS, anyone?).

Figure 2

Geo IP localization is a real standout too. You'll see it in play as you explore host details in Hosts view as seen in Figure 3.

Figure 3

You may find host coloring useful too should you wish to tag hosts for easy identification later as seen in Figure 4.

Figure 4

Finally, I am most excited about NetworkMinerCLI for command-line scripting support. 

I ran a PCAP taken from a VM infected with Trojan-Downloader.Win32.Banload.MC through NetworkMinerCLI and was amply rewarded for my efforts...right after I excluded the output directory from AV detection.

Figure 5 shows the command executed at the prompt coupled with the resulting assembled files and CSVs populated to the output directory as seen via Windows Explorer.

Figure 5

The assembled files included all the malicious binaries disguised as JPGs as downloaded from the evil server. File carving network forensic analysis juju with easy CLI scripting. Bonus!

In closing, NetworkMiner Professional 1.2 is a mature, highly useful tool and well worthy of consideration for purchase by investigators and analysts tasked with NFAT activity. 

I'm glad to provide further feedback via email and recommend you reach out to Erik as well via info [at] netresec.com if you have questions.

toolsmith: Xplico

Those of you who make use of Network Forensic Analysis tools (NFAT) such as NetworkMiner or Netwitness Investigator will certainly appreciate Xplico.
June's toolsmith covers Xplico, a project released under GPL that decodes packet captures (PCAP), extracting the likes of email content (POP, IMAP, and SMTP protocols), all HTTP content, VoIP calls (SIP), IM chats, FTP, TFTP, and many others.
If you'd like a breakdown on the protocols you can grapple with check out the Xplico status page.
You can imagine how useful Xplico might be for policy enforcement (spot the pr0n), malware detection (spot the Renocide), or shredding IM traffic (spot the data leak).
Experimenting with Xplico is also a great chance to check out Pcapr, Web 2.0 for packets. ;-)
Xplico inlcudes a highly functional Web UI with great case and session management as seen in Figure 1.


Figure 1

With a resurgence of discussion of APT given the recent bad news for RSA, as well as all the FUD spawned by Sony's endless woes, I thought a quick dissection of an Aurora attack PCAP would be worth the price of admission for you (yep, free) as seen in Figure 2.


Figure 2

You'll note the beginning of a JavaScript snippet that has only the worst of intentions for your favorite version of Internet Explorer as tucked in an HTML page.
Copy all that mayhem to a text file (in a sandbox, please), then submit it to VirusTotal (already done for you here) and you'll note 26 of 42 detections including Exploit:JS/Elecom.D.
Want to carve off just that transaction? Select the pcap under Info from the Site page under the Web menu selction as seen in Figure 3.


Figure 3

Voila!
Ping me via russ at holisticinfosec dot org if you'd like a copy of the above mentioned Aurora PCAPs.

Also, stand by for more on APT detection in outbound traffic in the next day or two.

Your gonna like this tool, I guarantee it.
Check out the article here and Xplico here .

Cheers.