At HackEire 2011, as I've previously mentioned, I was pretty ambitious with the network design. This had some interesting consequences, mostly good but yes, the odd one did cause a performance issue on the day :)
Anyway, one of the things that worked quite well was the NSM (Network Security Monitor) that I implemented. So for me, to put it simply, NSM is essentially something that makes an IDS useful as it gives the analyst a better view into their network as it provides the tools for the analyst to analyse their network traffic in an improved fashion so he/she may understand it better and so do a better job easier!!! This is what I wanted to do, understand the goings-on in the HackEire network better.
I won't talk about why this is as you're probably better following @taosecurity for a much better explanation of why a NSM rocks (or follow @dougburks, i.e. Mr SecurityOnion, for that matter).
As the
Google Code Page for Security Onion says -
Security Onion is a Linux distro for IDS (Intrusion Detection) and NSM (Network Security Monitoring). It's based on Xubuntu 10.04 and contains Snort, Suricata, Sguil, Squert, argus, Xplico, tcpreplay, scapy, hping, and many other security tools.
So as you can seem deciding where to start with
Security Onion is a difficult place as there's so much useful and cool stuff within it and it's definitely easier to configure than many commercial solutions :) Ever try figuring out how to install Sguil or Barnyard by yourself? So how did I configure Security Onion at HackEire? Well, consider this to be the only network diagram that you'll ever see of HackEire -
To configure this set-up, I manually edited the /etc/network/interfaces file and played with the brctl utilities until I had the set-up as I wanted. I didn't use Network Manager at all but maybe I should have for simplicity, ah well :)
Set-Up
There's really no point in me talking about this much when Doug's already discussed it so well here on the PaulDotCom
website.
The recommended installation steps are
here.
All I will add is that it's really simple :)
In this blog, I refer to the command-line but it's really easy to do configure a lot of things through the desktop interface as this screenshot shows that everything is there -
Sguil & Squert
Sguil and
Squert are two things that truly turn Security Onion from an IDS into a NSM and with these you can really make sense of what's going on in your network. Unfortunately, I've lost the screenshots from the day :( so here's some vanilla-type stuff -
Sguil has its own GUI that enables the analyst to interpret the alerts, view the session data or the raw packets themselves. I've generated the alerts below by accessing httpo://testmyids.com with 'curl' -
OVERALL (including the raw packet data in the bottom righ-hand corner)
RULE MATCHED
Squert is a web-based tool, launched through the browser (Firefox by default on Security Onion) that is used to query and view event data stored in a Sguil database.
Sguil and Squert both put a
context on the data that the analyst is viewing.
HTTPRY
So one of the cool things that Security Onion comes with is '
httpry', a packet sniffer that shows http traffic. As all of the HackEire competitors were browsing through the IDS device when attacking the Bhratach network, all their http traffic was being logged (unbeknownst to them :) ).
Below is a snippet of the 'ps' output showing httpry configuration residing in /etc/nsm/ids-br0/http_agent.conf with the exclude file in the same directory and the logs in /nsm/sensor_data_ids-br0/httpry/$date.log.
sguil 1436 0.0 0.2 6500 4392 ? S 16:58 0:00 httpry -f timestamp,source-ip,source-port,dest-ip,dest-port,method,host,request-uri,referer,user-agent -i br0 -o /nsm/sensor_data/ids-br0/httpry/2011-12-22.log -u sguil
root 1459 0.0 0.1 5512 2224 ? S 16:58 0:00 tclsh /usr/local/bin/httpry_agent.tcl -c /etc/nsm/ids-br0/httpry_agent.conf -e /etc/nsm/ids-br0/httpry_agent.exclude -f /nsm/sensor_data/ids-br0/httpry/2011-12-22.log
This is the standard configuration for Security Onion -
- Log files from the Sensor under /nsm/sensor_data/NAME-OF-SENSOR
- Configuration in /etc/nsm/NAME-OF-SENSOR
where NAME-OF-SENSOR is 'ids-interface' where interface is the monitored interface so in my case you'll see
ids-br0, where
br0 was the bridged interface combining eth0 and eth1.
There was almost a GB of data on the day so it's quite hard to analyse (with the limited time that I have), however, here's a few things that we saw -
There were two log files on the day due to a forced restart of the VM.
Using some shell fu, we see that there's 405434 lines/entries in httpry.log and of that total, 118314 contain dirbuster -
$ wc -l httpry.log && perl -ne 'if (m/dirbuster/i) {print $_}' httpry.log | wc -l
405434 httpry.log
118314
So roughly 29% of the http traffic through the IDS device was using DirBuster to attack a web-server that was very well locked down. This web-server actually had no none web vulnerabilities, hosted all of the forensic/packet/general challenges and was best attacked through pivoting off another box.
Sample log -
2011-11-23 14:16:34 10.20.5.189 59318 192.168.10.101 80 HEAD 192.168.10.101 / - DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)
$ awk '{print $3 "\t" $5}' dirbuster.log | uniq
10.20.5.189 192.168.10.101
10.20.5.164 192.168.10.101
So of the 36 people there on the day, 2 felt the need to bombard the network with DirBuster :)
As I mentioned on the day of the HackEire event, the second log file contained multiple attempts to penetrate one of the honeypots on the Bhratach network :) Like I said, there will be 'red herrings' on the day, it's your responsibility to read the questions and correctly fingerprint the services and servers.
Snort Log Files
As we were using Snort as the IDS on Security Onion (you can also use
Suricata, which also looks very cool but I haven't played with it yet), we logically had Snort log files, located under
/nsm/NAME-OF-SENSOR
Running 'ps auwx' shows that we've the Snort agent running with its configuration in /etc/nsm/NAME-OF-SENSOR/snort_agent.conf
root 1287 0.0 0.1 5536 2164 ? S 16:58 0:00 tclsh /usr/local/bin/snort_agent.tcl -c /etc/nsm/ids-br0/snort_agent.conf
Security Onion uses
Barnyard (an output spool reader for Snort that decouples the output process from Snort so Snort can perform better and not worry about logging to a database) for Snort logging (/etc/nsm/$interface/snort.conf is set-up by default to log in unified binary mode) -
root 1349 0.0 0.0 8028 896 ? S 16:58 0:03
barnyard2 -c /etc/nsm/ids-br0/barnyard2.conf -d /nsm/sensor_data/ids-br0 -f
snort.unified2 -w /etc/nsm/ids-br0/barnyard2.waldo -U
and Sguil logging its information to
/nsm/sensor_data/$interface/dailylogs/$date as expected -
sguil 1387 0.0 0.2 6204 4344 ? S 16:58 0:01 daemonlogger -u sguil -g sguil -i br0 -l
/nsm/sensor_data/ids-br0/dailylogs/2011-12-22 -n snort.log -s 134217728
These log files are stored in Snort's binary format and to run them is as simple as -
- snort -r snort.log.1322068350
which produces something like -
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
11/23-17:12:30.904710 10.20.5.245:39437 -> 192.168.10.101:80
TCP TTL:63 TOS:0x0 ID:48174 IpLen:20 DgmLen:266 DF
***AP*** Seq: 0x247B1B15 Ack: 0x3A3B75A2 Win: 0xFFFF TcpLen: 20
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
11/23-17:12:30.904838 10.20.5.203:38826 -> 192.168.10.101:80
TCP TTL:63 TOS:0x0 ID:9461 IpLen:20 DgmLen:455 DF
***AP*** Seq: 0xF6BD0441 Ack: 0xD626FD50 Win: 0x3908 TcpLen: 20
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
Basic Packet Statistics
I'd promised @dougburks some statistics on the day, hence one of the reasons why I was looking at the afore-mentioned snort log files because within the file they give some nice breakdown on protocols and packets -
=============================================================================
Run time for packet processing was 20.870504 seconds
Snort processed 592198 packets.
Snort ran for 0 days 0 hours 0 minutes 20 seconds
Pkts/sec: 29609
=============================================================================
Packet I/O Totals:
Received: 592198
Analyzed: 592198 (100.000%)
Dropped: 0 ( 0.000%)
Filtered: 0 ( 0.000%)
Outstanding: 0 ( 0.000%)
Injected: 0
=============================================================================
Breakdown by protocol (includes rebuilt packets):
Eth: 592198 (100.000%)
VLAN: 0 ( 0.000%)
IP4: 589797 ( 99.595%)
Frag: 0 ( 0.000%)
ICMP: 21830 ( 3.686%)
UDP: 13452 ( 2.272%)
TCP: 554515 ( 93.637%)
IP6: 22 ( 0.004%)
IP6 Ext: 22 ( 0.004%)
IP6 Opts: 0 ( 0.000%)
Frag6: 0 ( 0.000%)
ICMP6: 0 ( 0.000%)
UDP6: 22 ( 0.004%)
TCP6: 0 ( 0.000%)
-----------------------
--------------------------SNIPPED
Total: 592198
=============================================================================
which I'm not going to expand on in this blog post (due to lack of time).
These snippets are pretty useful as they give a good protocol breakdown on your traffic profile.
Performance
We were running IPTraf
which reported a maximum of roughly
5000 kbits/sec and
2500 packets/sec (obviously not in the screenshot above :) ), which admittedly isn't huge but Security Onion easily handled the load.
- CPU - didn't exceed 20%. We did reconfigure the VM to use two vCPUs but it didn't make any difference as we expected memory was our issue but as explained below, we could do little about it other than reboot!!
- Memory - we were using up to 800MB and we'd only assigned 1GB to the VM. As we were using Xen Server 5.6 (for stability reasons), we were unable to dynamically assign more memory (we'll upgrade for next year to 2GB and XS6).
- Both 'top' and 'Cacti' reported large run queues (9-10) at regular intervals with 'tclsh' taking up a good chunk of memory -
Below are processes called by 'tclsh'. I am unsure as to which one was taking up most memory, however, I suspect the 'sguild' process simply because it's logically the most intensive. I didn't have time to troubleshoot on the day so didn't catch it unfortunately.
root 1099 1.7 57.6 2186564 1186708 ? S 16:57 6:28 tclsh /usr/local/bin/sguild -c /etc/nsm/securityonion/sguild.conf -a /etc/nsm/securityonion/autocat.conf -g /etc/nsm/securityonion/sguild.queries -A /etc/nsm/securityonion/sguild.access -C /etc/nsm/securityonion/certs
root 1228 0.0 0.1 5664 2716 ? S 16:57 0:00 tclsh /usr/local/bin/pcap_agent.tcl -c /etc/nsm/ids-br0/pcap_agent.conf
root 1253 0.0 0.1 6180 3304 ? S 16:57 0:01 tclsh /usr/local/bin/sancp_agent.tcl -c /etc/nsm/ids-br0/sancp_agent.conf
root 1287 0.0 0.1 5536 2164 ? S 16:58 0:00 tclsh /usr/local/bin/snort_agent.tcl -c /etc/nsm/ids-br0/snort_agent.conf
root 1459 0.0 0.1 5512 2224 ? S 16:58 0:00 tclsh /usr/local/bin/httpry_agent.tcl -c /etc/nsm/ids-br0/httpry_agent.conf -e /etc/nsm/ids-br0/httpry_agent.exclude -f /nsm/sensor_data/ids-br0/httpry/2011-12-22.log
root 1567 0.0 0.2 12628 6084 ? S 16:58 0:00 tclsh /etc/nsm/ossec/ossec_agent.tcl -o -f /var/ossec/logs/alerts/alerts.log -i 127.0.0.1 -p 5 -c /etc/nsm/ossec/ossec_agent.conf
Modifying Rules
So there are two rules files -
- downloaded-rules
- local-rules
These files reside in
/etc/nsm/NAME-OF-SENSOR.
It's very easy to modify the local-rules, e.g. here's two that I used to monitor GUI access to devices that had GUI access open -
----------------------------------------------------------------------------------------------------------------------
more local-rules.txt
alert tcp any any -> $GUI_NET 80 (msg:"Alert HTTP Access to Router & LBGUIs";sid:1000005; rev:1;)
alert tcp any any -> $GUI_NET 443 (msg:"Alert HTTPS Access to Router & LB GUIs";sid:1000006; rev:1;)
----------------------------------------------------------------------------------------------------------------------
I had defined the $GUI_NET variable previously in my Snort configuration for 'ids-br0'.
I also played a little with Pulledpork, which will download and modify the rules. The files are located in
/etc/pulledpork and are updated daily at 07:01 with manual updates by running
/usr/local/bin/pulledpork_update.sh. It is very simple :)
