splunk with bluecoat app

Sorry for the unorganized nature of this blog; just me throwing out some information.

In order to get the splunk with bluecoat application working (at http://splunk-base.splunk.com/apps/22335/splunk-for-bluecoat).

This product in conjunction with ProxySG does not work out of the box you need to:

Go into bluecoat and create a new log format of type:
splunk
date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-ip cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) cs-ip sc-bytes cs-bytes sr-bytes rs-bytes x-virus-id cs(Content-Type) timestamp localtime s-computername

Then edit your props.conf in /opt/splunk/etc/apps/SplunkforBlueCoat and make sure TZ=UTC is set in bcoat_proxysg.

Then edit transforms.conf and modify the line to look like this:
[delimExtractions]
DELIMS=" "
FIELDS="date","time","time_taken","src_ip","user","user_group","x_exception_id","filter_result","category","http_referrer","http_response","action","http_method","http_content_type","uri_scheme","dest_host","dest_ip","dest_port","uri_path","uri_query","uri_extension","http_user_agent","dvc_ip","sc_bytes","cs_bytes","sr_bytes","rs_bytes","x_virus_id","cs_content_type"

Next in bluecoat set up your logs for splunk to be a custom client to a TCP port previously setup in your indexer and verify it is text and continuous.

Then go into the VPM and in a web access layer make a new rule set to ANY/ANY whose action has the object "Access Logging Object" with Enable Logging set to splunk.

Your bluecoat search app will now start filling up and because of it sending time in both GMT and local the bluecoat results should appear right next to your firewall and other event logs!

How to completely automate ossec deployment via puppet

Yes you can completely automated ossec deployment by puppet.

Before you start and you have a working puppet system, you need to have a method of puppet to notify ossec about machines, hence these scripts:

on your puppet machine put the following files:

/etc/cron.hourly/puppetca-to-ossec

#!/bin/sh
/usr/local/bin/populate-ossec.sh

/usr/local/bin/populate-ossec.sh

#!/bin/sh
/usr/sbin/puppetca --list -all|grep +|cut -f 2 -d' ' > /etc/puppet/files/ossec/puppet-servers.txt

This will generate a list of machines managed by puppet as potential candidates for ossec.

The output of the file looks something like this:
/etc/puppet/files/ossec/puppet-servers.txt

server1.yourdomain.com
server2.yourdomain.com
server3.yourdomain.com
server4.yourdomain.com

next you need to install ossec on puppet machines; I used http://projects.puppetlabs.com/projects/1/wiki/OSSEC-HIDS_Patterns as a starting point.

put the following in:
/etc/puppet/modules/ossec/manifests/init.pp

import "*"

/etc/puppet/modules/ossec/manifests/ossec.pp

# /etc/puppet/modules/ossec/manifests/ossec.pp
# install ossec-hids-agent on monitored servers and workstations
# This module downloads the defined version of ossec from a defined website
# extracts the files, configures the preloaded-vars.conf file to unattended install
# runs the installation script and then copies and manages the master ossec.conf
# and rules files.

# call this module via: node 'name' {include ossec::server}
# replace server with local or agent depending on the type of ossec install you
# want to perform.

class ossec {


# I will move this define to a common definition module later
define download_file(
$site="ossec.yourdomain.com",
$cwd="",
$creates="",
$require="",
$user="") {

exec { $name:
command => "/usr/bin/wget ${site}/${name}",
cwd => $cwd,
creates => "${cwd}/${name}",
require => $require,
user => $user,
}
}



class install{

$ossecversion = "ossec-hids-2.5.1"
$ossecfile = "$ossecversion.tar.gz"
$workdir = "/opt/working"


file { "/opt/working":
ensure => directory,
owner => root,
group => root,
mode => 760,
}

download_file {"${ossecfile}":
site => "http://www.ossec.net/files", # best to use a local copy if working with lots of machines
cwd => "${workdir}",
creates => "${workdir}/$name",
require => File["/opt/working"],
user => root,
}

exec {"extract-ossec":
cwd => "${workdir}",
command => "/bin/tar xzf ${ossecfile}",
creates => "${workdir}/${ossecversion}",
require => Download_file["${ossecfile}"],
user => root,
}

}


class server inherits ossec::install {

$ossectype = "server"

file {"ossecvars":
path => "${workdir}/${ossecversion}/etc/preloaded-vars.conf",
ensure => present,
content => template("ossec/preloaded-vars.conf-${ossectype}"),
require => Exec["extract-ossec"],

}

if ($operatingsystem == "Debian" ) {
package { gcc: ensure => installed }
package { make: ensure => installed }
exec { "/bin/mkdir -p /var/ossec-server": }
}

#set the number of aggents ossec supports default is 256, which frankly isn't enough
# exec {"install-ossec-setclients":
# cwd => "${workdir/${ossecversion}/src",
# command => "${workdir}/${ossecversion/src/echo HEXTRA=-DMAX_AGENTS=1024 >> Config.OS"
# user => root
# }

exec {"install-ossec":
cwd => "${workdir}/${ossecversion}",
command => "${workdir}/${ossecversion}/install.sh",
creates => "/var/ossec-server/etc",
user => root,
require => File["ossecvars"],
}


service { "ossec":
enable => true,
ensure => running,
}

# manage ossec.conf file
file { "ossec.conf":
path => "/var/ossec-server/etc/ossec.conf",
ensure => present, owner => root, group => ossec, mode => 550,
content => template("ossec/ossec-conf-${ossectype}.erb"),
}

# manage the /var/ossec-server/rules
file { "ossec-rules":
path => "/var/ossec-server/rules",
checksum => "mtime",
ensure => directory, owner => root, group => ossec, mode => 550,
source => "puppet://$server/ossec/ossec-rules",
recurse => true,
ignore => [ ".svn" ],
}


# manage the /var/ossec-server/etc/shared
file { "ossec-shared":
path => "/var/ossec-server/etc/shared",
checksum => "mtime",
ensure => directory, owner => root, group => ossec, mode => 550,
source => "puppet://$server/ossec/shared",
recurse => true,
ignore => [ ".svn", "merged.mg" ],
}
# ensure presence and permissions of existence of merged.mg
file { "merged.mg":
path => "/var/ossec-server/etc/shared/merged.mg",
ensure => present, owner => ossecr, group => ossec, mode => 644
}
# manage the batch ossec manager
file { "ossec-batch-manager.pl":
path => "/usr/local/bin/ossec-batch-manager.pl",
checksum => "mtime",
ensure => present, owner => root, group => ossec, mode => 755,
source => "puppet://$server/ossec/ossec-batch-manager.pl"
}
# script for generating keys
file { "genkeys.sh":
path => "/usr/local/bin/genkeys.sh",
checksum => "mtime",
ensure => present, owner => root, group => root, mode => 755,
source => "puppet://$server/ossec/genkeys.sh"
}

#following file is auto-generated from the puppet database
file { "puppet-servers.txt":
path => "/usr/local/etc/puppet-servers.txt",
checksum => "mtime",
ensure => directory, owner => root, group => root, mode => 755,
source => "puppet://$server/ossec/puppet-servers.txt"
}


file { "distributekeys.sh":
path => "/usr/local/bin/distributekeys.sh",
checksum => "mtime",
ensure => present, owner => root, group => root, mode => 755,
source => "puppet://$server/ossec/distributekeys.sh"
}
#stick a cronjob to execute once a hour to create keys for the any new hosts
file { "ossec-genkeys.sh":
path => "/etc/cron.hourly/ossec-genkeys.sh",
checksum => "mtime",
ensure => present, owner => root, group => root, mode => 755,
source => "puppet://$server/ossec/cron.hourly/ossec-genkeys.sh"
}
exec {ossec-restart:
command => "/var/ossec-server/bin/ossec-control restart",
subscribe => File[ "ossec.conf" , "ossec-rules" ],
refreshonly => true, # Only run command if monitored files change
}

}



class local inherits ossec::install {

$ossectype = "local"

file {"ossecvars":
path => "${workdir}/${ossecversion}/etc/preloaded-vars.conf",
ensure => present,
content => template("ossec/preloaded-vars.conf-${ossectype}"),
require => Exec["extract-ossec"],

}

exec {"install-ossec":
cwd => "${workdir}/${ossecversion}",
command => "${workdir}/${ossecversion}/install.sh",
creates => "/var/ossec/etc",
user => root,
require => File["ossecvars"],
}


service { "ossec":
enable => true,
ensure => running,
}

# manage ossec.conf file
file { "ossec.conf":
path => "/var/ossec/etc/ossec.conf",
ensure => present, owner => root, group => ossec, mode => 550,
content => template("ossec/ossec-conf-${ossectype}.erb"),
}
# manage the /var/ossec/rules
file { "ossec-rules":
path => "/var/ossec/rules",
checksum => "mtime",
ensure => directory, owner => root, group => ossec, mode => 550,
source => "puppet://$server/ossec/ossec-rules",
recurse => true,
ignore => [ ".svn" ],
}

#managed the /var/ossec/etc/shared
file { "ossec-shared":
path => "/var/ossec/etc/shared",
checksum => "mtime",
ensure => directory, owner => root, group => ossec, mode => 550,
source => "puppet://$server/ossec/shared",
recurse => true,
ignore => [ ".svn" ],
}


exec {ossec-restart:
command => "/var/ossec/bin/ossec-control restart",
subscribe => File[ "ossec.conf" , "ossec-rules" ],
refreshonly => true, # Only run command if monitored files change
}
}


class agent inherits ossec::install {
$ossectype = "agent"

file {"ossecvars":
path => "${workdir}/${ossecversion}/etc/preloaded-vars.conf",
ensure => present,
content => template("ossec/preloaded-vars.conf-${ossectype}"),
require => Exec["extract-ossec"],

}

if ($operatingsystem == "Debian" ) {
package { gcc: ensure => installed }
package { make: ensure => installed }
exec { "/bin/mkdir -p /var/ossec": }
}


exec {"install-ossec":
cwd => "${workdir}/${ossecversion}",
command => "${workdir}/${ossecversion}/install.sh",
creates => "/var/ossec/etc",
user => root,
require => File["ossecvars"],
}


service { "ossec":
enable => true,
ensure => running,
}

# manage ossec.conf file
file { "ossec.conf":
path => "/var/ossec/etc/ossec.conf",
ensure => present, owner => root, group => ossec, mode => 550,
content => template("ossec/ossec-conf-${ossectype}.erb"),
}

exec {ossec-restart:
command => "/var/ossec/bin/ossec-control restart",
subscribe => File["ossec.conf"],
refreshonly => true, # Only run command if monitored files change
}
#
group { 'ossec':
ensure => present
}


user { 'ossec-push':
uid => '809',
ensure => 'present',
gid => ossec,
home => '/var/ossec',
shell => '/bin/bash'
}

file { "/var/ossec/.ssh":
ensure => directory,
owner => ossec-push,
group => ossec,
mode => 650
require => User["push-ossec"]
}

file { "authorized_keys-push-ossec":
path => "/var/ossec/.ssh/authorized_keys",
checksum => "mtime",
ensure => present, owner => push-ossec, group => ossec, mode => 644,
source => "puppet://$server/ossec/authorized_keys",
require => User["push-ossec"]
}


file { "/var/ossec/etc":
ensure => directory,
owner => root,
group => ossec,
mode => 550
}

file { "/var/ossec/etc/client.keys":
ensure => present, owner => root, group => ossec, mode => 660
}
}


}

next edit the stuff in your template directory
ossec-conf-agent.erb
ossec-conf-server.erb
preloaded-vars.conf-agent
preloaded-vars.conf-server

you can get all files involved at http://vector.xyxx.com/~myrond/ossec-puppet.tar.gz

If you look at the tarball for various files that you need to change for your system.

You'll need to run a ssh-keygen -t rsa on your ossec machine as root so you can generate a key then copy that key to your puppet machine and modify the authorized_keys file such that it looks like this:

from="ossec.yourdomain.com",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjwSEVT7lp1wZTft+lYivqVQ4Wetwk+rxn5b2RIdm1DnlnM5es1+/kXyn92YkJxqfBxJ1K54QSsN0rjKjUbc5wTvosrvpffSBQ5HrWgzJe3EhXJ+tUV3Hz5AjXgzb4pZnamdpI5YEUfM3Y02CMup/NCClmfu1mlhNaq+1AgLzh7hy13CZHjfdI0ui6g4qORkj7F7LwSt/RgMbrUzt5iSfID7FEyGPMtjh1XleEtPg+2UatHmYUVOeiJg0d7pkJL+UjRaLR9TfNjCRfTDv0EX8DF1Om4ZvXIcy7WhQ/zhP/F72YOr13kU9yj22cARpqjnT3A/RfRHIrASg7OO9Q2eB root@ossec

This file is pushed to ALL machines so the ossec machine can connect

The genkeys.sh which is pushed out to the ossec server is also very important
Put it in your /etc/puppet/files/ossec directory (it will be grabbed in modules/common/files/modules/ossec/manifests/ossec.pp to be pushed out)

#!/bin/sh
# THIS FILE IS MAINTAINED BY PUPPET
TEMPFILE45=`/bin/mktemp`
cat /usr/local/etc/puppet-servers.txt |xargs -i host {} |grep "has address"|cut -f 1,4 -d ' ' > $TEMPFILE45


cat $TEMPFILE45 | awk '{print $1" "$2;}' |while read x y; do /usr/local/bin/ossec-batch-manager.pl -a --name $x --ip $y 2>/dev/null; done
rm $TEMPFILE45

this parses the puppet-servers.txt file which is pushed out from puppet and creates keys for all NEW boxes managed by puppet. The majority of all of this work is done by Jeff Schroeder's script ossec-batch-manager.pl

distributekeys.sh

#!/bin/sh
# THIS FILE IS MAINTAINED BY PUPPET

cat /var/ossec-server/etc/client.keys|awk '{print $1" "$2" "$3" "$4}'| while read a b c d;
do
KEYTODISTRIBUTE=`mktemp`
echo $a $b $c $d > $KEYTODISTRIBUTE
#echo id=$a
#echo name=$b
#echo ip=$c
#echo key=$d
scp -B -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $KEYTODISTRIBUTE ossec-push@$c:/var/ossec/etc/client.keys >/dev/null 2>/dev/null
#scp $KEYTODISTRIBUTE
rm $KEYTODISTRIBUTE
done

This file takes each key parses it and then sends it to its destination via the account ossec-push that was created earlier with ossec You can run this whenever you want too or setup a cron job or you could set it to run whenever the puppet-servers.txt file changes.

ALWAYS have a test vm somewhere to test this on before going to production!

I highly recommend loading the following modules via puppet:
ssh
lsb
apt

you can see a bunch of stuff via the file listed above.

I would have pasted more stuff in this blog entry but it was a lot of cuting and pasting, download the file and look at it.

Saw TiMER

As some of you may (or may not know) the company I work for gives out tickets for events and stuff.

I haven't actually wrote about this stuff before but nevertheless it does happen.

The other week there was the Tribeca Film Festival and in order to decide what film to go to you put your name into a computerized lottery.

I won 2 tickets to see TiMER which I saw with my cousin Mandy. I didn't really expect much as it was an independent film and I didn't know anything about it.

When I got there, the lines were long and there was a long line of people waiting for tickets IF the primary people who already owned tickets didn't make it.

The Film was based upon a story that for a small nominal fee you could implant a device in your wrist that would light up and beep if you ever met your soul-mate. It also had a little device which would count down the number of days/hours till you meet your soul mate.

As you can imagine all kinds of problems occur as some people's TiMER's count out 30 some years before they meet their soul mate.

The question is asked? Do you wait 40 years for your soul mate? Do you take someone who likes you right now? And if you are already married to someone and you find out they are not your soul-mate what do you do then?

Also it seems you can fall in love with people who are not your soul mate as well.

Do you trust a little device in your arm?

The film turned out to be very good and the actresses and director spoke in front of the movie afterward and took questions. Both me and my cousin Mandy really liked the movie.

I'm not sure when/if you can see it as it doesn't have a distributor but based on how good the movie was I would be surprised if it wasn't picked up.

Still in NY, and I need your help

I'm still in NYC but I need YOUR help!

Here is the quest if your choose to accept:

Post something that I should do in NYC, I'll do it and post about it! I will also mail/post a pic/something to the person who suggests the activity a souvenir from the event!

I was going to post this, this weekend.

I'm not sure if anyone actually reads this blog anymore because I kind of let it die for a while... but I'm thinking I have at least one person who has it subscribed!

an update in forever

I haven't wrote anything... in forever I know so things I've been up too:

Went to alaska for a week a half
Went white water rafting
Went to Georgia
Went climbing in the gunks several times w/ trad gear
Torsten came and visited me
Debbie came and visited me
Went to some halloween stuff
Ate out a lot
This and that

chances are I'm not going to be updating this blog for a while.

I've kind of lost interest in writing a blog for some reason, I might start it up again in the future but for now... no.

Going out w/ Peng and friends to dinner tonight should be fun.

-Myron