System logs are very important for the continue health of your system. They provide a standard location to find errors, information, debug messages, and alerts. They can be used for diagnosis in order to prevent problems, and they are a valuable resource for troubleshooting. syslog-ng system logger has flexibility, simplicity and security.
The syslog protocol is a very simplistic protocol: the syslog sender sends a small textual message (less than 1024 bytes) to the syslog receiver. The receiver is commonly called “syslog daemon” or “syslog server”. Syslog messages can be sent via UDP and/or TCP.
Why to log things?
A legal obligation for companies.
Debugging and troubleshooting problems and reasons for downtime.
Security Audit of intrusions, attacks and invalid authorizations and authentications.
Provide statistics of events in the network.
Configuring syslog-ng
There are five basic things to configure
Options: configure various global configuration parameters.
options {
chain_hostnames (no);
create_dirs (yes);
dir_perm (0755);
dns_cache (yes);
……
};
Filters: Filters perform log routing inside syslog-ng. You can write a boolean expression using internal functions, which has to evaluate to true for the message to pass.
An expression may contain the operators “and”, “or” and “not”, and the following functions:
facility()
level()
program()
host()
match()
Each of the above functions check the corresponding field of a log message for matching (e.g. program() checks whether the given program sent the message, or not). You can use extended regular expressions for matching.
filter { expression; };
for eg
filter f_linux_host_deny { host(“linux.host”) and match(“deny”); };
Source : A source is a collection of source drivers, which collect messages using a given method. For instance there’s a source driver for AF_UNIX, SOCK_STREAM style sockets, which is used by the Linux syslog() call.
source { source-driver(params); source-driver(params); … };
for eg
source proxy_log {
udp(ip(xxx.xxx.xxx.xxx) port(514));
tcp(ip(xxx.xxx.xxx.xxx) port(514));
};
destination: A destination is where log is sent if filtering rules match. Similarly to sources, destinations are comprised of one or more drivers, each of which define how messages are handled. To declare a destination in the configuration file, you’ll need a destination statement, whose syntax is as following:
destination { destination-driver(params); destination-driver(params); … };
for eg
destination indexlog {
file(“/logs/indexlog/$YEAR/$MONTH/$DAY/$HOUR” template(“$MIN:$SEC,$HOST,$PROGRAM,$FACILITY,$PRIORITY,$MSGONLY\n”) template-escape(yes)
owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes));};
There are various macros available for destination for eg MIN, SEC, HOST, PROGRAM etc
Log: Now you have sources, destinations and filters. To connect these together you need the log statement:
log { source s1; source s2; …
filter f1; filter f2; …
destination d1; destination d2; … };
for eg
log {
source(proxy_log);
destination(indexlog);
};
Messages coming from any of the listed sources, and matching against all the listed filters (which effectively ANDs them) are sent to all of the listed destinations.
Centralized logging using Database
create a FIFO pipe
$mkfifo /var/run/db.pipe
create database using this schema
Name Type
—————————————– ——– —————————-
MSG_RCV_TIME TIMESTAMP(6)
MSG_SENT_TIME TIMESTAMP(6)
HOSTNAME VARCHAR2(256)
PROGRAM VARCHAR2(512)
MESSAGE VARCHAR2(2048)
FACILITY VARCHAR2(7)
SEVERITY VARCHAR2(7)
Event import script write following
For oracle
$cat import_live.sh
nohup sqlplus username/password < /var/run/ora.pipe >> /dev/null
For mysql
if [ -e /var/run/ora.pipe ];
then
while [ -/var/run/ora.pipe]
do
mysql -u username –password=password syslogdb < /var/run/ora.pipe
done
else
mkfifo /var/run/ora.pipe
fi
Create start|stop init script
#!/bin/sh
if test $# -lt 1
then
echo “Usage syslog.init start|stop”
exit 0
fi
case “$1” in
“start”)
echo “STARTING…………..”
# uncomment next line for oracle
# su – oracle -c “path/import_live.sh &”
# uncomment next line for mysql
# su – mysql -c “path/import_live.sh &”
path/syslog-ng -f /etc/syslog-ng.conf
;;
“stop”)
echo “STOPING…………..”
pkill -9 syslog-ng
pkill -9 import_live.sh
;;
*)
echo “Usage syslog.init start|stop”
esac
Create log source on central logging, in source remember that IP you should give IP address of central log host itself(not the IP of machine from where you receive logs)
source proxy_log {
udp(ip(centralloghostip) port(514));
tcp(ip(centralloghostip) port(514));
};
Create log destination for central logging, in destination add following
destination d_database {
pipe(“/var/run/ora.pipe”
template(“INSERT INTO syslogdb (hostname, facility, SEVERITY, MSG_SENT_TIME,program, message,MSG_RCV_TIME) VALUES
( ‘$HOST’,’$FACILITY’, ‘$PRIORITY’, to_date(‘$MONTH $DAY$ $YEAR $HOUR:$MIN:$SEC’, ‘mm dd yyyy hh24:mi:ss’), ‘$PROGRAM’, ‘$MSG
ONLY’,SYSTIMESTAMP);\ncommit;\n”) template-escape(yes));
};
Create logs using “log” on central logging system
log {
source(proxy_log);
destination(d_database);
};