Syslog messages in EFK stack

You can use opensource syslog-ng tool to capture syslog messages from various sources, like remote servers, devices, local logs and so on. These logs can then be forwarded to fuentd on EFK stack.

Here is a sample config of syslog-ng config to receive logs from devices and hosts on tcp and udp port 514. Received logs will then be forwarded to fluentd on efk_host on port 5140/udp.

source network {
        udp(ip(0.0.0.0) port(514));
        tcp(ip(0.0.0.0) port(514));
};

destination efk_stack {
 udp("efk_host" port(5140));
};

log {
	source(network);
        destination(efk_stack);
};

Here is a sample config of td-agent.conf on fluentd. This config will receive messages on port 5140/udp, add prefix tag “mka.logs” and then pump messages into elasticsearch database by matching tag values “mka.logs”, “mka.logs.*”, “mka.logs.*.*” or anything like that. Logs messages will settle down in syslog-* index pattern match in elasticsearch and using same index pattern you can visualize log messages in Kibana.

In parse section of following config, you can configure syslog format RFC rfc3164, rfc5424 or leave it auto.

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  protocol_type udp
  tag mka.logs
  <parse>
    message_format auto
  </parse>
</source>

<match mka.**>
    @type elasticsearch
    include_tag_key true
    host localhost
    port 9200
    logstash_format true
    logstash_prefix syslog
</match>