Querying elasticsearch with REST json

You may like to read ELK installation and configuration and ELK with syslog and SNMP before this.

Recently I was looking to build some useful queries to search elasticsearch database having syslog messages from my syslog-ng servers. My requirement was to pull records within datetime ranges matching with set of hosts/IPs and text patterns within syslog messages. You can also mention maximum number of results you want in return by setting value of “size”. You can use any REST client and fire JSON query towards Elasticsearch host. I have used simple curl tool on cli.

Make sure that elasticsearch is enabled on non-loopback interface. Add following lines in elasticsearch.yml and restart elasticsearch service to enable elasticsearch on all network interfaces. You can follow more secure way to enable elasticsearch on network at Secure Elasticsearch and Kibana using apache.

http.port: 9200
http.host: 0.0.0.0

To query all logs within a datetime range, replace datetime with your data.

curl -XGET "http://192.168.0.25:9200/_search" -H 'Content-Type: application/json' -d'
{
  "size": 100, 
    "query": {
        "range" : {
            "@timestamp" : {
                "gte": "00:00:00 11/02/2019",
                "lte": "08:59:00 13/02/2019",
                "format": "HH:mm:ss dd/MM/yyyy "
            }
        }
    }
}'

To query logs matching multiple hosts, you can use both IPs and Hostnames in this query within a datetime range.

Replace hosts and datetime range with your data values.

curl -XGET "http://192.168.0.25:9200/_search" -H 'Content-Type: application/json' -d'
{
  "size": 100,
  "query": {
    "bool": {
      "should": [
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.34"
          }
        },
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.100"
          }
        }
      ],
      "minimum_should_match": 1,
      "filter": {
        "range": {
          "@timestamp": {
            "gte": "00:00:00 10/02/2019",
            "lte": "08:59:00 11/02/2019",
            "format": "HH:mm:ss dd/MM/yyyy "
          }
        }
      }
    }
  }
}'

To query datetime range for a set of hosts matching particular text in syslog message

curl -XGET "http://192.168.0.25:9200/_search" -H 'Content-Type: application/json' -d'
{
  "size": 100,
  "query": {
    "bool": {
      "should": [
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.34"
          }
        },
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.100"
          }
        }
      ],
      "minimum_should_match": 1,
      "filter": {
        "range": {
          "@timestamp": {
            "gte": "00:00:00 10/02/2019",
            "lte": "08:59:00 11/02/2019",
            "format": "HH:mm:ss dd/MM/yyyy "
          }
        }
      },
      "filter": [
        {
          "match_phrase": {
            "syslog_message": "Router"
          }
        }
      ]
    }
  }
}'

Query to search in date time range, matching hosts and text pattern in syslog message

curl -XGET "http://192.168.0.25:9200/_search" -H 'Content-Type: application/json' -d'
{
  "size": 100,
  "query": {
    "bool": {
      "should": [
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.100"
          }
        },
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.34"
          }
        }
      ],
      "minimum_should_match": 1,
      "filter": {
        "range": {
          "@timestamp": {
            "gte": "00:00:00 10/02/2019",
            "lte": "08:59:00 11/02/2019",
            "format": "HH:mm:ss dd/MM/yyyy "
          }
        }
      }, 
      "filter": [
        {
          "wildcard": {
            "syslog_message": {
              "value": "rout*"
            }
          }
        }
      ]
    }
  }
}'

Lucene style queries with AND OR NOT operators

curl -XGET "http://192.168.0.25:9200/_search" -H 'Content-Type: application/json' -d'
{
  "size": 100,
  "query": {
    "bool": {
      "should": [
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.25"
          }
        },
        {
          "match_phrase": {
            "syslog_hostname": "192.168.0.34"
          }
        }
      ],
      "minimum_should_match": 1,
      "filter": {
        "range": {
          "@timestamp": {
            "gte": "now-1d",
            "lte": "now",
            "format": "HH:mm:ss dd/MM/yyyy "
          }
        }
      },
      "must": [
        {
          "query_string": {
            "fields": ["syslog_message"],
            "query": "critical OR error"
          }
        }
      ]
    }
  }
}'

Here is a python example to fire search queries

#!/usr/local/bin/python3
import requests
import json

elasticuri="http://127.0.0.1:9200/_search"
elheaders={'Content-type': 'application/json'}
elquery={
        "size": 100,
        "query": {
           "range" : {
             "@timestamp" : {
                "gte": "now-7d",
                "lte": "now"
             }
           }
        }
}

getdata=requests.get(url=elasticuri, data=json.dumps(elquery), headers=elheaders)

results=json.loads(getdata.text)
print(results)

You can use query builder (Devtools) available on kibana to draft and test more used cases.