Blog | HanG321 Blog

Discovered logstash last year, finally I have played around logstash and elasticsearch during weekend, pretty easy to walk through Getting Started. However when I try to customize pattern for work-related log, its documentation does not come with good example and its API changed between 1.2 and 1.3… some search result from google using deprecated version. Anyway here is only document what I played so far.

Preparation:
Download a “flat” jar (logstash integrated with elasticsearch now): http://download.elasticsearch.org/logstash/logstash/logstash-1.3.3-flatjar.jar
required curl, if it does not installed yet: $ sudo apt-get install curl

Tasting:
– Walk through “Getting started with logstash (standalone server example)”
– For production, input would better be log4j rather than file, here just a test.

config file:

Since the log4j timestamp does not use ISO format, need to define it. see http://logstash.net/docs/1.3.3/filters/grok#patterns_dir
create a folder named “pattern”, create a file inside and save with follow content.

LOGDATETIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}

1	LOGDATETIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}

input {
  stdin {
    type => "stdin-type"
  }

  file {
    type => "xmllog"
		# path must be absolute
    path => "/workspaces/logstatsh/log4j-xml.log"
  }
}

filter {
  grok {
		patterns_dir => "./patterns"
		# sample log: 2014-03-08 11:50:02.001 INFO [loggingHandler] (logger.java:123) - <?xml .....>
    match => [ "message", "%{LOGDATETIME:logDateTime}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:thread}\]%{SPACE}\(%{JAVACLASS:class}:%{NUMBER:line}\)%{SPACE}-%{SPACE}(?<xmlData>.*$)" ]

		# remove fields don't care, e.g. line number
		remove_field => [ "line" ]
  }

	date {
		# matched pattern will replace "@timestamp" field to logDateTime (parsed above)
		match => ["logDateTime", "YYYY-MM-dd HH:mm:ss.SSS"]
	}

	xml {
		source => "xmlData"
		target => "parsedXml"
	}
}

output {
  stdout { debug => true }
  elasticsearch { embedded => true }
}

input {

stdin {

type => "stdin-type"

}

file {

type => "xmllog"

# path must be absolute

path => "/workspaces/logstatsh/log4j-xml.log"

}

filter {

grok {

patterns_dir => "./patterns"

# sample log: 2014-03-08 11:50:02.001 INFO [loggingHandler] (logger.java:123) - <?xml .....>

match => [ "message", "%{LOGDATETIME:logDateTime}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:thread}\]%{SPACE}$%{JAVACLASS:class}:%{NUMBER:line}$%{SPACE}-%{SPACE}(?<xmlData>.*$)" ]

# remove fields don't care, e.g. line number

remove_field => [ "line" ]

}

date {

# matched pattern will replace "@timestamp" field to logDateTime (parsed above)

match => ["logDateTime", "YYYY-MM-dd HH:mm:ss.SSS"]

}

xml {

source => "xmlData"

target => "parsedXml"

}

output {

stdout { debug => true }

elasticsearch { embedded => true }

}

N.B. in xml filter, remember to define target => “xxxx”! Otherwise console debug logged xmlparsefailure and NoMethodException. With it, all first level children xml tags will be indexed and fields will be created automatically, no need to setup XPath.

okay, now run $ java -jar logstash-1.3.3-flatjar.jar agent -f test.conf -- web

By paste following line to ‘standard in’ or listened log file: (xml from w3school example)

2014-03-09 19:01:03.005 INFO [loggingHandler] (logger.java:123) - <note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>

1	2014-03-09 19:01:03.005 INFO [loggingHandler] (logger.java:123) - <note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>

then, by debug=true, logstash shell console display information. And it will send to elasticsearch with following result.

Now go to http://localhost:9292/index.html#/dashboard/file/logstash.json to see the result!

{
  "_index": "logstash-2014.03.09",
  "_type": "stdin-type",
  "_id": "94ycZEdfQDCF_c5PxP-zlg",
  "_score": null,
  "_source": {
    "message": "2014-03-09 19:01:03.005 INFO [loggingHandler] (logger.java:123) - <note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>",
    "@version": "1",
    "@timestamp": "2014-03-09T19:01:03.005+10:00",
    "type": "stdin-type",
    "host": "hang321-mintvm",
    "logDateTime": "2014-03-09 19:01:03.005",
    "level": "INFO",
    "thread": "loggingHandler",
    "class": "logger.java",
    "xmlData": "<note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>",
    "parsedXml": {
      "to": [
        "Tove"
      ],
      "from": [
        "Jani"
      ],
      "heading": [
        "Reminder"
      ],
      "body": [
        "Don't forget me this weekend!"
      ]
    }
  },
  "sort": [
    1394355663005,
    1394355663005
  ]
}

{

"_index": "logstash-2014.03.09",

"_type": "stdin-type",

"_id": "94ycZEdfQDCF_c5PxP-zlg",

"_score": null,

"_source": {

"message": "2014-03-09 19:01:03.005 INFO [loggingHandler] (logger.java:123) - <note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>",

"@version": "1",

"@timestamp": "2014-03-09T19:01:03.005+10:00",

"type": "stdin-type",

"host": "hang321-mintvm",

"logDateTime": "2014-03-09 19:01:03.005",

"level": "INFO",

"thread": "loggingHandler",

"class": "logger.java",

"xmlData": "<note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>",

"parsedXml": {

"to": [

"Tove"

"from": [

"Jani"

"heading": [

"Reminder"

"body": [

"Don't forget me this weekend!"

]

}

"sort": [

1394355663005,

1394355663005

]

}

Hangover Part 3 / Blind Detective盲探 / Saving Mr. Banks

logstash & elasticsearch

spotify offline mode

May 2025
M	T	W	T	F	S	S
« Feb
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31