Aim

Sometime we need to analyze ours logs and perform some analysis on it. We can't perform analysis by reading the log file directly because it will be very time-consuming and the data in unstructured.

In this post we will try to read log file and dump that log data into Elasticsearch using Logstash.

Architecture

Discover Menu

Prerequisite

Operating System

  • We will be using Ubuntu 18.04 to build this data pipeline

Logstash

Fake Apache Log Generator

Elasticsearch and Kibana

Step 1 : Generating the log data

  • Follow the prerequisite step for Fake Apache Log Generator
  • we will be generating continuous amount of log at an interval of 2 second
python apache-fake-log-gen.py -n 0 -s 2 -o LOG
  • This will create a log file with a prefix as access_log
  • Some sample contents in the file will be
56.228.203.49 - - [01/Sep/2020:00:23:32 +0530] "PUT /wp-admin HTTP/1.0" 200 4988 "http://www.mccoy.com/about.htm" "Mozilla/5.0 (Macintosh; PPC Mac OS X 10_7_1 rv:4.0; en-US) AppleWebKit/533.2.1 (KHTML, like Gecko) Version/4.0.4 Safari/533.2.1"
49.40.236.125 - - [01/Sep/2020:00:23:34 +0530] "GET /explore HTTP/1.0" 200 4977 "http://www.shelton-young.org/" "Mozilla/5.0 (Windows; U; Windows 98) AppleWebKit/535.21.4 (KHTML, like Gecko) Version/4.0.5 Safari/535.21.4"
251.137.140.55 - - [01/Sep/2020:00:23:36 +0530] "POST /apps/cart.jsp?appID=4049 HTTP/1.0" 500 4966 "http://jones.com/search/tags/tags/search/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7_8) AppleWebKit/5330 (KHTML, like Gecko) Chrome/15.0.888.0 Safari/5330"

Step 2 : Setting up the Logstash

  • Follow the prerequisite step of installing Logstash.

Logstash pipeline config folder : /etc/logstash/conf.d/

  • The above folder holds all the pipeline configuration.
  • From now on we will call this config folder as LOGSTASH_CONFIG_FOLDER
  • create a file with name pipeline.conf inside LOGSTASH_CONFIG_FOLDER
  • You can view the logs of Logstash using below command. This will open the logs in tail mode.
 journalctl -u logstash -f

Step 2.1 : Logstash input configuration for reading log file

  • The first thing while creating Logstash pipeline is to define the input configuration
  • As we are reading data from a file, so we will be using File input plugin

input {
  file {
    id => "apache-log-reader"
    path => ["/home/selftuts/sandbox/apache-logs/*.log"]
    start_position => "beginning"
    mode => "tail"
  }
}

The different configuration for file input plugins are

  • id : This is a unique id to the plugin configuration. This helps while debugging the pipeline in case of failures.
  • path : An array of file path. These files provided here will be considered for reading. In our case we will be reading all log files present inside /home/selftuts/sandbox/apache-logs/ folder.
  • start_position : When a new file is encountered then it will read from be beginning of the file.
  • mode : This is never ending stream and will read the new logs as soon as they are written the log files.

Step 2.2 : Logstash output configuration for publishing log data to Elasticsearch

output {
  elasticsearch {
        hosts => ["192.168.0.10:9200"]
        index => "app-log"
  }
}
  • we will provide the address of Elasticsearch inside the hosts field
  • Then log data will be published inside app-log index.

Final Logstash configuration will be

input {
  file {
    id => "apache-log-reader"
    path => ["/home/selftuts/sandbox/apache-logs/*.log"]
    start_position => "beginning"
    mode => "tail"
  }
}

output {
  elasticsearch {
        hosts => ["192.168.0.10:9200"]
        index => "app-log"
  }
}

Step 3 : Start Logstash service and view logs

  • You need to start/restart Logstash server
sudo systemctl restart logstash
  • You can view logstash logs using journalctl
journalctl -u logstash -f
  • If there is no error in the logs then Logstash pipeline was loaded successfully, and it will start to consume data from log file and publish that to Elasticsearch.

Step 4 : Access data in Kibana

Kibana Management Option

  • Click on Index Patterns and then click on Create Index Pattern

Kibana Create Index Pattern

  • Search for your index (app-data) then click next step
  • Then click on show advanced options and select @timestamp from the drop down box
  • Finally click on create index pattern and your index will be created

Viewing Data in Kibana

  • Once the index has been created then you can go to Discover menu of Kibana

Discover Menu

  • You will see the index app-data is already selected and the data is present.

Elastic-search-index-data

Happy Coding

0 0 vote
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x