Aim

In this tutorial we will be installing Logstash on Ubuntu 18.04 machine. Logstash is a part of ELK stack and it is useful in collecting data from any source and then dump that data to any sink. Thus Logstash plays an important role in movement of data in the ELK ecosystem.

Prerequisite for Logstash installation on Ubuntu 18.04

  • Java must be installed on the on Ubuntu 18.04

Step 1 : Download and install the Public Signing Key

  wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Step 2 : Installing dependencies

  • You may need to install the apt-transport-https package on Debian before proceeding
  sudo apt-get install apt-transport-https

Step 3 : Add elastic search repository

  echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Step 4 : Update the apt repository

  sudo apt-get update

Step 5 : Finally install Logstash

sudo apt-get install logstash

Check if installation is successful

  • You need to check the service status to and verify that the Logstash server has been successfully installed
 systemctl status logstash
  • You will get output of this kind and this will denote that the service is inactive
   logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

Starting Logstash

  • You can start the Logstash in background
systemctl start logstash
  • once you have started the Logstash server then you can again check the server status using
systemctl status logstash
  • This time you will get below output and this denotes that the Logstash server is up and running
 logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-08-02 18:28:28 UTC; 3s ago
 Main PID: 26917 (java)
    Tasks: 21 (limit: 4915)
   CGroup: /system.slice/logstash.service
           └─26917 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8

Where to put your Logstash configuration

  • By default Logstash reads the configuration from /etc/logstash/conf.d/ directory
  • You can create your own pipeline by providing the input, filter and output configuration in a file and then put that file the above location
  • For example we can create a file with name kafka-elastic.conf.
cd /etc/logstash/conf.d/
touch kafka-elastic.conf
  • we can create a pipeline where Logstash will read data from Kafka topic and push that data to elastic search

Kafka Bootstrap server is running at localhost:9093

Elasticsearch server is running at localhost:9200

input {
    kafka {
            bootstrap_servers => "localhost:9093"
            topics => ["registered_user"]
    }
}

output {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "registered-user"
      workers => 1
    }
}

Checking the Logstash logs

  • You can find Logstash logs inside /var/log/logstash folder

Happy Coding

0 0 vote
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x