The EFK (Elasticsearch, Fluentd and Kibana) stack is an open-source alternative to paid log management, log search and log visualization services like Splunk, SumoLogic and Graylog (Graylog is open source but enterprise support is paid). These services are used to search large amounts of log data for better insights, tracking, visualisation and analytical purposes. The EFK stack i.e. Elasticsearch, Fluentd and Kibana are separate open source projects that together make an amazing open source centralized log management stack that is not only free to use and easy to setup/install but also scalable and can handle really large amounts of log data in realtime. This article documents how to setup Elasticsearch, Fluentd and Kibana and putting it all together to get the best out of your boring log data. So Without further ado, Let’s jump right into the setup/installation process.
Let us first create a folder that we’ll put our EFK stack into. I chose to use work in /Users/amyth/installs/efk. You can choose any location that you would like to work on, Really.
| 1 | mkdir -p /Users/amyth/installs/efk |
1. Installing & Running Elasticsearch
1.1 Java Installation
Let us get started by installing Java as it is one of the core dependencies of elastichsearch.
| 12 | sudo apt-get updatesudo apt-get install openjdk-7-jre |
Once the installation is finished, Confirm it by checking the java version using the following command.
| 1 | java -version |
and you should see something like the following.
| 123 | java version “1.7.0_75″Java(TM) SE Runtime Environment (build 1.7.0_75-b13)Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) |
1.2 Installing Elasticsearch
Next, download elasticsearch (v2.1.0)Â Â and uncompress the downloaded package.
| 12 | tar -xzvf elasticsearch-2.1.0.tar.gzmv elasticsearch-2.1.0 ~/installs/efk/ |
Now let’s run the an elasticsearch instance by cd’ing into the elasticsearch folder and running the elasticsearch script in the bin folder. To run elasticsearch as a daemon use the -d argument while calling the script.
| 12 | cd ~/installs/efk/elasticsearch-2.1.0./bin/elasticsearch |
or to run it as a daemon
| 12 | cd ~/installs/efk/elasticsearch-2.1.0./bin/elasticsearch -d |
After running Elasticsearch, confirm you have a running instance by navigating to 127.0.0.1:9200 and you should see something like the following:
| 123456789101112 | { “name” : “Cerise”, “cluster_name” : “elasticsearch”, “version” : { “number” : “2.1.0”, “build_hash” : “72cd1f1a3eee09505e036106146dc1949dc5dc87”, “build_timestamp” : “2015-11-18T22:40:03Z”, “build_snapshot” : false, “lucene_version” : “5.3.1” }, “tagline” : “You Know, for Search”} |
2. Installing & Running Kibana
Now let us install and configure Kibana. First download kibana from this download page. Once downloaded move the download file to our efk install location and uncompress the downloaded file.
| 123 | mv ~/Downloads/kibana-4.3.0-darwin-x64.tar.gz ~/installs/efkcd ~/installs/efktar -xzvf kibana-4.3.0-darwin-x64.tar.gz |
Next, lets run kibana using the following command
| 12 | cd kibana-4.3.0-darwin-x64./bin/kibana |
Now in your web browser navigate to http://0.0.0.0:5601 and you should see the kibana dashboard. Something like the following image.
kibana
Now before we create indices, let’s get the third and final pillar to our stack up and running.
3. Installing & Running Fluentd
For installation of Fluentd, it provides a bash script that automates the installation process. These Bash scripts are available for:
- ubuntu: Trusty, Precise and Lucid
- debian:Â Jessie, Wheezy and Squeeze.
Simply get and run these scripts using one of the following commands below (based on your operating system)
| 1234567891011121314151617 | ## Ubuntu Trustycurl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh ## Ubuntu Precisecurl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent2.sh | sh ## Ubuntu Lucidcurl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-lucid-td-agent2.sh | sh ## Debian Jessiecurl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent2.sh | sh ## Debian Wheezycurl -L https://toolbelt.treasuredata.com/sh/install-debian-wheezy-td-agent2.sh | sh ## Debian Squeezecurl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | sh |
Once Installed, Let’s start the td-agent.
| 1 | /etc/init.d/td-agent restart |
To make sure you have td-agent running, try the status command
| 1 | /etc/init.d/td-agent status |
4. Put together EFK, Elasticsearch, Fluentd and Kibana stack.
4.1 Get Required Fluentd Plugins
Now let us put all of it together to make it work. First we need a few fluentd plugins installed. Let’s install them by using the following commands.
| 123 | sudo apt-get install make libcurl4-gnutls-dev –yessudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearchsudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer |
4.2 Send Syslog to Elasticsearch via Fluentd
Next, we want to send some log data through fluentd to elasticsearch. In this case we’ll configure fluentd to forward the syslog data to ES. In order to do so, opent file /etc/td-agent/td-agent.conf and replace the existing configuration with the configuration below.
| 123456789101112131415161718192021 | <source> type syslog port 5140 tag system</source> <match system.*.*> type record_reformer tag efkl facility ${tag_parts[1]} severity ${tag_parts[2]}</match> <match efkl> type copy <store> type elasticsearch logstash_format true flush_interval 15s </store></match> |
Now let’s launch fluentd using the following commands.
| 12345 | ## Ubuntusudo service td-agent start ## Mac OS Xsudo launchctl load /Library/LaunchDaemons/td-agent.plist |
We would also require to tell syslog/rsyslog to stream the log data to fluentd. So let’s open the syslog configuration file.
| 12345 | ## Ubuntusudo vim /etc/rsyslog.conf ## Mac OS Xsudo vim /etc/rsyslog.conf |
and add the following line to it. This tells syslog to forward the log data to host 127.0.0.1 which is our local host on port 5140. As fluentd listens to port 5140 by default.
| 1 | *.* @127.0.0.1:5140 |
Now to reload the configuration so that it include our recent changes, Let’s restart the syslog/rsyslog service.
| 123456 | ## Ubuntusudo service rsyslog restart ## Mac OS Xsudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plistsudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist |
Now let’s create an elasticsearch index named kibana where dynamic mapping is enabled.
| 1 | curl -XPUT ‘http://localhost:9200/kibana/’ -d ‘{“index.mapper.dynamic”: true}’ |
Now go to your kibana dashboard by navigating in your web browser to ‘http://0.0.0.0:5601′ and choose the settings tab and enter kibana* in the “index name or pattern” field. Then uncheck “Index contains time-based events” and click on the create button.
![(EFK) Elasticsearch, Fluentd, Kibana Setup – [Step By Step Guide]](https://techstricks.com/wp-content/themes/rehub-theme/images/default/noimage_70_70.png)
