What is ELK Stack?

ELK stack is combination of three open source projects Elasticsearch, Logstash & Kibana. Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. Logstash is a log aggregator that collects and processes data from multiple sources simultaneously, transforms it, and then sends it to data store like Elastisearch. Kibana provides a user interface, allowing users to visualize, query and analyze their data via graphs and charts.

Docker …

Docker provides the ability to package and run an application in a loosely isolated environment called a container. Docker is a bit like a virtual machine. But unlike a virtual machine, rather than creating a whole virtual operating system, Docker allows applications to use the same Linux kernel as the system that they’re running on and only requires applications to be packaged and shipped.

Setup Ubuntu on Oracle VM VirtualBox

Like most other corporate laptops mine one is also running windows OS and I chose to go for Ubuntu budgie, beautiful looking OS among all other Ubuntu flavors. Installation procedure is almost same for any other linux distro so you aren’t forced to go for Ubuntu or Ubuntu budgie to follow the tutorial.

  1. Download virtualbox from this link and install it.
  2. Download Ubuntu iso here
  3. Follow this tutorial to install downloaded ubuntu on virtualbox

At this step your VM should be up & running. Let me show you how latest Ubuntu budgie distro looks like

Ubuntu Budgie 20.04 LTS
Ubuntu Budgie 20.04 LTS

Setup Docker

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update

Now we’re ready for docker installation.

apt-cache policy docker-ce

You’ll see output like this, although the version number for Docker may be different:

docker-ce:
  Installed: (none)
  Candidate: 5:19.03.9~3-0~ubuntu-focal
  Version table:
     5:19.03.9~3-0~ubuntu-focal 500
        500 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages

As you can notice from the output above, docker is not installed. Command below will install docker -

sudo apt install docker-ce

You’ve installed docker without any error at this point and it must be running. Let’s find out docker daemon status -

sudo systemctl status docker
Docker status
Docker is up & running

ELK setup using Docker

Specific version combinations of Elasticsearch, Logstash and Kibana can be pulled using tags. The available tags are listed on Docker Hub’s sebp/elk image page or GitHub repository page. I visited Docker Hub link given above and could see various elk versions listed as tags. A screenshot below for reference -

By default, if no tag is indicated (or if using the tag latest), the latest version of the image will be pulled. Let’s go ahead and setup ELK 7.8.0

sudo docker pull sebp/elk:780

docker will begin pulling stack and it will take some time to complete.

docker in progress

You can verify if image is pulled successfully using docker image ls

Now we are ready to spin up the image. Use below command to run the stack:

sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk:780

When I tried above command, I received long list of error complaining about low virtual memory (screenshot below for reference):

docker elk stack error
Error running docker elk image

On googling I stumbled upon this stackoverflow question and resolution is to increase vm.max_map_count.

Insert the new entry into the /etc/sysctl.conf file with the required parameter:

vm.max_map_count = 262144

it makes changes permanent. In order to make this change effective in current terminal session, we also need to execute below command :

sysctl -w vm.max_map_count=262144

This should probably solve the issue, let’s give it a try again and launch it -

sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk:780

If all 3 components (ELK) get launched successfully, we can access Kibana in any of the web browser by visiting http://localhost:5601/

Kibana running via Docker on ubuntu 20.04 LTE