Logstash installation

This article is about logstash install on a Debian/Ubuntu server. A dedicated user will be created, as well as the folder tree which will be reserved to production files (logstash instances will run as services), and a test environment will be setup.

The dedicated user will be named runner in this article, but feel free to rename it. This user will have access to sensible files (such as system logs), thus it is advised to give him a very secure password.

The main folders are :

  • /etc/logstash : services configuration files
  • /opt/logstash : files, libraries and plugins for logstash
  • ~/bin, ~/etc : user’s home directory, contaning scripts allowing to easily run logstash in test and dev mode
  1. Create the user

    The user must have access to system log files. On Ubuntu, it means that it has to belong to the adm group.

    You may check that it’s ok if it can read apache logs (for instance).

  2. Download and install Logstash binaries

    Grab the last jar on official site (http://logstash.net), then put it in /opt/logstash.

    Lastly, we need to run logstash easily from the command line to test our first config files. A way to do it is to create following script in runner user’s ~/bin folder :

    From the home folder, it will then be possible to test ~/etc/test.conf with bin/run.sh test.

  3. Redis install

    Logstash needs a fast object buffer on the server to store log datas before processing. Redis does the job perfectly (more infos on http://redis.io/). On may follow the procedure explained on the site, but on Ubuntu a simple  :

    will suffice.

  4. ElasticSearch Install

    ElasticSearch will store and index all the messages.

    You can install it as a service easily with  :

    (Then copy this init file)

    And finish the install with :

    Important : Elasticsearch needs to be able to open a lot of files. At least much more than the limit set by default in Linux distribs. Thus, you must follow this guide, to make it work, or you’ll get error messages like “Index failed for [...]” or “Too many open files”.

    More informations about Elasticsearch and log collecting can be found here.

  5. Testing all this

    Logstash test

    This command :

    must give something like (only first line matters):

    Redis test

    Run the redis-cli command to enter interactive mode. On may then test that the llen (returns the length of a list) command gives 0 :

    Elasticsearch test

    If the elasticsearch server is running, it should answer to the 9200 port. Just type curl which should give something like :

You’re done ! The main components are setup and ready. Next step is : a first flux configuration.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© Crocoware technical blog