A simple Logstash Redis filter powered by ruby



Hi there! It’s been a while 👋👋

I’ve been working on a lot of behind the scenes development for HoneypotDB, including changes to the ingest process and Logstash pipelines.

One neat new HoneypotDB feature is an IP scoring system, enabling the ability for IP addresses to be graded on how dodgy they’re being, with scores mapped to the MITRE ATT&CK framework!

To prevent the scoring system for being spanned, IP addresses will be added to a Redis based queue (really a list), before being bull out by score workers for processing.

Unfortunately, there isn’t an official Redis filter plugin, we have input and output, but no filters. Community filters are there, but they haven’t been updated in a while.

Luckily, Logstash allows you to execute arbitrary Ruby code as a Logstash filter, which is perfect.

The code

So, getting straight to it, you first need to install the redis ruby gem into Logstash, this can be achieved with the below command, and then restarting Logstash for good measure.

/usr/share/logstash/bin/ruby -S gem install redis

I send my events to Logstash as JSON, if you want to do the same you’ll need the json ruby gem to property generate it. It an be installed into Logstash in the same way.

/usr/share/logstash/bin/ruby -S gem install json

Once your gems have been installed and logstash restarted, you can add your filter.

The below ruby code is actually really simple, first we initialise the ruby filter, pulling in the redis gem and instancing a new redis client.

require "redis"; require "json"; $rc = "redis://:[email protected]_IP:6379/0")

We then save the data from our event to a ruby variable called ‘data’. This variable stores this data as a dictionary (a hash in ruby), so can be exported as JSON easily.

data = { "example_key_1" => event.get("[example][field_1]"), "example_key_2" => event.get("[example][field_2]") }

Using this data variable, we can then push it to redis. This example uses the Redis lpush command to add the event to a list under the name REDIS_KEY.

You can do pretty much any redis command, including setting data to specific keys etc. Take a look at the GitHib for redis-rb and Redis command reference for more info.

A full logstash filter example is below:

filter  {

    ruby {
        init => 'require "redis"; require "json"; $rc = "redis://:[email protected]_IP:6379/0")'
        code => 'data = { "example_key_1" => event.get("[example][field_1]"), "example_key_2" => event.get("[example][field_2]") };$rc.lpush("REDIS_KEY", data.to_json)'