Redis Sentinel Reset

Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention certain kinds of failures. Redis Sentinel also provides other collateral tasks such as monitoring, notifications and acts as a configuration provider for clients.

1. Redis Cli Sentinel
2. Redis Sentinel Cluster Setup
3. Python Redis Sentinel
4. Redis Sentinel Error Connection Reset By Peer
5. Redis Sentinel Configuration

1. Redis connection reset (ECONNRESET) Connecting to Redis via the CLI. Configuring Redis Using an alternate local Redis instance. If you use Redis Sentinel, do not activate client support for SSL. Redis 6 supports SSL, and you can configure it to work with GitLab only as an external service.
2. # Host and port we will listen for requests on bind 127.0.0.1 port 16380 # # 'redis-cluster' is the name of our cluster # # each sentinel process is paired with a redis-server process # sentinel monitor redis-cluster 127.0.0.1 6380 2 sentinel down-after-milliseconds redis-cluster 5000 sentinel parallel-syncs redis-cluster 1 sentinel failover.

Redis Sentinel provides a simple and automatic high availability (HA) solution for Redis. If you’re familiar with how MongoDB elections work, this isn’t too far off. To start, you have a given master replicating to N number of slaves. From there, you have Sentinel daemons running, be it on your application servers or on the servers Redis is running on. These keep track of the master’s health.

If a Sentinel detects that a master is non-responsive, it will broadcast a SDOWN (Subjectively down) message to the other sentinels. Then, once a quorum is reached that a master is down, it will broadcast an ODOWN (Objectively down), and the new master will be elected. Since you need a quorum, or majority, of sentinels to agree to reach the ODOWN state, it’s always best practice to have an odd number of Sentinels running to avoid ties.

Note: it is highly recommended to use a version of Redis from the 2.8 branch or higher for best performance with Sentinel.

How it works

Sentinels handle the failover by re-writing config files of the Redis instances that are running. Let’s go through a scenario:

Say we have a master “A” replicating to slaves “B” and “C”. We have three Sentinels (s1, s2, s3) running on our application servers, which write to Redis. At this point “A”, our current master, goes offline. Our sentinels all see “A” as offline, and send SDOWN messages to each other. Then they all agree that “A” is down, so “A” is set to be in ODOWN status. From here, an election happens to see who is most ahead, and in this case “B” is chosen as the new master.

Redis Cli Sentinel

The config file for “B” is set so that it is no longer the slave of anyone. Meanwhile, the config file for “C” is rewritten so that it is no longer the slave of “A” but rather “B.” From here, everything continues on as normal. Should “A” come back online, the Sentinels will recognize this, and rewrite the configuration file for “A” to be the slave of “B,” since “B” is the current master.

Configuration

Configuring Sentinels isn’t as hard as one would think. In fact, one of the most difficult things is choosing where to place your Sentinel processes. I personally recommend running them on your app servers if at all possible. Presumably if you’re setting this up, you’re concerned about write availability to your master. As such, Sentinels provide insight to whether or not your application server can talk to the master. You are of course welcome to run Sentinels on your Redis instance servers as well.

To start with the configuration step, please reference the example file found here. This is an example sentinel.conf found with Redis 2.8.4 on Ubuntu 14.04, but should work with any 2.8.x version of Redis. I’ve taken the liberty of adding two lines to the top that I like to use in practice:

This puts the sentinel process in daemonize mode, and logs all it’s messages to a log file instead of stdout.

There are a lot of configurable options in here, and most are commented very well. However, for this post we’ll focus on just two.

The most important part is telling the Sentinels where your current master resides. This is referenced in this line:

This tells the Sentinel to monitor “mymaster” (this is an arbitrary name, feel free to name it as you see fit) and a given IP on a given port, as well as how many Sentinels are required to meet a quorum for failover (the minimum being 2). The parts you will likely want to change here are the IP address of your master and it’s port, if it’s not running on the standard port 6379.

Next, you may want to change the following line:

This is the amount of time you would like a sentinel to wait before it declares a master in SDOWN. The default is 30 seconds, I typically like to lower this a bit to 10 seconds. You don’t want to reduce this too low; otherwise you may have issues with failovers happening too often.

Feel free to take a look at some of the other options. One that may interest a lot of users is the notification script, if you like to keep track of failovers when they happen.

Once you have your sentinel.conf configured as you see fit, start the daemon with the following command:

Testing Failover

Once you have all your sentinels online, it’s possible to do a dry run for failover to make sure it’s all configured correctly.

First thing’s first. Connect to your Sentinel via the redis-cli:

If you’d like to receive some information about sentinel, simply run this command:

This will give you information, such as who is the current master, how many slaves it has, and how many sentinels are monitoring it.

To test failover, simply execute:

This will force an ODOWN on the current master and cause a failover to happen. Shortly after, if you run the “INFO” command again, you should now see a new master listed.

Conclusion

Hopefully this has been helpful to demystify Redis and Sentinel. Should you have any questions at all, feel free to post below!

• Troubleshooting

Using an alternate local Redis instance

Omnibus GitLab includes Redis by default. To direct the GitLabapplication to your own locally running Redis instance:

1. Edit /etc/gitlab/gitlab.rb:

2. Reconfigure GitLab for the changes to take effect:

Making the bundled Redis reachable via TCP

Use the following settings if you want to make the Redis instancemanaged by Omnibus GitLab reachable via TCP:

1. Edit /etc/gitlab/gitlab.rb:

2. Save the file and reconfigure GitLab for the changes to take effect:

Setting up a Redis-only server using Omnibus GitLab

If you’d like to set up Redis in a separate server than the GitLab application,you can use thebundled Redis in Omnibus GitLab.

Running with multiple Redis instances

See https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html#running-multiple-redis-clusters.

Redis Sentinel

See https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html.

Using Redis in a failover setup

See https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html.

Redis Sentinel Cluster Setup

Using Google Cloud Memorystore

Python Redis Sentinel

Google Cloud Memorystore does not support the Redis CLIENTcommand.By default Sidekiq will attempt to set the CLIENT for debuggingpurposes. This can be disabled via this configuration setting:

Increasing the number of Redis connections beyond the default

By default Redis will only accept 10,000 client connections. If you needmore that 10,000 connections set the maxclients attribute to suit your needs.Be advised that adjusting the maxclients attribute means that you will also needto take into account your systems settings for fs.file-max (for example sysctl -w fs.file-max=20000)

Tuning the TCP stack for Redis

The following settings are to enable a more performant Redis server instance. tcp_timeout isa value set in seconds that the Redis server waits before terminating an idle TCP connection.The tcp_keepalive is a tunable setting in seconds to TCP ACKs to clients in absence ofcommunication.

Setting the Redis Cache instance as an LRU

Using multiple Redis instances allows you to configure Redis as a LeastRecently Used cache. Note you should onlydo this for the Redis cache instance; the Redis queues and shared state instancesshould never be configured as an LRU, since they contain data (e.g. Sidekiqjobs) that is expected to be persistent.

To cap memory usage at 32GB, you can use:

Using Secure Sockets Layer (SSL)

Redis 5.x does NOT support SSL out of the box. However, you can encrypt aRedis connection using stunnel.AWS ElasticCache also supports Redis over SSL.

Support for SSL has the following limitations:

• Omnibus GitLab doesn’t include stunnel or other tools to provide encryptionfor the Redis server. However, GitLab does provide client support by usingthe rediss:// (as opposed to redis://) URL scheme.
• Omnibus GitLab bundles Redis Sentinel 5.0.x which does NOT support SSL.If you use Redis Sentinel, do not activate client support for SSL.Redis 6 supports SSL, and you canconfigure it to work with GitLab only as anexternal service.

To activate GitLab client support for SSL:

1. Add the following line to /etc/gitlab/gitlab.rb:

2. Reconfigure GitLab for the changes to take effect:

SSL certificates

If you’re using custom SSL certificates for Redis, be sure to add themto the trusted certificates.

Renamed commands

By default, the KEYS command is disabled as a security measure.

If you’d like to obfuscate or disable this command, or other commands, edit the redis['rename_commands'] setting in /etc/gitlab/gitlab.rb to look like:

• OTHER_COMMAND is the command you want to modify
• VALUE should be one of:
  1. A new command name.
  2. ', which completely disables the command.

To disable this functionality:

1. Set redis['rename_commands'] = {} in your /etc/gitlab/gitlab.rb file
2. Run sudo gitlab-ctl reconfigure

Redis Sentinel Error Connection Reset By Peer

Lazy freeing

Redis 4 introduced lazy freeing. This can improve performance when freeing large values.

This setting defaults to false. To enable it, you can use:

Troubleshooting

x509: certificate signed by unknown authority

This error message suggests that the SSL certificates have not beenproperly added to the list of trusted certificates for the server. Tocheck whether this is an issue:

1. Check Workhorse logs in /var/log/gitlab/gitlab-workhorse/current.

2. If you see messages that look like:

   The first line should show rediss as the scheme with the addressof the Redis server. The second line indicates the certificate isnot properly trusted on this server. See the previous section.

3. Verify that the SSL certificate is working via these troubleshootingsteps.

NOAUTH Authentication required

A Redis server may require a password sent via an AUTH message beforecommands are accepted. A NOAUTH Authentication required error messagesuggests the client is not sending a password. GitLab logs may helptroubleshoot this error:

1. Check Workhorse logs in /var/log/gitlab/gitlab-workhorse/current.

2. If you see messages that look like:

3. Check that the Redis client password specified in /etc/gitlab/gitlab.rb is correct:

4. If you are using the Omnibus-provided Redis server, check that the server has the same password:

Redis connection reset (ECONNRESET)

If you see Redis::ConnectionError: Connection lost (ECONNRESET) in theGitLab Rails logs (/var/log/gitlab-rails/production.log), this mightindicate that the server is expecting SSL but the client is notconfigured to use it.

1. Check that the server is actually listening to the port via SSL.For example:

2. Check /var/opt/gitlab/gitlab-rails/etc/resque.yml. Youshould see something like:

3. If redis:// is present instead of rediss://, the redis_sslparameter may not have been configured properly, or the reconfigurestep may not have been run.

Connecting to Redis via the CLI

When connecting to Redis for troubleshooting you can use:

• Redis via Unix domain sockets:

• Redis via TCP:

• Password to authenticate to Redis if required:

Help & feedback

Get Help

If you didn't find what you were looking for, search the docs.

If you want help with something specific, and could use community support, post on the GitLab forum.

Redis Sentinel Configuration

For problems setting up or using this feature (depending on your GitLab subscription).