RobustIRC Admin Guide

This chapter describes how RobustIRC behaves when faced with various failure scenarios. We assume that each RobustIRC node is running on a different machine, ideally on a different physical machine, as otherwise a failure of the host machine will lead to not only a single-node failure, but a multiple-node failure.

The number of nodes necessary to establish quorum depends on the number of nodes in the network. As an example, in a 3-node network, you need \(\left\lfloor \frac{n}{2} \right\rfloor+1 = 2\) nodes to reach quorum. In a 5-node network, you need \(\left\lfloor \frac{n}{2} \right\rfloor+1 = 3\) nodes to reach quorum.

If the IRC network you’re running is small, chances are you don’t have a good idea of how many messages/second your network is handling. There are a couple of ways to get estimates:

mibbit has a list of networks, and you can also see their non-secret channels. Looking at the channels of the biggest networks, there are typically about 300 users in each channel. Of course, there are outlier channels with 3000+ users, which typically host warez or offer some other kind of automated content.

freenode cites 320 GiB/month as an estimate for the traffic required to run a server in the freenode network. If you assume an average message size of 100 bytes (the maximum being 512 bytes), this translates to roughly \(\frac{320 * 1024^3}{30 * 24 * 60 * 60} / 100 = 1325\) messages/s.

netsplit.de has a list of the biggest networks (excluding some that don’t want to be counted, like freenode). The number of users for the top 10 range from 10,000 to 50,000 users.

We also monitored IRCNet for a week and observed an average number of messages of about 2000 messages/s.

TODO

TODO

The RobustIRC bridge is a program which bridges (translates) between the RobustIRC protocol and standard IRC, as defined per RFC2812.

There are two places where the bridge can run, each with their own benefits and drawback:

For running a RobustIRC network, you need at least 3 different servers. While technically you can run 3 RobustIRC processes on the same server, that doesn’t make a lot of sense: the point of RobustIRC is to be resilient to certain failures, and when you put all your RobustIRC processes into the same failure domain, you don’t achieve that.

The ideal configuration for RobustIRC is to have each server in an entirely separate failure domain, i.e. on a different machine, in a different rack, with different power, with different network connectivity, in a different physical datacenter. For traditional hosting this typically means chosing different hosting providers, with cloud providers it means running in different availability zones.

That said, pay attention to the network latency between your failure domains. See Latency for how to determine the network latency and what it means.

With regards to the number of servers, a network of 3 servers continues to work when 1 server is unreachable. A network of 5 servers continues to work when 2 servers are unreachable, and so on. In general, a network of \(n\) servers continues to work when \(n - (\left\lfloor \frac{n}{2} \right\rfloor+1)\) servers are unreachable.

Therefore, always use an odd number of servers in your network. Even numbers don’t increase the reliability, so they only increase the message commit latency due to increased quorum size.

RobustIRC provides a status page that you can access with your web browser. Simply connect to the host/port on which the server is listening (see the -peer_addr flag) and use “robustirc” as user with -network_password as password when asked to authenticate.

As an example, assume you’re running a node with -peer_addr=alp.robustirc.net:60667 and -network_password=topsecret. The URL for the status page is https://robustirc:[email protected]:60667/

The same port is used for the status page, communication between the nodes and communication with the clients.

When bringing up your network for the first time, you need to run each node with a special command line parameter: the first node you start needs the -singlenode flag, and all other nodes need the -join=<address> flag, where <address> is the -peer_addr value of the first node.

Bootstrapping is finished once the network converged, meaning all Status pages display a node state of either “Follower” or “Leader” (as opposed to “Candidate” or “<nil>”).

Once bootstrapping is finished, be sure to remove the -singlenode and -join flags! Afterwards, see Updates for how to use robustirc-rollingrestart to restart each node once. Remember to use systemctl daemon-reload to make changes to service files effective if you’re using systemd. Neglecting to remove the flags could lead to data loss or split brain scenarios (for -singlenode) or unavailability after the list of peers changes (for -join).

Again, NEVER use the -singlenode flag after the initial bootstrapping.

We strongly recommend using Docker since it makes running RobustIRC much easier.

The Docker container we provide always has a “stable” tag pointing at the most recently released version that was tested for at least 7 days on the robustirc.net network. We recommend you follow the “stable” tag, but if you prefer, you can directly follow the “latest” tag instead.

In order to update to a newer version, all you need to do is run the newer RobustIRC binary. You will never need to manually migrate the on-disk data. This allows you to do automatic or semi-automatic updates: you could use an ExecStartPre directive to automatically pull the new docker container as outlined in the Docker section above. If you chose to not use docker, the equivalent action would be to install the new RobustIRC binary on all nodes.

To add a new node to the network, for example in preparation of decomissioning a server, start it with specifying the -peer_addr value of any healthy server in the -join flag.

As an example, if your network consists of these three healthy nodes:

…and you wanted to introduce a node running with -peer_addr=libri.robustirc.net:60667, you can start that node with -join=alp.robustirc.net:60667.

As explained in Bootstrapping, only specify -join once, then remove the flag again!

To remove an existing node from the network, for example before decomissioning a server, use the robustirc-removepeer tool, e.g.:

The robustirc-removepeer tool has safety checks in place to make sure that the network can still reach quorum after removing the node you ask it to remove.

Since RobustIRC was released, the hashicorp/raft package which RobustIRC is using has introduced a new protocol version.

To upgrade your network from Raft protocol version 1 (the default) to version 3 (the new version), use the following steps:

In case your network becomes partitioned, you have two options:

Note that it is only safe to remove and add nodes as long as the network still has a raft leader, i.e. as long as the majority of nodes are healthy. The robustirc-removepeer tool described in section Removing nodes automatically checks that for you.

In particular, DO NOT USE the -singlenode and -join flags to introduce new nodes to the network, or you might end up in a split-brain scenario.

If you want to provide a stable network, you should strive to keep all nodes of the network healthy. In technical terms, you want to keep your capacity at least at n+1, where n is the number of nodes you absolutely need. In an example with 3 nodes, while the network still functions with only 2 nodes being healthy, it is advisable to always try to have 3 nodes healthy, otherwise a single hardware failure will make your network freeze.

In order to detect unhealthy nodes, we recommend using Prometheus. Please refer to the Prometheus documentation for details.

RobustIRC exports Prometheus metrics by default and we provide example config files:

For the robustirc.net network, we use Prometheus in combination with Pushover to get alerted about problems and we try to fix them swiftly.

As for graphs, we provide an example dashboard JSON file for Grafana.

See the next subsections for detailed descriptions of each of these parameters.

If you run your network really hot and notice that leadership is often lost, you need to increase these timeouts to allow for more slack. Otherwise, you can ignore this entire section.

Timeouts must fulfill this relation:

5ms < LeaderLeaseTimeout ≤ HeartbeatTimeout ≤ ElectionTimeout

default: 500ms (LeaderLeaseTimeout) ≤ 1000ms (HeartbeatTimeout) ≤ 1000ms (ElectionTimeout)

As a rule of thumb, figure out the latency of the slowest network link between your nodes, e.g. by using ping(8). Then, set HeartbeatTimeout to 10 times that latency so that brief network latency spikes are not a problem. Set all the other timeouts to the same value.

TODO

TODO

TrailingLogs is the number of Raft log entries which are kept after taking a snapshot. If you have enough log entries to cover a brief node failure (e.g. a flaky network), Raft does not need to send an entire snapshot over the network, so recovery of the failed node may be quicker.

In case you configure this parameter too low, recovery after a node failure may consume more bandwidth and may take longer.

In case you configure this parameter too high, the disk usage of RobustIRC will be higher, as log compactions will occur less frequently.

As a rule of thumb: look at your network’s messages/second, multiply that by the time of a typical outage, e.g. 5 minutes.

If leadership flaps, consult the Raft Configuration section to verify that your timeouts are set appropriate for the network situation at hand.

In case your timeouts are configured correctly, it’s worth checking whether the leadership flaps are correlated in time with periods of high CPU or disk utilization. In that case, if possible, try increasing the available resources. For example, allow the VM in which RobustIRC runs to use a second CPU core in case only one was allocated.