CoreOS is suited for all kinds of use cases. In the previous blog posts we’ve shown you how to submit and run a Node.js application inside a Docker container on a CoreOS cluster and also how to scale this Node.js app to multiple instances across the cluster.

This guide will show you how to run multiple database instances on a CoreOS cluster. For that, we’re going to kick off a RethinkDB cluster.

CoreOS Series Overview

Preparation

This guide is a hands-on guide. That means, you can directly use the code snippets and commands on your own CoreOS cluster to test the functionality. To do so, you need to have a running CoreOS cluster and at least (SSH) access to one of the machines within the cluster. If you don’t have a CoreOS cluster set up yet, follow this guide on how to create a new one and move on when you’re done.

Requirements

• a running CoreOS cluster

If you’ve read the previous articles within this series of CoreOS posts, you may think that we need a Docker container image that we can submit and execute on the CoreOS cluster. You’re right: we need a container image. Nonetheless, the RethinkDB team already did the work for us and because we just need a pure database, it’s totally sufficient to use the plain RethinkDB container from Docker Hub.

Create Unit Files

The hands-on part is starting now. We’re jumping right in and create the required unit files. We’ll create two unit files: the discovery service and the actual RethinkDB instances. With the help of the discovery services, we’ll spread the word about the instances and announce them to etcd by saving the unit id.

Discovery Unit File: rethinkdb-discovery@.service

We use the discovery service to announce each RethinkDB instances to etcd and save instances private IP addresses. This will be used by the actual RethinkDB instance unit to join the actual cluster.

Within the [Unit] block, we have the description and a BindsTo directive. With the binding, we tell fleet to automatically start the rethinkdb-discovery-service when starting the rethinkdb-service.

[Unit]
Description=Announce RethinkDB@%i service  
BindsTo=rethinkdb@%i.service

[Service]
EnvironmentFile=/etc/environment  
ExecStart=/bin/sh -c "while true; do etcdctl set /announce/services/rethinkdb%i ${COREOS_PRIVATE_IPV4} --ttl 60; sleep 45; done"  
ExecStop=/usr/bin/etcdctl rm /announce/services/rethinkdb%i

[X-Fleet]
MachineOf=rethinkdb@%i.service  

Additionally, we set the constraint that the discovery services runs aside the rethinkdb service with the same id (specified by the %i placeholder.

RethinkDB Instance Unit File: rethinkdb@.service

The next thing on our list is the creation of the rethinkdb@.service file. We’ll use this unit file to bootstrap multiple RethinkDB instances on the CoreOS cluster and let them join each other to build a RethinkDB cluster itself. RethinkDB itself will handle all the leader election, since version 2.2, RethinkDB comes with automatic failover in error situations. That handles all the hassle for us. RethinkDB selects the master/slaves by itself.

Within the [Unit] block, we first set the description and afterwards the required services. We tell fleet to start the rethinkdb services after docker and etcd2 services and also launch the rethinkdb-discovery service first, if not already done.

The [Service] block contains the import commands to run the RethinkDB docker container, expose the required ports for RethinkDB’s web UI, client and cluster connections (8080, 28015, 29015). Also, we pass the --cannonical-address <host:port> to tell RethinkDB the correct host address. Starting multiple RethinkDB instances in containers or behind firewalls will cause issues otherwise.

Further, we’re binding to all hosts IP addresses so that the instance is accessible via private IPs. Please keep in mind, that this is a security issue and you should restrict access to your RethinkDB instances via iptables. For now, we’ll accept the downsides.

[Unit]
Description=RethinkDB@%i service  
After=docker.service  
After=etcd2.service  
Requires=rethinkdb-discovery@%i.service

[Service]
EnvironmentFile=/etc/environment  
TimeoutStartSec=0  
ExecStartPre=-/usr/bin/docker kill rethinkdb%i  
ExecStartPre=-/usr/bin/docker rm rethinkdb%i  
ExecStartPre=-/usr/bin/mkdir -p /home/core/docker-volumes/rethinkdb  
ExecStartPre=/usr/bin/docker pull rethinkdb  
ExecStart=/bin/sh -c ‚/usr/bin/docker run --name rethinkdb%i   \  
    -p ${COREOS_PRIVATE_IPV4}:8080:8080                        \
    -p ${COREOS_PRIVATE_IPV4}:28015:28015                      \
    -p ${COREOS_PRIVATE_IPV4}:29015:29015                      \
    -v /home/core/docker-volumes/rethinkdb/:/data/             \
    rethinkdb rethinkdb --bind all                             \
    --canonical-address ${COREOS_PRIVATE_IPV4}                 \
    $(/usr/bin/etcdctl ls /announce/services |                 \
        xargs -I {} /usr/bin/etcdctl get {} |                  \
        sed s/^/"--join "/ | sed s/$/":29015"/ |               \
        tr "\n" " ")’

ExecStop=/usr/bin/docker stop rethinkdb%i

[X-Fleet]
Conflicts=rethinkdb@*.service  

Last, but not least: we tell fleet within the [X-Fleet] block to only start one RethinkDB instance per machine. If there’s already an active RethinkDB unit on a given machine, fleet won’t launch another one.

Run Multiple RethinkDB Units on CoreOS Cluster

Alright, now that we have the unit files in place, we can submit them to the CoreOS cluster and start the RethinkDB instances using fleet (actually we’re using fleet’s command line utility fleetctl).

$ fleetctl submit rethinkdb@.service rethinkdb-discovery@.service
Unit rethinkdb@.service inactive  
Unit rethinkdb-discovery@.service inactive

$ fleetctl list-unit-files
UNIT                           HASH      DSTATE      STATE      TARGET  
rethinkdb-discovery@.service   76b05ae   inactive    inactive   -  
rethinkdb@.service             f414eb6   inactive    inactive   -  

With fleetctl list-unit-files we verify that both unit files were correctly submitted to fleet and they’re available to be loaded and start units on a given machine. This is what we’re doing now: starting five instances of RethinkDB on our cluster. Therefore, we pass our desired range of one to five ({1..5}) machines to the fleetctl start … command. Fleet automatically translates the range defined after the @ symbol to a single digit and results in unit names rethinkdb1, rethinkdb2, … ,rethinkdb5.

$ fleetctl start rethinkdb@{1..5}.service
Unit rethinkdb@1.service inactive  
Unit rethinkdb@2.service inactive  
Unit rethinkdb@3.service inactive  
Unit rethinkdb@4.service inactive  
Unit rethinkdb@5.service inactive  
Unit rethinkdb@5.service launched on c25d4773.../172.17.8.102  
Unit rethinkdb@3.service launched on 5f5d1d27.../172.17.8.103  
Unit rethinkdb@4.service launched on ae6e1cc3.../172.17.8.101  
Unit rethinkdb@2.service launched on 5128f5bb.../172.17.8.105  
Unit rethinkdb@1.service launched on 31ff0ca0.../172.17.8.104  

Using fleetctl start … also tells fleet to submit the units to machines before starting them. Fleet selects the machines for execution and prints the respective machine ID and IP address. Our CoreOS cluster consists of five machines and due to our decision to start five RethinkDB instances, each machine will be selected by fleet to run the database.

Result

We’ve started the RethinkDB units and can finally verify if the machines within our cluster run a RethinkDB instance. Because we’ve started five units and have five machines within our CoreOS cluster, each machine hosts and runs a RethinkDB database. Let’s print the list of units managed by fleet using fleetctl last-units.

$ fleetctl list-units
UNIT                             MACHINE                     ACTIVE   SUB  
rethinkdb-discovery@1.service    31ff0ca0.../172.17.8.104    active   running  
rethinkdb-discovery@2.service    5128f5bb.../172.17.8.105    active   running  
rethinkdb-discovery@3.service    5f5d1d27.../172.17.8.103    active   running  
rethinkdb-discovery@4.service    ae6e1cc3.../172.17.8.101    active   running  
rethinkdb-discovery@5.service    c25d4773.../172.17.8.102    active   running  
rethinkdb@1.service              31ff0ca0.../172.17.8.104    active   running  
rethinkdb@2.service              5128f5bb.../172.17.8.105    active   running  
rethinkdb@3.service              5f5d1d27.../172.17.8.103    active   running  
rethinkdb@4.service              ae6e1cc3.../172.17.8.101    active   running  
rethinkdb@5.service              c25d4773.../172.17.8.102    active   running  

Things went like a charm and every RethinkDB unit is already in active/running status.

Because you choose RethinkDB to install on the cluster, you already know its web UI. We can choose any machine from our CoreOS cluster and visit the respective IP address in the browser.

What Comes Next

Within this post, you’ve learned how to prepare the unit files for RethinkDB instances. Further, you’ve started a RethinkDB cluster consisting of five machines on a CoreOS cluster.

With this post, we end this eight-part series of CoreOS posts. If you like the posts, please leave a comment, tweet us or just reach out the way you like most :) We’re very happy about any feedback and comments. We hope you’ve enjoyed and learned a lot throughout this series. If we recognize high interest in CoreOS posts, we’ll definitely continue and publish new posts within the future. So long, stay motivated and keep rocking!

---

Additional Resources

• RethinkDB
• RethinkDB Docker Image
• RethinkDB and CoreOS by Neumino