If you’ve read the previous articles within this CoreOS series, you’re already familiar with an essential component within the ecosystem: etcd. This post will introduce two other fundamental components of the CoreOS: fleet and systemd

CoreOS Series Overview

systemd

systemd is an init system providing functionality to manage processes on your machines. It has the basic out-of-the-box features like starting, stopping, and managing processing during system boot. systemd uses units to manage services and requires you to write specific unit files. Those units can then be started on any machine supporting systemd as the process manager. Also, systemd understands multiple unit types. In regard to CoreOS, we’re only interested in target units.

Be aware, that systemd has been around much longer than CoreOS. The CoreOS development team just decided to use it as the process manager of their operation system.

Basic Unit File

Unit files used by systemd have a very specific structure. You have to at least define the content blocks of [Unit] and [Service].

The [Unit] block contains general information about your app like the name and required services to be booted upfront your app starts.

Within [Service] you can define specific commands which get executed during system boot. This part can also have multiple *Start blocks like ExecStartPre, ExecStart, ExecStop. Also, it’s allowed to define a timeout before the start commands are fired (TimeoutStartSec).

[Unit]
Description=Hello World

[Service]
ExecStart=/bin/bash -c "while true; do echo \"Hello, world\"; sleep 1; done"

[X-Fleet]
MachineOf=database.service  

You’ve seen another block defined by [X-Fleet]. This block contains fleet specific configuration telling fleet to schedule services on specific machines, start a single or multiple services side by side or on different machines. You’ve various configuration options and fleet will do the dirty work of handling the service constrains within the cluster. The example unit above requires the database.service to be submitted to the same machine as the Hello World service. You can use keywords like Conflicts to indicate that services must not run on the same machine.

Submitting and starting the unit file from above will result in a stream of Hello, world outputs on the command line. This service won’t be started within a docker container and runs directly on the hosting machine. Don’t worry, we’ll cover the topic on how to submit apps inside a docker container to a CoreOS cluster in a later blog post.

fleet

fleet is the component that ties etcd and systemd together to a distributed init system. To put it simply, fleet takes systemd from a single machine to cluster level. fleet is geared towards system units and not a container orchestration system. If you’re searching for a container scheduling platform, please have a look at Kubernetes.

fleet ships with every CoreOS machine and also brings its own command line utility fleetctl. It’s sufficient to log in to a given CoreOS machine within the cluster and you’re able to manage services via fleetctl. We’ll walk you through the important commands to get an overview of your machines and services running within your cluster.

Using fleet

If you followed this series and have your own CoreOS cluster, you can go ahead and submit the commands to your cluster. If you don’t have a cluster yet, you can follow our guide on how to set up a CoreOS cluster.

List Machines in Cluster

Use fleetctl list-machines to output the list of machines within your cluster. Since we’re using CoreOS cluster size of 5 machines, we expect a list of 5 servers without any metadata (we didn’t define metadata for the servers :))

$ fleetctl list-machines
MACHINE         IP               METADATA  
15789145...     172.17.8.105     -  
18e3c860...     172.17.8.104     -  
4d1f67e1...     172.17.8.101     -  
b057ceee...     172.17.8.102     -  
b1c716d7...     172.17.8.103     -  

Alright, everything is fine and all servers are listed and available by fleet.

In the following, we’re using the hello.service from above, submit and start it within the cluster.

List Unit Files in Fleet Cluster

To get an overview of the unit files available within the cluster, you can run fleetctl list-unit-files. This will print the unit’s name, the assigned hash value, the state and target machine. The target represents the machine where a service is currently scheduled.

$ fleetctl list-unit-files
UNIT            HASH      DSTATE     STATE      TARGET  
hello.service   e55c0ae   launched   launched   4d1f67e1.../172.17.8.101  

List Active Units

If you’re only interested in the state of active units, you can print them using fleetctl list-units. This will give you an overview of the unit and where it’s currently located.

$ fleetctl list-units
UNIT            MACHINE                    ACTIVE   SUB  
hello.service   113f16a7.../172.17.8.103   active   running  

Unit/Service Management

Unit files represent services. That means, the actual service running somewhere within the CoreOS cluster is created from a given unit file. This section will walk you through the service management and how to load unit files into your cluster and also, how to remove them if not required anymore.

Add & Remove Units

You need to specifically submit a unit file to your cluster to be loaded and available to fleet. Alright, we’re starting kind of low-level by creating a unit file based on the content of the exemplary hello.service from fleet’s examples.

Log into one of your CoreOS machines (using SSH) and either git clone the fleet repository to access the examples or just create a new file called hello.service on the machine. Once you’re done, use fleetctl submit hello.service to submit this service to fleet.

fleetctl submit hello.service  

This won’t have any additional output and you can check the status of your hello.service using fleetctl list-unit-files

$ fleetctl list-unit-files
UNIT            HASH      DSTATE     STATE      TARGET  
hello.service   e55c0ae   inactive   inactive   -  

You can remove services from your cluster using fleetctl destroy <name.service>. This will at first stop the service on the executing host, instructing systemd to execute and tear down operations and finally remove the unit file from the cluster. This won’t let you start the service again until you re-submit it.

Load & Unload Units

Submitting units to your cluster makes them available for every machine. However, it doesn’t mean the unit gets loaded on a server or is even prepared for execution. If you just want to load a service to a server without starting it, fleet got you covered with fleetctl load <name.service>.

$ fleetctl load hello.service
Unit hello.service loaded on 15789145.../172.17.8.105

$ fleetctl list-unit-files
UNIT            HASH      DSTATE     STATE      TARGET  
hello.service   e55c0ae   loaded     loaded     15789145.../172.17.8.105  

You see, the hello.service was loaded to the machine 172.17.8.105. The service is still waiting for the kickoff signal to start.

If you want to unload an already loaded service from a machine, use the fleetctl unload <name.serivce> command. This will remove the unit file from the assigned machine.

$ fleetctl unload hello.service
Unit hello.service inactive

$ fleetctl list-unit-files
UNIT            HASH      DSTATE     STATE      TARGET  
hello.service   e55c0ae   inactive   inactive   -  

Start & Stop Units

Submitting units to the cluster and loading them to machines won’t start them. You need to manually start them using fleet’s command line utility: fleetctl start <name.service>. You can directly start units after submitting them to the cluster and skip the load command.

$ fleetctl start hello.service
Unit hello.service launched on 15789145.../172.17.8.105

$ fleetctl list-unit-files
UNIT            HASH      DSTATE     STATE      TARGET  
hello.service   e55c0ae   launched   launched   15789145.../172.17.8.105  

The output after starting the service will show you the machine which now runs the service.

In case you need to stop the app due to whatever reason, use fleetctl stop <name.service>. This will immediately stop the service and return it into loaded state.

$ fleetctl stop hello.service
Unit hello.service loaded on 15789145.../172.17.8.105

$ fleetctl list-unit-files
UNIT            HASH      DSTATE     STATE      TARGET  
hello.service   e55c0ae   loaded     loaded     15789145.../172.17.8.105  

Query Unit Status & Fetch Unit Logs

Submitting, loading, and starting your units on a server might run smooth and no error occur. If you’re like me, you want additionally query the current unit status and verify that the service is running actively and the log outputs are fine.

Fleet’s intuitive command line utility of course has a command to show the current status of your service: fleetctl status <name.service>. This will print out basic information about your service, executed commands and also the latest log entries.

$ fleetctl status hello.service
● hello.service - Hello World
   Loaded: loaded (/run/fleet/units/hello.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Tue 2015-12-15 09:20:38 UTC; 3min 21s ago
 Main PID: 1120 (bash)
   Memory: 372.0K
      CPU: 150ms
   CGroup: /system.slice/hello.service
           ├─1120 /bin/bash -c while true; do echo "Hello, world"; sleep 1; done
           └─1352 sleep 1

Dec 15 09:23:49 core-05 bash[1120]: Hello, world  
Dec 15 09:23:50 core-05 bash[1120]: Hello, world  
Dec 15 09:23:51 core-05 bash[1120]: Hello, world  
Dec 15 09:23:52 core-05 bash[1120]: Hello, world  
Dec 15 09:23:53 core-05 bash[1120]: Hello, world  
Dec 15 09:23:54 core-05 bash[1120]: Hello, world  
Dec 15 09:23:55 core-05 bash[1120]: Hello, world  
Dec 15 09:23:56 core-05 bash[1120]: Hello, world  
Dec 15 09:23:57 core-05 bash[1120]: Hello, world  
Dec 15 09:23:58 core-05 bash[1120]: Hello, world  

The service is in active (running) state, the expected Hello, world logs appear. Everything works like a charm :)

Another way to check the log entries is the journalctl command. Fleet wraps the utility inside the fleetctl and makes the commands available at fleetctl journal. By default, the last 10 log entries are displayed when using fleetctl journal <name.service>.

$ fleetctl journal hello.service
-- Logs begin at Fri 2015-08-07 09:13:17 UTC, end at Tue 2015-12-15 09:27:54 UTC. --
Dec 15 09:27:45 core-05 bash[1120]: Hello, world  
Dec 15 09:27:46 core-05 bash[1120]: Hello, world  
Dec 15 09:27:47 core-05 bash[1120]: Hello, world  
Dec 15 09:27:48 core-05 bash[1120]: Hello, world  
Dec 15 09:27:49 core-05 bash[1120]: Hello, world  
Dec 15 09:27:50 core-05 bash[1120]: Hello, world  
Dec 15 09:27:51 core-05 bash[1120]: Hello, world  
Dec 15 09:27:52 core-05 bash[1120]: Hello, world  
Dec 15 09:27:53 core-05 bash[1120]: Hello, world  
Dec 15 09:27:54 core-05 bash[1120]: Hello, world  

You can specify the number of lines displayed by passing the --lines <number> parameter to the fleetctl journal command.

$ fleetctl journal --lines 3 hello.service
-- Logs begin at Fri 2015-08-07 09:13:17 UTC, end at Tue 2015-12-15 09:29:14 UTC. --
Dec 15 09:29:12 core-05 bash[1120]: Hello, world  
Dec 15 09:29:13 core-05 bash[1120]: Hello, world  
Dec 15 09:29:14 core-05 bash[1120]: Hello, world  

To receive real-time logs, you can follow the journal using the -f parameter. This will output logs directly once they occur. Our example will print a new log message every second.

$ fleetctl journal -f hello.service
-- Logs begin at Fri 2015-08-07 09:13:17 UTC. --
Dec 15 09:30:38 core-05 bash[1120]: Hello, world  
…

What Comes Next

This post drives you through the important commands to create and manage units within a CoreOS cluster. You’ve learned the basics on systemd, the init system and fleet, which leverages systemd and etcd to a cluster-level init system. Also, you’ve got your hands dirty using fleet’s command line utility fleetctl. It provides the necessary commands to manage services on your cluster.

Next week, we’ll finally touch a more complex example by creating and running an app in our CoreOS cluster. We’ll go through the process of creating an app that will be delivered within a docker container and ultimately run inside the cluster.

---

Additional Resources

• More information on unit files
• fleet service examples