If you’ve read the previous articles within this CoreOS series, you’re already familiar with an essential component within the ecosystem: etcd. This post will introduce two other fundamental components of the CoreOS: fleet and systemd
CoreOS Series Overview
- Fleet & How to Manage Services Within Your Cluster
- Run Your Node.js App on Cluster
systemd is an init system providing functionality to manage processes on your machines. It has the basic out-of-the-box features like starting, stopping, and managing processing during system boot. systemd uses units to manage services and requires you to write specific unit files. Those units can then be started on any machine supporting systemd as the process manager. Also, systemd understands multiple unit types. In regard to CoreOS, we’re only interested in target units.
Be aware, that systemd has been around much longer than CoreOS. The CoreOS development team just decided to use it as the process manager of their operation system.
Basic Unit File
Unit files used by systemd have a very specific structure. You have to at least define the content blocks of [Unit] and [Service].
The [Unit] block contains general information about your app like the name and required services to be booted upfront your app starts.
Within [Service] you can define specific commands which get executed during system boot. This part can also have multiple
*Start blocks like
ExecStop. Also, it’s allowed to define a timeout before the start commands are fired (
[Unit] Description=Hello World [Service] ExecStart=/bin/bash -c "while true; do echo \"Hello, world\"; sleep 1; done" [X-Fleet] MachineOf=database.service
You’ve seen another block defined by [X-Fleet]. This block contains fleet specific configuration telling fleet to schedule services on specific machines, start a single or multiple services side by side or on different machines. You’ve various configuration options and fleet will do the dirty work of handling the service constrains within the cluster. The example unit above requires the
database.service to be submitted to the same machine as the
Hello World service. You can use keywords like
Conflicts to indicate that services must not run on the same machine.
Submitting and starting the unit file from above will result in a stream of
Hello, world outputs on the command line. This service won’t be started within a docker container and runs directly on the hosting machine. Don’t worry, we’ll cover the topic on how to submit apps inside a docker container to a CoreOS cluster in a later blog post.
fleet is the component that ties etcd and systemd together to a distributed init system. To put it simply,
systemd from a single machine to cluster level.
fleet is geared towards system units and not a container orchestration system. If you’re searching for a container scheduling platform, please have a look at Kubernetes.
fleet ships with every CoreOS machine and also brings its own command line utility
fleetctl. It’s sufficient to log in to a given CoreOS machine within the cluster and you’re able to manage services via
fleetctl. We’ll walk you through the important commands to get an overview of your machines and services running within your cluster.
If you followed this series and have your own CoreOS cluster, you can go ahead and submit the commands to your cluster. If you don’t have a cluster yet, you can follow our guide on how to set up a CoreOS cluster.
List Machines in Cluster
fleetctl list-machines to output the list of machines within your cluster. Since we’re using CoreOS cluster size of 5 machines, we expect a list of 5 servers without any metadata (we didn’t define metadata for the servers :))
$ fleetctl list-machines MACHINE IP METADATA 15789145... 172.17.8.105 - 18e3c860... 172.17.8.104 - 4d1f67e1... 172.17.8.101 - b057ceee... 172.17.8.102 - b1c716d7... 172.17.8.103 -
Alright, everything is fine and all servers are listed and available by fleet.
In the following, we’re using the
hello.service from above, submit and start it within the cluster.
List Unit Files in Fleet Cluster
To get an overview of the unit files available within the cluster, you can run
fleetctl list-unit-files. This will print the unit’s name, the assigned hash value, the state and target machine. The target represents the machine where a service is currently scheduled.
$ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET hello.service e55c0ae launched launched 4d1f67e1.../172.17.8.101
List Active Units
If you’re only interested in the state of active units, you can print them using
fleetctl list-units. This will give you an overview of the unit and where it’s currently located.
$ fleetctl list-units UNIT MACHINE ACTIVE SUB hello.service 113f16a7.../172.17.8.103 active running
Unit files represent services. That means, the actual service running somewhere within the CoreOS cluster is created from a given unit file. This section will walk you through the service management and how to load unit files into your cluster and also, how to remove them if not required anymore.
Add & Remove Units
You need to specifically submit a unit file to your cluster to be loaded and available to fleet. Alright, we’re starting kind of low-level by creating a unit file based on the content of the exemplary
hello.service from fleet’s examples.
Log into one of your CoreOS machines (using SSH) and either git clone the fleet repository to access the examples or just create a new file called
hello.service on the machine. Once you’re done, use
fleetctl submit hello.service to submit this service to fleet.
fleetctl submit hello.service
This won’t have any additional output and you can check the status of your
$ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET hello.service e55c0ae inactive inactive -
You can remove services from your cluster using
fleetctl destroy <name.service>. This will at first stop the service on the executing host, instructing systemd to execute and tear down operations and finally remove the unit file from the cluster. This won’t let you start the service again until you re-submit it.
Load & Unload Units
Submitting units to your cluster makes them available for every machine. However, it doesn’t mean the unit gets loaded on a server or is even prepared for execution. If you just want to load a service to a server without starting it, fleet got you covered with
fleetctl load <name.service>.
$ fleetctl load hello.service Unit hello.service loaded on 15789145.../172.17.8.105 $ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET hello.service e55c0ae loaded loaded 15789145.../172.17.8.105
You see, the
hello.service was loaded to the machine
172.17.8.105. The service is still waiting for the kickoff signal to start.
If you want to unload an already loaded service from a machine, use the
fleetctl unload <name.serivce> command. This will remove the unit file from the assigned machine.
$ fleetctl unload hello.service Unit hello.service inactive $ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET hello.service e55c0ae inactive inactive -
Start & Stop Units
Submitting units to the cluster and loading them to machines won’t start them. You need to manually start them using fleet’s command line utility:
fleetctl start <name.service>. You can directly start units after submitting them to the cluster and skip the load command.
$ fleetctl start hello.service Unit hello.service launched on 15789145.../172.17.8.105 $ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET hello.service e55c0ae launched launched 15789145.../172.17.8.105
The output after starting the service will show you the machine which now runs the service.
In case you need to stop the app due to whatever reason, use
fleetctl stop <name.service>. This will immediately stop the service and return it into
$ fleetctl stop hello.service Unit hello.service loaded on 15789145.../172.17.8.105 $ fleetctl list-unit-files UNIT HASH DSTATE STATE TARGET hello.service e55c0ae loaded loaded 15789145.../172.17.8.105
Query Unit Status & Fetch Unit Logs
Submitting, loading, and starting your units on a server might run smooth and no error occur. If you’re like me, you want additionally query the current unit status and verify that the service is running actively and the log outputs are fine.
Fleet’s intuitive command line utility of course has a command to show the current status of your service:
fleetctl status <name.service>. This will print out basic information about your service, executed commands and also the latest log entries.
$ fleetctl status hello.service ● hello.service - Hello World Loaded: loaded (/run/fleet/units/hello.service; linked-runtime; vendor preset: disabled) Active: active (running) since Tue 2015-12-15 09:20:38 UTC; 3min 21s ago Main PID: 1120 (bash) Memory: 372.0K CPU: 150ms CGroup: /system.slice/hello.service ├─1120 /bin/bash -c while true; do echo "Hello, world"; sleep 1; done └─1352 sleep 1 Dec 15 09:23:49 core-05 bash: Hello, world Dec 15 09:23:50 core-05 bash: Hello, world Dec 15 09:23:51 core-05 bash: Hello, world Dec 15 09:23:52 core-05 bash: Hello, world Dec 15 09:23:53 core-05 bash: Hello, world Dec 15 09:23:54 core-05 bash: Hello, world Dec 15 09:23:55 core-05 bash: Hello, world Dec 15 09:23:56 core-05 bash: Hello, world Dec 15 09:23:57 core-05 bash: Hello, world Dec 15 09:23:58 core-05 bash: Hello, world
The service is in
active (running) state, the expected
Hello, world logs appear. Everything works like a charm :)
Another way to check the log entries is the
journalctl command. Fleet wraps the utility inside the
fleetctl and makes the commands available at
fleetctl journal. By default, the last 10 log entries are displayed when using
fleetctl journal <name.service>.
$ fleetctl journal hello.service -- Logs begin at Fri 2015-08-07 09:13:17 UTC, end at Tue 2015-12-15 09:27:54 UTC. -- Dec 15 09:27:45 core-05 bash: Hello, world Dec 15 09:27:46 core-05 bash: Hello, world Dec 15 09:27:47 core-05 bash: Hello, world Dec 15 09:27:48 core-05 bash: Hello, world Dec 15 09:27:49 core-05 bash: Hello, world Dec 15 09:27:50 core-05 bash: Hello, world Dec 15 09:27:51 core-05 bash: Hello, world Dec 15 09:27:52 core-05 bash: Hello, world Dec 15 09:27:53 core-05 bash: Hello, world Dec 15 09:27:54 core-05 bash: Hello, world
You can specify the number of lines displayed by passing the
--lines <number> parameter to the
fleetctl journal command.
$ fleetctl journal --lines 3 hello.service -- Logs begin at Fri 2015-08-07 09:13:17 UTC, end at Tue 2015-12-15 09:29:14 UTC. -- Dec 15 09:29:12 core-05 bash: Hello, world Dec 15 09:29:13 core-05 bash: Hello, world Dec 15 09:29:14 core-05 bash: Hello, world
To receive real-time logs, you can follow the journal using the
-f parameter. This will output logs directly once they occur. Our example will print a new log message every second.
$ fleetctl journal -f hello.service -- Logs begin at Fri 2015-08-07 09:13:17 UTC. -- Dec 15 09:30:38 core-05 bash: Hello, world …
What Comes Next
This post drives you through the important commands to create and manage units within a CoreOS cluster. You’ve learned the basics on systemd, the init system and fleet, which leverages systemd and etcd to a cluster-level init system. Also, you’ve got your hands dirty using fleet’s command line utility
fleetctl. It provides the necessary commands to manage services on your cluster.
Next week, we’ll finally touch a more complex example by creating and running an app in our CoreOS cluster. We’ll go through the process of creating an app that will be delivered within a docker container and ultimately run inside the cluster.