This is the second part about etcd and completely geared towards etcd’s HTTP/JSON API and the command line utility etcdctl. Within the previous post on etcd, we described how etcd can bootstrap a new cluster with the help of an existing one. Further, we showed you two configuration opportunities: command line or #cloud-config.

Within this article, we deep-dive on etcd communication. That means, we’ll have a look at etcd’s API and available functionality to get, set, watch keys and their values. Additionally, we clarify the confusion about used ports by etcd.

CoreOS Series Overview

Introduction

etcd is installed on every CoreOS node within your cluster and exposes a RESTful HTTP/JSON API to read and write data. Application containers can use etcd to share connection strings for service discovery. There are multiple ways to read and write data into etcd: directly via HTTP API with a tool like curl, language specific libraries or etcd’s command line utility etcdctl.

Ports

Before version 2.0, etcd exposed two different ports than today for client and server communication. Originally, etcd exposed port 4001 for client and port 7001 for server-to-server communication. With the launch of etcd 2.0, IANA assigned etcd two different ports than previously: 2379 for client and 2380 for server-to-server communication.

etcd has backwards compatibility and you can still use the old ports. If you’re new to CoreOS and etcd or just starting out, you can restrict yourself to only use the IANA assigned ports 2379 and 2380.

Within this series, we define etcd’s configuration to listen on the new as well as the old ports.

Communicate with etcd via etcdctl

etcd offers multiple configuration options and has an extensive functionality set. The data within the distributed key-value-store is stored in files within directories. This type of hierarchy is similar to a file system.

As already mentioned, you can use the command line utility etcdctl to get, set, or watch keys and their values. We’ll cover the HTTP/JSON API later within this post. For now, we walk through the basic commands to get, set and watch keys with the help of etcdctl.

Write to etcd: Set Values and Create Directories

Before we read values from etcd, we start with writing the typical foo-bar key-value-pair. This way, we can write our typical test values first and read them later to verify the operation succeeded successfully.

etcd handles the key-value-pairs like a file system and you can write keys as you would create files within a directory. Write your key-value-pairs with a leading slash. Let’s set the /foo key to illustrate the example.

etcdctl set /foo bar  

Using a hierarchy will result in a key like /f1/f2/foo/bar.

etcdctl set /bar/foo "This is CoreOS"  

You can’t override an existing key with a directory having the same name as the key. For example: creating a key /foo and afterwards trying to create a key /foo/bar will result in an error since the key foo already exists. You need to delete the key first to write the directory-style key-value-pair.

etcd allows to update values for existing keys.

etcdctl update /bar/foo "This is updated CoreOS"  

There is a command to create a new directory: mkdir.

etcdctl mkdir /fooDir  

Directories can also be updated and have a limited lifetime.

etcdctl updatedir /fooDir --ttl 10  

Create new keys within a directory using either set or mk. The mk command only creates the new key if it doesn’t exist. The set command will override existing keys.

etcdctl set /fooDir/foo bar  
etcdctl mk /fooDir/bar foo  

Expiring Keys with TTL

etcd supports expiring keys. Define the time to live flag in seconds and etcd will automatically delete your key-value-pair once the time is up.

etcdctl set /foo bar --ttl 5  

Read from etcd: Get Keys and Directories

You can get the value for a given key using etcdctl:

etcdctl get /foo  

The command above just prints the current value if existing. You can extend the output of etcdctl using the -o extended flag. This will print out additional, parseble information.

etcdctl -o extended get /foo  

Key: /foo  
Created-Index: 14963  
Modified-Index: 14963  
TTL: 0  
Etcd-Index: 14985  
Raft-Index: 47174  
Raft-Term: 67

bar  

Directories can contain other further directories and keys. List the root directory with

etcdctl ls  

/foo
/coreos.com
/bar
/f1

There can be directories and keys within a given directory. Distinct both types using the -p flag.

etcdctl ls -p  

/bar/
/f1/
/foo
/coreos.com/

Distinguishable by the trailing slash, everything without one is a key. Here, /foo is just a key and everything else a directory.

etcdctl saves you time and can list the contents of every directory recursively.

etcdctl ls -recursive  

/bar
/bar/foo
/f1
/f1/f2
/f1/f2/foo
/f1/f2/foo/bar
/foo
/coreos.com
/coreos.com/updateengine
/coreos.com/updateengine/rebootlock
/coreos.com/updateengine/rebootlock/semaphore

Delete Keys and Directories from etcd

etcdctl commands are oriented towards typical unix commands line instructions. You’ll find this behavior also when deleting keys and directories.

Deleting a key

etcdctl rm /foo  

You’re completely right, if you’re intuitively thinking of the rmdir command to delete a directory.

Delete an empty directory:

etcdctl rmdir /foo/bar/dirname  

You can also use the --dir flag for the rm command to delete an empty repository:

etcdctl rm /foo/bar/dirname --dir  

Completely removing the hierarchy including all sub-directories and keys included:

etcdctl rm /foo/bar/dirname --recursive  

etcd provides the functionality to conditionally delete key-value-pairs matching a given condition respective the value or raft index.

etcdctl rm /bar/foo --with-value "This is CoreOS"  

etcdctl rm /foo --with-index 42  

Watch Key-Value-Changes

etcd distributes keys and their values throughout the cluster. Every machine within the cluster shares the data with its peers. You’re able to watch specific keys for value changes and react to them properly.

Watch for a key change:

etcdctl watch /foo  

The process will stop watching the key /foo once it changes its value. Further changes get distributed unnoticed. To listen forever, just add the --forever flag.

etcdctl watch /foo --forever  

Continuously watching a key will grab the command line and you need to get out with CTRL/CMD+c.

Hidden Keys and Values

One feature of etcd that’s living in secret is hidden keys. These hidden keys get distributed like normal ones, but they don’t appear when listing keys and directories.

Hidden keys begin with an underscore.

You can use etcdctl to show them when directly specifying the name

etcdctl ls --recursive /_coreos.com  

You can check the distribution across the cluster by setting a hidden key _foo:

etcdctl set _foo bar  

Using another cluster member and getting the value of _foo will print bar, even though the key won’t get listed using ls.

etcdctl set _foo bar  
bar  

Communicate with etcd via HTTP/JSON API

etcd provides an easy to use HTTP/JSON API to get, set and watch keys and their values. Internally, etcd transparently redirects all reads to the leader. That means, you need to add the -L flag to every curl command to follow redirects and location headers from etcd. Response will be in JSON format.

Generally, every command available in etcdctl is also working when using the HTTP/JSON API. Use the HTTP verbs GET, PUT, DELETE to get, watch, set, and delete keys. You can also add flags by passing them as query parameters within the url with your request. Let’s just jump right into some examples.

Set a key and value:

curl -L -X PUT http://127.0.0.1:2379/v2/keys/foo -d value="bar"  

Get the key and value:

curl -L http://127.0.0.1:2379/v2/keys/foo  

You can list all existing keys within etcd:

curl -L http://127.0.0.1:2379/v2/keys/  

{
  "action":"get",
  "node":{
    "dir":true,
    "nodes":[
      {
        "key":"/coreos.com",
        "dir":true,
        "modifiedIndex":6,
        "createdIndex":6
      },
      {
        "key":"/discovery",
        "value":"",
        "modifiedIndex":24836,
        "createdIndex":24836
      },
      {
        "key":"/foo",
        "value":"bar",
        "modifiedIndex":51366,
        "createdIndex":51366
      }
    ]
  }
}

Remember the flags from etcdctl like --recursive when listing all existing keys? You can do that as well with the HTTP/JSON API by passing the flags as query parameters within the url.

curl -L http://127.0.0.1:2379/v2/keys/?recursive=true  

{
  "action":"get",
  "node":{
    "dir":true,
    "nodes":[
      {
        "key":"/message",
        "value":"Hello",
        "modifiedIndex":59089,
        "createdIndex":59089
      },
      {
        "key":"/coreos.com",
        "dir":true,
        "nodes":[
          {
            "key":"/coreos.com/updateengine",
            "dir":true,
            "nodes":[
              {
                "key":"/coreos.com/updateengine/rebootlock",
                "dir":true,
                "nodes":[
                  {
                    "key":"/coreos.com/updateengine/rebootlock/semaphore",
                    "value":"{\"semaphore\":1,\"max\":1,\"holders\":[]}",
                    "modifiedIndex":54495,
                    "createdIndex":6
                  }
                ],
                "modifiedIndex":6,
                "createdIndex":6
              }
            ],
            "modifiedIndex":6,
            "createdIndex":6
          }
        ],
        "modifiedIndex":6,
        "createdIndex":6
      },
      {
        "key":"/discovery",
        "value":"",
        "modifiedIndex":24836,
        "createdIndex":24836
      },
      {
        "key":"/foo",
        "value":"bar",
        "modifiedIndex":51366,
        "createdIndex":51366
      }
    ]
  }
}

Watch a key for changes: wait=true

curl -L http://127.0.0.1:2379/v2/keys/foo?wait=true  

Watching for values changes on keys is of course available via API. The thing is, that long polling operations may fail due to request timeouts or server restarts. Clients have to deal with this situations and retry the watch operations in such cases.

Delete a key:

curl http://127.0.0.1:2379/v2/keys/foo -XDELETE  

Actually, there is much more functionality to cover for the HTTP/JSON API. We already guided you through most of the commands when touching etcdctl. If you want to soak up more about the HTTP/JSON API of etcd, head over to their API documents on GitHub.

Outlook

This post showed you the commands to get, set, watch and delete keys and directories. CoreOS ships with the command line utility etcdctl to interact with etcd. Additionally, you can use the HTTP/JSON API for client communication.

Within the next post, we’ll deep-dive into CoreOS’s distributed init system fleet.

---

Additional Resources

• etcd Configuration & Cluster Discovery
• etcd repository on GitHub
• etcd documentation on coreos.com
• etcdctl overview on GitHub