path: root/content/blog/2022-v0.7-released.md

+++
title="Garage v0.7: Kubernetes and OpenTelemetry"
date=2022-04-04
+++

*We just published Garage v0.7, our second public beta release. In this post, we do a quick tour of its 2 new features: Kubernetes integration and OpenTelemetry support.*

<!-- more -->

---

Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute.
Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0.

But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a.
This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it!

You can get this release using our binaries or the package provided by your distribution.
We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs).
Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage).
Feel free to [reach out to us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package Garage, tell us and we'll add it to the documentation.

Speaking about the changes of this new version, it obviously includes many bug fixes. 
We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), take a look, we might have fixed something that annoyed you!
Besides bugfixes, there is two new features: a better Kubernetes integration and support for OpenTelemetry.

## Kubernetes integration

Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on Kubernetes.
In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to simplify cluster discovery.
Garage can self-apply the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) to your cluster, or you can manage it manually.

Let's see practically how it works with a minimalistic example (not secured nor suitable for production). 
You can run it on [minikube](https://minikube.sigs.k8s.io) if you want a more interactive reading.

Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containing Garage's configuration (let's name it `config.yaml`):

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: garage-config
  namespace: default
data:
  garage.toml: |-
    metadata_dir = "/mnt/fast"
    data_dir = "/mnt/slow"

    replication_mode = "3"

    rpc_bind_addr = "[::]:3901"
    rpc_secret = "<secret>"

    bootstrap_peers = []

    kubernetes_namespace = "default"
    kubernetes_service_name = "garage-daemon"
    kubernetes_skip_crd = false

    [s3_api]
    s3_region = "garage"
    api_bind_addr = "[::]:3900"
    root_domain = ".s3.garage.tld"

    [s3_web]
    bind_addr = "[::]:3902"
    root_domain = ".web.garage.tld"
    index = "index.html"
```

The 3 important parameters are `kubernetes_namespace`, `kubernetes_service_name`, and `kubernetes_skip_crd`.
Configure them according to your planned deployment. 
The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on startup.
In this example, we keep it to `false`, which means we allow Garage to automatically create the CRD.

Apply this configuration on your cluster:

```bash
kubectl apply -f config.yaml
```

Allowing Garage to create the CRD is not enough, the process must have enough permissions.
A quick unsecure way to add the permission is to create a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) to give admin rights to our local user, effectively breaking Kubernetes' security model (we name this file `admin.yml`):

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: garage-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:serviceaccount:default:default
```

Apply it:

```bash
kubectl apply -f admin.yaml
```

Finally, we create a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) to run our service (`service.yaml`):

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: garage
spec:
  selector:
    matchLabels:
      app: garage
  serviceName: "garage"
  replicas: 3
  template:
    metadata:
      labels:
        app: garage
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: garage
        image: dxflrs/amd64_garage:v0.7.0
        ports:
        - containerPort: 3900
          name: s3-api
        - containerPort: 3902
          name: web-api
        volumeMounts:
        - name: fast
          mountPath: /mnt/fast
        - name: slow
          mountPath: /mnt/slow
        - name: etc
          mountPath: /etc/garage.toml
          subPath: garage.toml
      volumes:
      - name: etc
        configMap:
          name: garage-config
  volumeClaimTemplates:
  - metadata:
      name: fast
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi
  - metadata:
      name: slow
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi
```

Garage is a stateful program, so it needs a stable place to store its data and metadata.
This feature is provided by Kubernetes' [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used only from a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/), hence the choice of this K8S object to deploy our service.

Kubernetes has many "drivers" for Persistent Volumes, for production uses we recommend **only** the `local` driver.
Using other drivers may lead to huge performance issues or data corruption, probably both in practice.

In the example, we are claiming 2 volumes of 100MB. 
We use 2 volumes instead of 1 because Garage separates its metadata from its data.
By having 2 volumes, you can reserve a smaller capacity on a SSD for the metadata and a larger capacity on a regular HDD for the data.
Do not forget to change the reserved capacity, 100MB is only suitable for testing.

*Note how we are mounting our ConfigMap: we need to set the `subpath` property to mount only the `garage.toml` file and not the whole `/etc` folder that would prevent K8S from writing its own files
in `/etc` and fail the pod.*

You can apply this file with:

```bash
kubectl apply -f service.yaml
```

Now, you are ready to interact with your cluster, each instance must have discovered the other ones:

```bash
kubectl exec -it garage-0 --container garage -- /garage status
# ==== HEALTHY NODES ====
# ID                Hostname  Address                   Tags              Zone  Capacity
# e6284331c321a23c  garage-0  172.17.0.5:3901           NO ROLE ASSIGNED
# 570ff9b0ed3648a7  garage-2  [::ffff:172.17.0.7]:3901  NO ROLE ASSIGNED
# e1990a2069429428  garage-1  [::ffff:172.17.0.6]:3901  NO ROLE ASSIGNED
```

Of course, to have a full deployment, you will probably want to deploy a [Service](https://kubernetes.io/docs/concepts/services-networking/service/) in front of your cluster and/or a reverse proxy.

If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster.
We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl).

## OpenTelemetry support

[OpenTelemetry](https://opentelemetry.io/) standardizes how software generate and collect system telemetry, namely metrics, logs and traces.
By implementing this standard in Garage, we hope that it will help you to better monitor, manage and tune your cluster.
Note that to fully leverage this feature, you must be already familiar with monitoring stacks like [Prometheus](https://prometheus.io/)+[Grafana](https://grafana.com/) or [ElasticSearch](https://www.elastic.co/elasticsearch/)+[Kibana](https://www.elastic.co/kibana/).

To activate OpenTelemetry on Garage, you must add to your configuration file the following entries (supposing that your collector is also on localhost):

```toml
[admin]
api_bind_addr = "127.0.0.1:3903"
trace_sink = "http://localhost:4317"
```

We provide [some files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/telemetry) to help you quickly bootstrap a testing monitoring stack.
It includes a docker-compose file and a pre-configured Grafana dashboard.
You can use them if you want to reproduce the following examples.

Now that your telemetry data is collected and stored, you can visualize it.

Grafana is particularly adapted to understand how your cluster is performing from a "bird's eye view".
For example, the following graph shows S3 API calls sent to your node per time-unit,
you can use it to better understand how your users are interacting with your cluster.

![A screenshot of a plot made by Grafana depicting the number of requests per time units grouped by endpoints](/images/blog/api_rate.png)

Thanks to this graph, we know that starting at 14:55, an important upload has been started.
This upload is made of many small files, as we see many PutObject calls that are often used for small files.
It also has some large objects, as we observe some Multipart Uploads requests.
Conversely, at this time, no read are done as the corresponding read enpoints (ListBuckets, ListObjectsv2, etc.) receive 0 request per time unit.


Garage also collects metrics from lower level parts of the system.
You can use them to better understand how Garage is interacting with your OS and your hardware.

![A screenshot of a plot made by Grafana depicting the write speed (in MB/s) during time.](/images/blog/writes.png)

This plot has been captured at the same moment than the previous one.
We do not see a correlation between the writes and the API requests for the full upload but only for its beginning.
However, it maps well to Multipart Uploads requests: this is expected because small files will be throttled by other parts of the system, while large files will be able to saturate the writes of your disk.

This simple example, done on a test cluster, covers only 2 metrics over the 20+ ones that we already defined but we were still able to precisely describe our cluster usage and identifies where bottlenecks could be.
We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster.

While metrics are good to have a large, general overview of your system, they are however not adapted to dig and pinpoint a specific performance problem on a specific code path.
Thankfully, we also have a solution for this problem: traces.

Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana,
we get the following visualization:

![A screenshot of APM depicting the trace of a PutObject call](/images/blog/apm.png)

On the top of the screenshot, we see the latency distribution of all PutObject requests.
We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run.
Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that
a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one).

Below, you can select the request you want to inspect, and then see its stacktrace.
You can break down these lines in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`).

With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, to finally identify which part of the code took time.


## And next?

roadmap: k2v, allocation simulator, s3 compatibility, community feedback, whitepaper