From c0f5b325abe5287bf05d2e9dcec83ce86352b2fd Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Tue, 5 Apr 2022 00:56:59 +0200 Subject: Intro --- content/blog/2022-v0.7-released.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 content/blog/2022-v0.7-released.md (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md new file mode 100644 index 0000000..8c34703 --- /dev/null +++ b/content/blog/2022-v0.7-released.md @@ -0,0 +1,27 @@ ++++ +title="Garage v0.7: a tour of the new features" +date=2022-04-04 ++++ + +*We just published Garage v0.7, our second public beta release. In this post, we do a quick tour of its 2 new features: Kubernetes integration and OpenTelemetry support.* + + + +--- + +Two months ago, we were impressed by the success of our open beta launch at the FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute. +Since this event, we continued improving Garage, and 2 months after the initial release, we are happy to announce a new version: v0.7.0. +We would like to thank all the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. +This is also for the first time for Garage that we have contributions from outside of our organization: we are very proud and we want to renew our commitment to foster an open community around Garage. + +If you want to test this new version, you have 2 solutions: using our binaries or the ones from your OS. +We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for Linux (amd64, i386, aarch64 and armv6) and their associated [Docker containers](https://hub.docker.com/u/dxflrs). +Garage is also packaged by some OS/distributions, we are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). +Feel free to [reach us](mailto:garagehq@deuxfleurs.fr) if you are packaging or planning to package Garage, we are willing to adapt our software to make packaging easier and we plan to reference your work in our documentation. +Obviously, this new version includes many bug fixes that are listed in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), but also 2 new features: Kubernetes integration and OpenTelemetry support, we review them in the following. + +## Kubernetes integration + +## OpenTelemetry support + +T -- cgit v1.2.3 From 187e4dcbefeb1718433bdac15e19a48f94730680 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Tue, 5 Apr 2022 16:11:22 +0200 Subject: Add Kubernetes part --- content/blog/2022-v0.7-released.md | 185 ++++++++++++++++++++++++++++++++++++- 1 file changed, 180 insertions(+), 5 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index 8c34703..be2939a 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -1,5 +1,5 @@ +++ -title="Garage v0.7: a tour of the new features" +title="Garage v0.7: Kubernetes and OpenTelemetry" date=2022-04-04 +++ @@ -11,17 +11,192 @@ date=2022-04-04 Two months ago, we were impressed by the success of our open beta launch at the FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute. Since this event, we continued improving Garage, and 2 months after the initial release, we are happy to announce a new version: v0.7.0. -We would like to thank all the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. -This is also for the first time for Garage that we have contributions from outside of our organization: we are very proud and we want to renew our commitment to foster an open community around Garage. +Before all, we would like to thank all the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. +This is also the first time for Garage that we have contributions from the outside: we are very happy as we want to build a community-driven project. If you want to test this new version, you have 2 solutions: using our binaries or the ones from your OS. We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for Linux (amd64, i386, aarch64 and armv6) and their associated [Docker containers](https://hub.docker.com/u/dxflrs). Garage is also packaged by some OS/distributions, we are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). Feel free to [reach us](mailto:garagehq@deuxfleurs.fr) if you are packaging or planning to package Garage, we are willing to adapt our software to make packaging easier and we plan to reference your work in our documentation. -Obviously, this new version includes many bug fixes that are listed in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), but also 2 new features: Kubernetes integration and OpenTelemetry support, we review them in the following. + +Speaking about the changes of this new version, it obviously includes many bug fixes. +We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), take a look, we might have fixed something that annoyed you! +In this blog post, we want to introduce you to another aspect of this new release, its 2 new features: a better Kubernetes integration and support for OpenTelemetry. ## Kubernetes integration +Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a coordinating pod to deploy Garage on Kubernetes. +In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to ease your deployments. +Garage is even able to automatically create the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) before using it to discover other peers. + +Let's see practically how it works with a minimalistic example (not secured nor suitable for production). +You can run it on [minikube](https://minikube.sigs.k8s.io) if you a more interactive reading. + +Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containg Garage's configuration (let's name it `config.yaml`): + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: garage-config + namespace: default +data: + garage.toml: |- + metadata_dir = "/mnt/fast" + data_dir = "/mnt/slow" + + replication_mode = "3" + + rpc_bind_addr = "[::]:3901" + rpc_secret = "" + + bootstrap_peers = [] + + kubernetes_namespace = "default" + kubernetes_service_name = "garage-daemon" + kubernetes_skip_crd = false + + [s3_api] + s3_region = "garage" + api_bind_addr = "[::]:3900" + root_domain = ".s3.garage.tld" + + [s3_web] + bind_addr = "[::]:3902" + root_domain = ".web.garage.tld" + index = "index.html" +``` + +The 3 important parameters are `kubernetes_namespace`, `kubernetes_service_name`, and `kubernetes_skip_crd`. +Configure them according to your planned deployment. +The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on boot. +In this example, we keep it to `false`, which means we allow Garage to automatically create the CRD. + +Apply this configuration on your cluster: + +```bash +kubectl apply -f config.yaml +``` + +Allowing Garage to create the CRD is not enough, the process must have enough permissions. +A quick unsecure way to add the permission is to create a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) to give admin rights to our local user, effectively breaking Kubernetes' security model (we name this file `admin.yml`): + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: garage-admin +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: User + name: system:serviceaccount:default:default +``` + +Apply it: + +```bash +kubectl apply -f admin.yaml +``` + +Finally, we create a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) to run our service (`service.yaml`): + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: garage +spec: + selector: + matchLabels: + app: garage + serviceName: "garage" + replicas: 3 + template: + metadata: + labels: + app: garage + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: garage + image: dxflrs/amd64_garage:v0.7.0 + ports: + - containerPort: 3900 + name: s3-api + - containerPort: 3902 + name: web-api + volumeMounts: + - name: fast + mountPath: /mnt/fast + - name: slow + mountPath: /mnt/slow + - name: etc + mountPath: /etc/garage.toml + subPath: garage.toml + volumes: + - name: etc + configMap: + name: garage-config + volumeClaimTemplates: + - metadata: + name: fast + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 100Mi + - metadata: + name: slow + spec: + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: 100Mi +``` + +Garage is a stateful program, so it needs a stable place to store its data and metadata. +This feature is provided by Kubernetes' [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used only from a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/), hence the choice of this K8S object to deploy our service. + +Kubernetes has many "drivers" for Persistent Volumes, for production uses we recommend **only** the `local` driver. +Using other drivers may lead to huge performance issues or data corruption, probably both in practice. + +In the example, we are claiming 2 volumes of 100MB. +We use 2 volumes instead of 1 because Garage separates its metadata from its data. +By having 2 volumes, you can reserve a smaller capacity on a SSD for the metadata and a larger capacity on a regular HDD for the data. +Do not forget to change the reserved capacity, 100MB is only suitable for testing. + +*Note how we are mounting our ConfigMap: we need to set the `subpath` property to mount only the `garage.toml` file and not the whole `/etc` folder that would prevent K8S from writing its own files +in `/etc` and fail the pod.* + +You can apply this file with: + +```bash +kubectl apply -f service.yaml +``` + +Now, you are ready to interact with your cluster, each instance must have discovered the other ones: + +```bash +kubectl exec -it garage-0 --container garage -- /garage status +# ==== HEALTHY NODES ==== +# ID Hostname Address Tags Zone Capacity +# e6284331c321a23c garage-0 172.17.0.5:3901 NO ROLE ASSIGNED +# 570ff9b0ed3648a7 garage-2 [::ffff:172.17.0.7]:3901 NO ROLE ASSIGNED +# e1990a2069429428 garage-1 [::ffff:172.17.0.6]:3901 NO ROLE ASSIGNED +``` + +Of course, to have a full deployment, you will probably want to deploy a [Service](https://kubernetes.io/docs/concepts/services-networking/service/) in front of your cluster and/or a reverse proxy. + +If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster. +We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl). + ## OpenTelemetry support -T +## And next? + +roadmap: k2v, allocation simulator, s3 compatibility, community feedback, whitepaper + -- cgit v1.2.3 From 6df348b2a3e5d2377a8167aa15ff79a7fbec0206 Mon Sep 17 00:00:00 2001 From: Maximilien Date: Wed, 6 Apr 2022 10:28:13 +0200 Subject: Reword top half --- content/blog/2022-v0.7-released.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index be2939a..e1795d9 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -9,25 +9,26 @@ date=2022-04-04 --- -Two months ago, we were impressed by the success of our open beta launch at the FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute. -Since this event, we continued improving Garage, and 2 months after the initial release, we are happy to announce a new version: v0.7.0. -Before all, we would like to thank all the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. -This is also the first time for Garage that we have contributions from the outside: we are very happy as we want to build a community-driven project. +Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute. +Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0. -If you want to test this new version, you have 2 solutions: using our binaries or the ones from your OS. -We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for Linux (amd64, i386, aarch64 and armv6) and their associated [Docker containers](https://hub.docker.com/u/dxflrs). -Garage is also packaged by some OS/distributions, we are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). -Feel free to [reach us](mailto:garagehq@deuxfleurs.fr) if you are packaging or planning to package Garage, we are willing to adapt our software to make packaging easier and we plan to reference your work in our documentation. +But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. +This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it. + +As a noverlty as well, you can get this release using our binaries or the package provided by your distribution. +We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs). +Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). +Feel free to [reach us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package garage, tell us and we'll add it to the documentation. Speaking about the changes of this new version, it obviously includes many bug fixes. We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), take a look, we might have fixed something that annoyed you! -In this blog post, we want to introduce you to another aspect of this new release, its 2 new features: a better Kubernetes integration and support for OpenTelemetry. +Besides bugfixes, there is two new features: a better Kubernetes integration and support for OpenTelemetry. ## Kubernetes integration -Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a coordinating pod to deploy Garage on Kubernetes. -In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to ease your deployments. -Garage is even able to automatically create the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) before using it to discover other peers. +Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on Kubernetes. +In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to simplify cluster discovery. +Garage can self-apply the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) to your cluster, or you can manage it manually. Let's see practically how it works with a minimalistic example (not secured nor suitable for production). You can run it on [minikube](https://minikube.sigs.k8s.io) if you a more interactive reading. @@ -69,7 +70,7 @@ data: The 3 important parameters are `kubernetes_namespace`, `kubernetes_service_name`, and `kubernetes_skip_crd`. Configure them according to your planned deployment. -The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on boot. +The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on startup. In this example, we keep it to `false`, which means we allow Garage to automatically create the CRD. Apply this configuration on your cluster: -- cgit v1.2.3 From 3eb41b73e4297161de89fecdcc43d6c33c1836ee Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Wed, 6 Apr 2022 16:23:13 +0200 Subject: OpenTelemetry part --- content/blog/2022-v0.7-released.md | 69 +++++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 4 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index e1795d9..5fe3010 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -13,9 +13,9 @@ Two months ago, we were impressed by the success of our open beta launch at FOSD Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0. But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. -This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it. +This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it! -As a noverlty as well, you can get this release using our binaries or the package provided by your distribution. +You can get this release using our binaries or the package provided by your distribution. We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs). Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). Feel free to [reach us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package garage, tell us and we'll add it to the documentation. @@ -31,9 +31,9 @@ In this new version, Garage integrates a method to discover other peers by using Garage can self-apply the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) to your cluster, or you can manage it manually. Let's see practically how it works with a minimalistic example (not secured nor suitable for production). -You can run it on [minikube](https://minikube.sigs.k8s.io) if you a more interactive reading. +You can run it on [minikube](https://minikube.sigs.k8s.io) if you want a more interactive reading. -Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containg Garage's configuration (let's name it `config.yaml`): +Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containing Garage's configuration (let's name it `config.yaml`): ```yaml apiVersion: v1 @@ -197,6 +197,67 @@ We have not documented it yet but you can get a look at [our Nomad service](http ## OpenTelemetry support +[OpenTelemetry](https://opentelemetry.io/) standardizes how software generate and collect system telemetry, namely metrics, logs and traces. +By implementing this standard in Garage, we hope that it will help you to better monitor, manage and tune your cluster. +Note that to fully leverage this feature, you must be already familiar with monitoring stacks like [Prometheus](https://prometheus.io/)+[Grafana](https://grafana.com/) or [ElasticSearch](https://www.elastic.co/elasticsearch/)+[Kibana](https://www.elastic.co/kibana/). + +To activate OpenTelemetry on Garage, you must add to your configuration file the following entries (supposing that your collector is also on localhost): + +```toml +[admin] +api_bind_addr = "127.0.0.1:3903" +trace_sink = "http://localhost:4317" +``` + +We provide [some files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/telemetry) to help you quickly bootstrap a testing monitoring stack. +It includes a docker-compose file and a pre-configured Grafana dashboard. +You can use them if you want to reproduce the following examples. + +Now that your telemetry data is collected and stored, you can visualize it. + +Grafana is particularly adapted to understand how your cluster is performing from a "bird's eye view". +For example, the following graph shows S3 API calls sent to your node per time-unit, +you can use it to better understand how your users are interacting with your cluster. + +![A screenshot of a plot made by Grafana depicting the number of requests per time units grouped by endpoints](/images/blog/api_rate.png) + +Thanks to this graph, we know that starting at 14:55, an important upload has been started. +This upload is made of many small files, as we see many PutObject calls that are often used for small files. +It also has some large objects, as we observe some Multipart Uploads requests. +Conversely, at this time, no read are done as the corresponding read enpoints (ListBuckets, ListObjectsv2, etc.) receive 0 request per time unit. + + +Garage also collects metrics from lower level parts of the system. +You can use them to better understand how Garage is interacting with your OS and your hardware. + +![A screenshot of a plot made by Grafana depicting the write speed (in MB/s) during time.](/images/blog/writes.png) + +This plot has been captured at the same moment than the previous one. +We do not see a correlation between the writes and the API requests for the full upload but only for its beginning. +However, it maps well to Multipart Uploads requests: this is expected because small files will be throttled by other parts of the system, while large files will be able to saturate the writes of your disk. + +This simple example, done on a test cluster, covers only 2 metrics over the 20+ ones that we already defined but we were still able to precisely describe our cluster usage and identifies where bottlenecks could be. +We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster. + +While metrics are good to have a large, general overview of your system, they are however not adapted to dig and pinpoint a specific performance problem on a specific code path. +Thankfully, we also have a solution for this problem: traces. + +Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana, +we get the following visualization: + +![A screenshot of APM depicting the trace of a PutObject call](/images/blog/apm.png) + +On the top of the screenshot, we see the latency distribution of all PutObject requests. +We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run. +Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that +a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one). + +Below, you can select the request you want to inspect, and then see its stacktrace. +You can break down these lines in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`). + +With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, to finally identify which part of the code took time. + + ## And next? roadmap: k2v, allocation simulator, s3 compatibility, community feedback, whitepaper -- cgit v1.2.3 From 0f7861ec98d711dd41afa8de1ed9e3a3619711cf Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Wed, 6 Apr 2022 17:48:32 +0200 Subject: Fix LX remarks --- content/blog/2022-v0.7-released.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index 5fe3010..071d2ee 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -18,7 +18,7 @@ This is also our first time welcoming contributors external to the core team, an You can get this release using our binaries or the package provided by your distribution. We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs). Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). -Feel free to [reach us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package garage, tell us and we'll add it to the documentation. +Feel free to [reach out to us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package Garage, tell us and we'll add it to the documentation. Speaking about the changes of this new version, it obviously includes many bug fixes. We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), take a look, we might have fixed something that annoyed you! -- cgit v1.2.3 From b96dfce719580e3481af02c95f89c5d79ad5b3e7 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Wed, 6 Apr 2022 19:11:11 +0200 Subject: WIP roadmap --- content/blog/2022-v0.7-released.md | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index 071d2ee..c332a80 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -234,13 +234,14 @@ You can use them to better understand how Garage is interacting with your OS and This plot has been captured at the same moment than the previous one. We do not see a correlation between the writes and the API requests for the full upload but only for its beginning. -However, it maps well to Multipart Uploads requests: this is expected because small files will be throttled by other parts of the system, while large files will be able to saturate the writes of your disk. +More precisely, it maps well to Multipart Uploads requests, and this is expected. +Large files (of the Multipart Uploads) will saturate the writes of your disk but the uploading of small files (via the PutObject endpoint) will be throttled by other parts of the system. -This simple example, done on a test cluster, covers only 2 metrics over the 20+ ones that we already defined but we were still able to precisely describe our cluster usage and identifies where bottlenecks could be. +This simple example covers only 2 metrics over the 20+ ones that we already defined, but we were still able to precisely describe our cluster usage and identifies where bottlenecks could be. We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster. -While metrics are good to have a large, general overview of your system, they are however not adapted to dig and pinpoint a specific performance problem on a specific code path. -Thankfully, we also have a solution for this problem: traces. +While metrics are good to have a large, general overview of your system, they are however not adapted to dig and pinpoint a specific performance issue on a specific code path. +Thankfully, we also have a solution for this problem: tracing. Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana, we get the following visualization: @@ -251,14 +252,29 @@ On the top of the screenshot, we see the latency distribution of all PutObject r We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run. Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one). +Consequently, this request corresponds probably to a very tiny file. -Below, you can select the request you want to inspect, and then see its stacktrace. -You can break down these lines in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`). +Below this first histogram, you can select the request you want to inspect, and then see its stacktrace on the bottom part. +You can break down this trace in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`). With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, to finally identify which part of the code took time. +Keep in mind that this is our first iteration on telemetry for Garage, so things are a bit rough around the edges (step by step documentation is missing, our Grafana dashboard is a work in a progress, etc.). +In all cases, your feedback is welcome on our Matrix channel. ## And next? -roadmap: k2v, allocation simulator, s3 compatibility, community feedback, whitepaper +While we hope that Garage in its current state inspired you, we also understand that you may be curious about what will come next! +Currently, our goal is to reach v1.0, for which we want to work on these three desirable properties: *Feature complete*, *Understandability and manageability*, and *Correctness*. +**Feature complete**. We have already implemented a selected subset of S3 endpoints that works quite well, but we want to work on the corner cases that are not yet solved (eg. [#263](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues), [#248](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/248), [#204](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/204). Based on community feedbacks, we might consider implementing additional endpoints (eg. [#166](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/166) but we can make no promise (sorry!). Finally, we made a serie of observation: 1) the S3 API has a limited semantic, for example it is not adapted for append-only log data structures, 2) many projects require a database additionaly to the object store 3) we already implemented a key value store internally to handle S3 metadata. It leads us to the conclusion that we study the feasibility of providing a simple and totally optional key value interface that we refer as K2V. We are currently writing [an API draft](https://p.adnab.me/code/#/2/code/view/eUNPbfoUrMbCY+CoMXaqed4jmWlmvWALHNDcfuM-O5o/embed/present/) and will try to implement it in the following months. We would like it to be as close as possible as the original [Amazon Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf), or if you are more familiar with Cassandra, as the most possible minimalistic Cassandra. + +**Understandability and manageability**. We want a system that is understood and manageable by the largest possible amount of operators. To achieve such a goal, we can follow 2 paths: sharing knowledge and making better tools. We want to explore both approaches, and we identified specific subjects on which to work: 1) Garage's consistency model of the S3 API and the admin API, 2) Explaining how Garage can take its place in the existing ecosystem, including among the other distributed storage systems, but also in term of uses cases and business + + - Well understood, well explained + - Consistency Model + - Deployment Cas Typique + - web interface, rest admin + - Storage density vs reliability, deployment simulator + - Fast, tested and correct + - -- cgit v1.2.3 From 1f45e9d987380034570bc86310fd1dee42c8dcf1 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Wed, 6 Apr 2022 22:39:57 +0200 Subject: Finish the roadmap --- content/blog/2022-v0.7-released.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index c332a80..eaa06da 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -265,16 +265,19 @@ In all cases, your feedback is welcome on our Matrix channel. ## And next? While we hope that Garage in its current state inspired you, we also understand that you may be curious about what will come next! -Currently, our goal is to reach v1.0, for which we want to work on these three desirable properties: *Feature complete*, *Understandability and manageability*, and *Correctness*. +Currently, our goal is to reach v1.0, for which we want to work on these three desirable properties: *Feature completeness*, *Understandability and manageability*, and *Correctness*. -**Feature complete**. We have already implemented a selected subset of S3 endpoints that works quite well, but we want to work on the corner cases that are not yet solved (eg. [#263](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues), [#248](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/248), [#204](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/204). Based on community feedbacks, we might consider implementing additional endpoints (eg. [#166](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/166) but we can make no promise (sorry!). Finally, we made a serie of observation: 1) the S3 API has a limited semantic, for example it is not adapted for append-only log data structures, 2) many projects require a database additionaly to the object store 3) we already implemented a key value store internally to handle S3 metadata. It leads us to the conclusion that we study the feasibility of providing a simple and totally optional key value interface that we refer as K2V. We are currently writing [an API draft](https://p.adnab.me/code/#/2/code/view/eUNPbfoUrMbCY+CoMXaqed4jmWlmvWALHNDcfuM-O5o/embed/present/) and will try to implement it in the following months. We would like it to be as close as possible as the original [Amazon Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf), or if you are more familiar with Cassandra, as the most possible minimalistic Cassandra. +**Feature completeness**. We have already implemented a selected subset of S3 endpoints that works quite well, but we want to work on the corner cases that are not yet solved (eg. [#263](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues), [#248](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/248), [#204](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/204). Based on community feedbacks, we might consider implementing additional endpoints (eg. [#166](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/166) but we can't make any promise (sorry!). Finally, we made a serie of observation: 1) the S3 API has a limited semantic, for example it is not adapted for append-only log data structures, 2) many projects require a database additionaly to the object store, and 3) we already implemented a key value store internally to handle S3 metadata. Following these observations, we want to study the feasibility of providing a simple and totally optional key value interface that we refer to as K2V. We are currently writing [an API draft](https://p.adnab.me/code/#/2/code/view/eUNPbfoUrMbCY+CoMXaqed4jmWlmvWALHNDcfuM-O5o/embed/present/) and will try to implement it in the following months. We would like it to be as close as possible as the original [Amazon Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf), or if you like approximative comparisons, K2V could be to Cassandra what sqlite is to PostgreSQL. -**Understandability and manageability**. We want a system that is understood and manageable by the largest possible amount of operators. To achieve such a goal, we can follow 2 paths: sharing knowledge and making better tools. We want to explore both approaches, and we identified specific subjects on which to work: 1) Garage's consistency model of the S3 API and the admin API, 2) Explaining how Garage can take its place in the existing ecosystem, including among the other distributed storage systems, but also in term of uses cases and business +**Understandability and manageability**. We want a system that is understood and manageable by the largest possible amount of operators. We identified the following points that we would like to improve: 1) Explaining Garage's consistency model, both on the S3 and the admin API, 2) Explaining how Garage can take its place in the existing ecosystem, including among the other distributed storage systems (eg. Ceph, Minio, SeaweedFS, IPFS Cluster), but also in term of uses cases and deployments (how does it perform at scale, with which hardware, for which application, etc.) 3) Make possible to manage Garage from a REST API, possibly write a web GUI to make administration easier, 4) help people understand the reliability and storage density they will have for a specific Garage deployment, if possible through a simulator, 5) we might consider adding a system of quota to protect a cluster from a misbehaving user. + +**Correctness**. We know in theory that Garage's design works and scales. +But we still need to make sure that in practise our implementation is correct, and thus features these defined properties. +To convince ourselves, we consider verifying our consistency model implementation through [Jepsen](https://jepsen.io/). +We also plan to deploy Garage on multiple clusters and do a large serie of benchmarks. + +Please note that this roadmap is purely indicative, we are not committing to deliver these features. +We also don't know when v1.0 will be released, except "when it will be ready", but we would be happy if it could be by the end of 2022. +Finally, if you have some knowledge on one or more of thes points and would like to help, feel free to ping us on Matrix. +And see you soon for the next release! - - Well understood, well explained - - Consistency Model - - Deployment Cas Typique - - web interface, rest admin - - Storage density vs reliability, deployment simulator - - Fast, tested and correct - - -- cgit v1.2.3 From b93391bfa6666b224f46946c0455882d3c9324fb Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Thu, 7 Apr 2022 11:38:12 +0200 Subject: Remove the roadmap from this post --- content/blog/2022-v0.7-released.md | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index eaa06da..2b90373 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -262,22 +262,8 @@ With this example, we demonstrated that we can inspect Garage internals to find Keep in mind that this is our first iteration on telemetry for Garage, so things are a bit rough around the edges (step by step documentation is missing, our Grafana dashboard is a work in a progress, etc.). In all cases, your feedback is welcome on our Matrix channel. -## And next? -While we hope that Garage in its current state inspired you, we also understand that you may be curious about what will come next! -Currently, our goal is to reach v1.0, for which we want to work on these three desirable properties: *Feature completeness*, *Understandability and manageability*, and *Correctness*. +## Conclusion -**Feature completeness**. We have already implemented a selected subset of S3 endpoints that works quite well, but we want to work on the corner cases that are not yet solved (eg. [#263](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues), [#248](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/248), [#204](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/204). Based on community feedbacks, we might consider implementing additional endpoints (eg. [#166](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/166) but we can't make any promise (sorry!). Finally, we made a serie of observation: 1) the S3 API has a limited semantic, for example it is not adapted for append-only log data structures, 2) many projects require a database additionaly to the object store, and 3) we already implemented a key value store internally to handle S3 metadata. Following these observations, we want to study the feasibility of providing a simple and totally optional key value interface that we refer to as K2V. We are currently writing [an API draft](https://p.adnab.me/code/#/2/code/view/eUNPbfoUrMbCY+CoMXaqed4jmWlmvWALHNDcfuM-O5o/embed/present/) and will try to implement it in the following months. We would like it to be as close as possible as the original [Amazon Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf), or if you like approximative comparisons, K2V could be to Cassandra what sqlite is to PostgreSQL. -**Understandability and manageability**. We want a system that is understood and manageable by the largest possible amount of operators. We identified the following points that we would like to improve: 1) Explaining Garage's consistency model, both on the S3 and the admin API, 2) Explaining how Garage can take its place in the existing ecosystem, including among the other distributed storage systems (eg. Ceph, Minio, SeaweedFS, IPFS Cluster), but also in term of uses cases and deployments (how does it perform at scale, with which hardware, for which application, etc.) 3) Make possible to manage Garage from a REST API, possibly write a web GUI to make administration easier, 4) help people understand the reliability and storage density they will have for a specific Garage deployment, if possible through a simulator, 5) we might consider adding a system of quota to protect a cluster from a misbehaving user. - -**Correctness**. We know in theory that Garage's design works and scales. -But we still need to make sure that in practise our implementation is correct, and thus features these defined properties. -To convince ourselves, we consider verifying our consistency model implementation through [Jepsen](https://jepsen.io/). -We also plan to deploy Garage on multiple clusters and do a large serie of benchmarks. - -Please note that this roadmap is purely indicative, we are not committing to deliver these features. -We also don't know when v1.0 will be released, except "when it will be ready", but we would be happy if it could be by the end of 2022. -Finally, if you have some knowledge on one or more of thes points and would like to help, feel free to ping us on Matrix. -And see you soon for the next release! -- cgit v1.2.3 From df867dea639b8cf1f90e881780f5509a7fcf1a99 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Thu, 7 Apr 2022 18:05:43 +0200 Subject: Rework kube + conclusion --- content/blog/2022-v0.7-released.md | 178 ++++++------------------------------- 1 file changed, 25 insertions(+), 153 deletions(-) (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index 2b90373..be3a8a3 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -9,7 +9,7 @@ date=2022-04-04 --- -Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute. +Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute, and all requests were served by Garage without cache! Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0. But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. @@ -26,171 +26,40 @@ Besides bugfixes, there is two new features: a better Kubernetes integration and ## Kubernetes integration -Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on Kubernetes. -In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to simplify cluster discovery. -Garage can self-apply the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) to your cluster, or you can manage it manually. +Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S). +In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery. -Let's see practically how it works with a minimalistic example (not secured nor suitable for production). -You can run it on [minikube](https://minikube.sigs.k8s.io) if you want a more interactive reading. +CR discovery can be quickly enabled by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage config: -Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containing Garage's configuration (let's name it `config.yaml`): - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: garage-config - namespace: default -data: - garage.toml: |- - metadata_dir = "/mnt/fast" - data_dir = "/mnt/slow" - - replication_mode = "3" - - rpc_bind_addr = "[::]:3901" - rpc_secret = "" - - bootstrap_peers = [] - - kubernetes_namespace = "default" - kubernetes_service_name = "garage-daemon" - kubernetes_skip_crd = false - - [s3_api] - s3_region = "garage" - api_bind_addr = "[::]:3900" - root_domain = ".s3.garage.tld" - - [s3_web] - bind_addr = "[::]:3902" - root_domain = ".web.garage.tld" - index = "index.html" -``` - -The 3 important parameters are `kubernetes_namespace`, `kubernetes_service_name`, and `kubernetes_skip_crd`. -Configure them according to your planned deployment. -The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on startup. -In this example, we keep it to `false`, which means we allow Garage to automatically create the CRD. - -Apply this configuration on your cluster: - -```bash -kubectl apply -f config.yaml -``` - -Allowing Garage to create the CRD is not enough, the process must have enough permissions. -A quick unsecure way to add the permission is to create a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) to give admin rights to our local user, effectively breaking Kubernetes' security model (we name this file `admin.yml`): - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: garage-admin -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: -- apiGroup: rbac.authorization.k8s.io - kind: User - name: system:serviceaccount:default:default +```toml +kubernetes_namespace = "default" +kubernetes_service_name = "garage-daemon" ``` -Apply it: - -```bash -kubectl apply -f admin.yaml -``` +Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD). +If the CRD does not exist, Garage can create it for you. It is enabled by default but it requires some additional permissions. +If you prefer limiting accesses to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it: -Finally, we create a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) to run our service (`service.yaml`): - -```yaml -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: garage -spec: - selector: - matchLabels: - app: garage - serviceName: "garage" - replicas: 3 - template: - metadata: - labels: - app: garage - spec: - terminationGracePeriodSeconds: 10 - containers: - - name: garage - image: dxflrs/amd64_garage:v0.7.0 - ports: - - containerPort: 3900 - name: s3-api - - containerPort: 3902 - name: web-api - volumeMounts: - - name: fast - mountPath: /mnt/fast - - name: slow - mountPath: /mnt/slow - - name: etc - mountPath: /etc/garage.toml - subPath: garage.toml - volumes: - - name: etc - configMap: - name: garage-config - volumeClaimTemplates: - - metadata: - name: fast - spec: - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: 100Mi - - metadata: - name: slow - spec: - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: 100Mi +```toml +kubernetes_skip_crd = true ``` -Garage is a stateful program, so it needs a stable place to store its data and metadata. -This feature is provided by Kubernetes' [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used only from a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/), hence the choice of this K8S object to deploy our service. - -Kubernetes has many "drivers" for Persistent Volumes, for production uses we recommend **only** the `local` driver. -Using other drivers may lead to huge performance issues or data corruption, probably both in practice. - -In the example, we are claiming 2 volumes of 100MB. -We use 2 volumes instead of 1 because Garage separates its metadata from its data. -By having 2 volumes, you can reserve a smaller capacity on a SSD for the metadata and a larger capacity on a regular HDD for the data. -Do not forget to change the reserved capacity, 100MB is only suitable for testing. +If you want to try Garage on K8S, we currently only provide some basic [example files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/7e1ac51b580afa8e900206e7cc49791ed0a00d94/script/k8s). These files register a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/), a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding), and a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) with a [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). -*Note how we are mounting our ConfigMap: we need to set the `subpath` property to mount only the `garage.toml` file and not the whole `/etc` folder that would prevent K8S from writing its own files -in `/etc` and fail the pod.* - -You can apply this file with: - -```bash -kubectl apply -f service.yaml -``` - -Now, you are ready to interact with your cluster, each instance must have discovered the other ones: +Once these files deployed, you will be able to interact with Garage as follow: ```bash kubectl exec -it garage-0 --container garage -- /garage status # ==== HEALTHY NODES ==== -# ID Hostname Address Tags Zone Capacity -# e6284331c321a23c garage-0 172.17.0.5:3901 NO ROLE ASSIGNED -# 570ff9b0ed3648a7 garage-2 [::ffff:172.17.0.7]:3901 NO ROLE ASSIGNED -# e1990a2069429428 garage-1 [::ffff:172.17.0.6]:3901 NO ROLE ASSIGNED +# ID Hostname Address Tags Zone Capacity +# e628.. garage-0 172.17.0.5:3901 NO ROLE ASSIGNED +# 570f.. garage-2 172.17.0.7:3901 NO ROLE ASSIGNED +# e199.. garage-1 172.17.0.6:3901 NO ROLE ASSIGNED ``` -Of course, to have a full deployment, you will probably want to deploy a [Service](https://kubernetes.io/docs/concepts/services-networking/service/) in front of your cluster and/or a reverse proxy. +You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster. + +If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful how you handle your data in Kubernetes to not lose them. In the near future, we plan to release a proper Helm chart and write "best practises" on our documentation. If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster. We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl). @@ -265,5 +134,8 @@ In all cases, your feedback is welcome on our Matrix channel. ## Conclusion +This is only the first iteration of the Kubernetes and OpenTelemetry into Garage, so things are still a bit rough. +We plan to polish their integration in the coming months based on our experience and your feedback. - +You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will release a roadmap soon! +In the mean time, we hope you will enjoy Garave v0.7! -- cgit v1.2.3 From b7cfe7c09c3e2b42c0ef867c95600670ea7f2736 Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Fri, 8 Apr 2022 12:09:40 +0200 Subject: Move images & fix english in blog post --- content/blog/2022-v0.7-released.md | 141 -------------------------- content/blog/2022-v0.7-released/api_rate.png | Bin 0 -> 29088 bytes content/blog/2022-v0.7-released/apm.png | Bin 0 -> 119992 bytes content/blog/2022-v0.7-released/index.md | 143 +++++++++++++++++++++++++++ content/blog/2022-v0.7-released/writes.png | Bin 0 -> 17930 bytes 5 files changed, 143 insertions(+), 141 deletions(-) delete mode 100644 content/blog/2022-v0.7-released.md create mode 100644 content/blog/2022-v0.7-released/api_rate.png create mode 100644 content/blog/2022-v0.7-released/apm.png create mode 100644 content/blog/2022-v0.7-released/index.md create mode 100644 content/blog/2022-v0.7-released/writes.png (limited to 'content') diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md deleted file mode 100644 index be3a8a3..0000000 --- a/content/blog/2022-v0.7-released.md +++ /dev/null @@ -1,141 +0,0 @@ -+++ -title="Garage v0.7: Kubernetes and OpenTelemetry" -date=2022-04-04 -+++ - -*We just published Garage v0.7, our second public beta release. In this post, we do a quick tour of its 2 new features: Kubernetes integration and OpenTelemetry support.* - - - ---- - -Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute, and all requests were served by Garage without cache! -Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0. - -But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. -This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it! - -You can get this release using our binaries or the package provided by your distribution. -We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs). -Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). -Feel free to [reach out to us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage, we welcome maintainers and will upstream specific patches if that can help. If you already did package Garage, tell us and we'll add it to the documentation. - -Speaking about the changes of this new version, it obviously includes many bug fixes. -We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), take a look, we might have fixed something that annoyed you! -Besides bugfixes, there is two new features: a better Kubernetes integration and support for OpenTelemetry. - -## Kubernetes integration - -Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S). -In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery. - -CR discovery can be quickly enabled by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage config: - -```toml -kubernetes_namespace = "default" -kubernetes_service_name = "garage-daemon" -``` - -Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD). -If the CRD does not exist, Garage can create it for you. It is enabled by default but it requires some additional permissions. -If you prefer limiting accesses to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it: - -```toml -kubernetes_skip_crd = true -``` - -If you want to try Garage on K8S, we currently only provide some basic [example files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/7e1ac51b580afa8e900206e7cc49791ed0a00d94/script/k8s). These files register a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/), a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding), and a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) with a [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). - -Once these files deployed, you will be able to interact with Garage as follow: - -```bash -kubectl exec -it garage-0 --container garage -- /garage status -# ==== HEALTHY NODES ==== -# ID Hostname Address Tags Zone Capacity -# e628.. garage-0 172.17.0.5:3901 NO ROLE ASSIGNED -# 570f.. garage-2 172.17.0.7:3901 NO ROLE ASSIGNED -# e199.. garage-1 172.17.0.6:3901 NO ROLE ASSIGNED -``` - -You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster. - -If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful how you handle your data in Kubernetes to not lose them. In the near future, we plan to release a proper Helm chart and write "best practises" on our documentation. - -If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster. -We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl). - -## OpenTelemetry support - -[OpenTelemetry](https://opentelemetry.io/) standardizes how software generate and collect system telemetry, namely metrics, logs and traces. -By implementing this standard in Garage, we hope that it will help you to better monitor, manage and tune your cluster. -Note that to fully leverage this feature, you must be already familiar with monitoring stacks like [Prometheus](https://prometheus.io/)+[Grafana](https://grafana.com/) or [ElasticSearch](https://www.elastic.co/elasticsearch/)+[Kibana](https://www.elastic.co/kibana/). - -To activate OpenTelemetry on Garage, you must add to your configuration file the following entries (supposing that your collector is also on localhost): - -```toml -[admin] -api_bind_addr = "127.0.0.1:3903" -trace_sink = "http://localhost:4317" -``` - -We provide [some files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/telemetry) to help you quickly bootstrap a testing monitoring stack. -It includes a docker-compose file and a pre-configured Grafana dashboard. -You can use them if you want to reproduce the following examples. - -Now that your telemetry data is collected and stored, you can visualize it. - -Grafana is particularly adapted to understand how your cluster is performing from a "bird's eye view". -For example, the following graph shows S3 API calls sent to your node per time-unit, -you can use it to better understand how your users are interacting with your cluster. - -![A screenshot of a plot made by Grafana depicting the number of requests per time units grouped by endpoints](/images/blog/api_rate.png) - -Thanks to this graph, we know that starting at 14:55, an important upload has been started. -This upload is made of many small files, as we see many PutObject calls that are often used for small files. -It also has some large objects, as we observe some Multipart Uploads requests. -Conversely, at this time, no read are done as the corresponding read enpoints (ListBuckets, ListObjectsv2, etc.) receive 0 request per time unit. - - -Garage also collects metrics from lower level parts of the system. -You can use them to better understand how Garage is interacting with your OS and your hardware. - -![A screenshot of a plot made by Grafana depicting the write speed (in MB/s) during time.](/images/blog/writes.png) - -This plot has been captured at the same moment than the previous one. -We do not see a correlation between the writes and the API requests for the full upload but only for its beginning. -More precisely, it maps well to Multipart Uploads requests, and this is expected. -Large files (of the Multipart Uploads) will saturate the writes of your disk but the uploading of small files (via the PutObject endpoint) will be throttled by other parts of the system. - -This simple example covers only 2 metrics over the 20+ ones that we already defined, but we were still able to precisely describe our cluster usage and identifies where bottlenecks could be. -We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster. - -While metrics are good to have a large, general overview of your system, they are however not adapted to dig and pinpoint a specific performance issue on a specific code path. -Thankfully, we also have a solution for this problem: tracing. - -Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana, -we get the following visualization: - -![A screenshot of APM depicting the trace of a PutObject call](/images/blog/apm.png) - -On the top of the screenshot, we see the latency distribution of all PutObject requests. -We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run. -Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that -a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one). -Consequently, this request corresponds probably to a very tiny file. - -Below this first histogram, you can select the request you want to inspect, and then see its stacktrace on the bottom part. -You can break down this trace in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`). - -With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, to finally identify which part of the code took time. - -Keep in mind that this is our first iteration on telemetry for Garage, so things are a bit rough around the edges (step by step documentation is missing, our Grafana dashboard is a work in a progress, etc.). -In all cases, your feedback is welcome on our Matrix channel. - - -## Conclusion - -This is only the first iteration of the Kubernetes and OpenTelemetry into Garage, so things are still a bit rough. -We plan to polish their integration in the coming months based on our experience and your feedback. - -You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will release a roadmap soon! -In the mean time, we hope you will enjoy Garave v0.7! diff --git a/content/blog/2022-v0.7-released/api_rate.png b/content/blog/2022-v0.7-released/api_rate.png new file mode 100644 index 0000000..2c1534d Binary files /dev/null and b/content/blog/2022-v0.7-released/api_rate.png differ diff --git a/content/blog/2022-v0.7-released/apm.png b/content/blog/2022-v0.7-released/apm.png new file mode 100644 index 0000000..aade373 Binary files /dev/null and b/content/blog/2022-v0.7-released/apm.png differ diff --git a/content/blog/2022-v0.7-released/index.md b/content/blog/2022-v0.7-released/index.md new file mode 100644 index 0000000..1414a81 --- /dev/null +++ b/content/blog/2022-v0.7-released/index.md @@ -0,0 +1,143 @@ ++++ +title="Garage v0.7: Kubernetes and OpenTelemetry" +date=2022-04-04 ++++ + +*We just published Garage v0.7, our second public beta release. In this post, we do a quick tour of its 2 new features: Kubernetes integration and OpenTelemetry support.* + + + +--- + +Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, peaking at 100 views/minute, and all requests were served by Garage, without even using a caching frontend! +Since this event, we continued to improve Garage, and — 2 months after the initial release — we are happy to announce version 0.7.0. + +But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. +This is also our first time welcoming contributors external to the core team, and as we wish for Garage to be a community-driven project, we encourage it! + +You can get this release using our binaries or the package provided by your distribution. +We ship [statically compiled binaries](https://garagehq.deuxfleurs.fr/download/) for most common Linux architectures (amd64, i386, aarch64 and armv6) and associated [Docker containers](https://hub.docker.com/u/dxflrs). +Garage now is also packaged by third parties on some OS/distributions. We are currently aware of [FreeBSD](https://cgit.freebsd.org/ports/tree/www/garage/Makefile) and [AUR for Arch Linux](https://aur.archlinux.org/packages/garage). +Feel free to [reach out to us](mailto:garagehq@deuxfleurs.fr) if you are packaging (or planning to package) Garage; we welcome maintainers and will upstream specific patches if that can help. If you already did package Garage, please inform us and we'll add it to the documentation. + +Speaking about the changes of this new version, it obviously includes many bug fixes. +We listed them in our [changelogs](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases), so take a look, we might have fixed some issues you were having! +Besides bugfixes, there are two new major features in this release: better integration with Kubernetes, and support for observability via OpenTelemetry. + +## Kubernetes integration + +Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S). +In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery. + +CR discovery can be quickly enabled in Garage, by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage configuration file: + +```toml +kubernetes_namespace = "default" +kubernetes_service_name = "garage-daemon" +``` + +Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD). +If the CRD does not exist, Garage will create it for you. Automatic CRD creation is enabled by default, but it requires giving additional permissions to Garage to work. +If you prefer strictly controlling access to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it: + +```toml +kubernetes_skip_crd = true +``` + +If you want to try Garage on K8S, we currently only provide some basic [example files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/7e1ac51b580afa8e900206e7cc49791ed0a00d94/script/k8s). These files register a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/), a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding), and a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) with a [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). + +Once these files deployed, you will be able to interact with Garage as follow: + +```bash +kubectl exec -it garage-0 --container garage -- /garage status +# ==== HEALTHY NODES ==== +# ID Hostname Address Tags Zone Capacity +# e628.. garage-0 172.17.0.5:3901 NO ROLE ASSIGNED +# 570f.. garage-2 172.17.0.7:3901 NO ROLE ASSIGNED +# e199.. garage-1 172.17.0.6:3901 NO ROLE ASSIGNED +``` + +You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster. + +If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful of how you handle your data in Kubernetes in order not to lose it. In the near future, we plan to release a proper Helm chart and write "best practises" in our documentation. + +If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster, which is also well supported. +We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl). + +## OpenTelemetry support + +[OpenTelemetry](https://opentelemetry.io/) standardizes how software generates and collects system telemetry information, namely metrics, logs and traces. +By implementing this standard in Garage, we hope that it will help you to better monitor, manage and tune your cluster. +Note that to fully leverage this feature, you must be already familiar with monitoring stacks like [Prometheus](https://prometheus.io/)+[Grafana](https://grafana.com/) or [ElasticSearch](https://www.elastic.co/elasticsearch/)+[Kibana](https://www.elastic.co/kibana/). + +To activate OpenTelemetry on Garage, you must add to your configuration file the following entries (supposing that your collector is also on localhost): + +```toml +[admin] +api_bind_addr = "127.0.0.1:3903" +trace_sink = "http://localhost:4317" +``` + +The first line, `api_bind_address`, instructs Garage to expose an HTTP endpoint from which metrics can be obtained in Prometheus' data format. +The second line, `trace_sink`, instructs Garage to export tracing information to an OpenTelemetry collector at the given address. +These two options work independently and you can use them separately, depending on if you are interested only in metrics, traces, or both. + +We provide [some files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/telemetry) to help you quickly bootstrap a testing monitoring stack. +It includes a docker-compose file and a pre-configured Grafana dashboard. +You can use them if you want to reproduce the following examples. + +Grafana is particularly adapted to understand how your cluster is performing from a "bird's eye view". +For example, the following graph shows S3 API calls sent to your node per time-unit. +You can use it to better understand how your users are interacting with your cluster. + +![A screenshot of a plot made by Grafana depicting the number of requests per time units grouped by endpoints](api_rate.png) + +Thanks to this graph, we know that starting at 14:55, an important upload has been started. +This upload is made of many small files, as we see many PutObject calls that are often used for small files. +It also has some large objects, as we observe some multipart uploads requests. +Conversely, at this time, no read are done as the corresponding read enpoints (ListBuckets, ListObjectsV2, etc.) receive 0 request per time unit. + + +Garage also collects metrics from lower level parts of the system. +You can use them to better understand how Garage is interacting with your OS and your hardware. + +![A screenshot of a plot made by Grafana depicting the write speed (in MB/s) during time.](writes.png) + +This plot has been captured at the same moment than the previous one. +We do not see a correlation between the writes and the API requests for the full upload but only for its beginning. +More precisely, it maps well to multipart upload requests, and this is expected. +Large files (of the multipart uploads) will saturate the writes of your disk but the uploading of small files (via the PutObject endpoint) will be limited by other parts of the system. + +This simple example covers only 2 metrics over the 20+ ones that we already defined, but it still allowed us to precisely describe our cluster usage and identify where bottlenecks could be. +We are confident that cleverly using these metrics on a production cluster will give you many more valuable insights on your cluster. + +While metrics are good for having a large, general overview of your system, they are however not adapted for digging and pinpointing a specific performance issue on a specific code path. +Thankfully, we also have a solution for this problem: tracing. + +Using [Application Performance Monitoring](https://www.elastic.co/observability/application-performance-monitoring) (APM) in conjunction with Kibana, +we can get for instance the following visualization of what happens during a PutObject call (click to enlarge): + +[![A screenshot of APM depicting the trace of a PutObject call](apm.png)](apm.png) + +On the top of the screenshot, we see the latency distribution of all PutObject requests. +We learn that the selected request took ~1ms to execute, while 95% of all requests took less than 80ms to run. +Having some dispersion between requests is expected as Garage does not run on a strong real-time system, but in this case, you must also consider that +a request duration is impacted by the size of the object that is sent (a 10B object will be quicker to process than a 10MB one). +Consequently, this request probably corresponds to a very small file. + +Below this first histogram, you can select the request you want to inspect, and then see its trace on the bottom part. +The trace shown above can be broken down in 4 parts: fetching the API key to check authentication (`key get`), fetching the bucket identifier from its name (`bucket_alias get`), fetching the bucket configuration to check authorizations (`bucket_v2 get`), and finally inserting the object in the storage (`object insert`). + +With this example, we demonstrated that we can inspect Garage internals to find slow requests, then see which codepath has been taken by a request, and finally to identify which part of the code took time. + +Keep in mind that this is our first iteration on telemetry for Garage, so things are a bit rough around the edges (step by step documentation is missing, our Grafana dashboard is a work in a progress, etc.). +In all cases, your feedback is welcome on our Matrix channel. + + +## Conclusion + +This is only the first iteration of the Kubernetes and OpenTelemetry integrations in Garage, so things are still a bit rough. +We plan to polish their integration in the coming months based on our experience and your feedback. + +You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will soon release information on our roadmap! +In the mean time, we hope you will enjoy using Garage v0.7. diff --git a/content/blog/2022-v0.7-released/writes.png b/content/blog/2022-v0.7-released/writes.png new file mode 100644 index 0000000..bd3ab0e Binary files /dev/null and b/content/blog/2022-v0.7-released/writes.png differ -- cgit v1.2.3