aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorQuentin Dufour <quentin@deuxfleurs.fr>2022-04-07 18:05:43 +0200
committerQuentin Dufour <quentin@deuxfleurs.fr>2022-04-07 18:05:43 +0200
commitdf867dea639b8cf1f90e881780f5509a7fcf1a99 (patch)
treee1dcc20d779312725715ca77fd90862728a6e7d0
parentb93391bfa6666b224f46946c0455882d3c9324fb (diff)
downloadgaragehq.deuxfleurs.fr-df867dea639b8cf1f90e881780f5509a7fcf1a99.tar.gz
garagehq.deuxfleurs.fr-df867dea639b8cf1f90e881780f5509a7fcf1a99.zip
Rework kube + conclusion
-rw-r--r--content/blog/2022-v0.7-released.md178
1 files changed, 25 insertions, 153 deletions
diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md
index 2b90373..be3a8a3 100644
--- a/content/blog/2022-v0.7-released.md
+++ b/content/blog/2022-v0.7-released.md
@@ -9,7 +9,7 @@ date=2022-04-04
---
-Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute.
+Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute, and all requests were served by Garage without cache!
Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0.
But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a.
@@ -26,171 +26,40 @@ Besides bugfixes, there is two new features: a better Kubernetes integration and
## Kubernetes integration
-Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on Kubernetes.
-In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to simplify cluster discovery.
-Garage can self-apply the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) to your cluster, or you can manage it manually.
+Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S).
+In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery.
-Let's see practically how it works with a minimalistic example (not secured nor suitable for production).
-You can run it on [minikube](https://minikube.sigs.k8s.io) if you want a more interactive reading.
+CR discovery can be quickly enabled by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage config:
-Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containing Garage's configuration (let's name it `config.yaml`):
-
-```yaml
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: garage-config
- namespace: default
-data:
- garage.toml: |-
- metadata_dir = "/mnt/fast"
- data_dir = "/mnt/slow"
-
- replication_mode = "3"
-
- rpc_bind_addr = "[::]:3901"
- rpc_secret = "<secret>"
-
- bootstrap_peers = []
-
- kubernetes_namespace = "default"
- kubernetes_service_name = "garage-daemon"
- kubernetes_skip_crd = false
-
- [s3_api]
- s3_region = "garage"
- api_bind_addr = "[::]:3900"
- root_domain = ".s3.garage.tld"
-
- [s3_web]
- bind_addr = "[::]:3902"
- root_domain = ".web.garage.tld"
- index = "index.html"
-```
-
-The 3 important parameters are `kubernetes_namespace`, `kubernetes_service_name`, and `kubernetes_skip_crd`.
-Configure them according to your planned deployment.
-The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on startup.
-In this example, we keep it to `false`, which means we allow Garage to automatically create the CRD.
-
-Apply this configuration on your cluster:
-
-```bash
-kubectl apply -f config.yaml
-```
-
-Allowing Garage to create the CRD is not enough, the process must have enough permissions.
-A quick unsecure way to add the permission is to create a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) to give admin rights to our local user, effectively breaking Kubernetes' security model (we name this file `admin.yml`):
-
-```yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
- name: garage-admin
-roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: ClusterRole
- name: cluster-admin
-subjects:
-- apiGroup: rbac.authorization.k8s.io
- kind: User
- name: system:serviceaccount:default:default
+```toml
+kubernetes_namespace = "default"
+kubernetes_service_name = "garage-daemon"
```
-Apply it:
-
-```bash
-kubectl apply -f admin.yaml
-```
+Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD).
+If the CRD does not exist, Garage can create it for you. It is enabled by default but it requires some additional permissions.
+If you prefer limiting accesses to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it:
-Finally, we create a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) to run our service (`service.yaml`):
-
-```yaml
-apiVersion: apps/v1
-kind: StatefulSet
-metadata:
- name: garage
-spec:
- selector:
- matchLabels:
- app: garage
- serviceName: "garage"
- replicas: 3
- template:
- metadata:
- labels:
- app: garage
- spec:
- terminationGracePeriodSeconds: 10
- containers:
- - name: garage
- image: dxflrs/amd64_garage:v0.7.0
- ports:
- - containerPort: 3900
- name: s3-api
- - containerPort: 3902
- name: web-api
- volumeMounts:
- - name: fast
- mountPath: /mnt/fast
- - name: slow
- mountPath: /mnt/slow
- - name: etc
- mountPath: /etc/garage.toml
- subPath: garage.toml
- volumes:
- - name: etc
- configMap:
- name: garage-config
- volumeClaimTemplates:
- - metadata:
- name: fast
- spec:
- accessModes: [ "ReadWriteOnce" ]
- resources:
- requests:
- storage: 100Mi
- - metadata:
- name: slow
- spec:
- accessModes: [ "ReadWriteOnce" ]
- resources:
- requests:
- storage: 100Mi
+```toml
+kubernetes_skip_crd = true
```
-Garage is a stateful program, so it needs a stable place to store its data and metadata.
-This feature is provided by Kubernetes' [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used only from a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/), hence the choice of this K8S object to deploy our service.
-
-Kubernetes has many "drivers" for Persistent Volumes, for production uses we recommend **only** the `local` driver.
-Using other drivers may lead to huge performance issues or data corruption, probably both in practice.
-
-In the example, we are claiming 2 volumes of 100MB.
-We use 2 volumes instead of 1 because Garage separates its metadata from its data.
-By having 2 volumes, you can reserve a smaller capacity on a SSD for the metadata and a larger capacity on a regular HDD for the data.
-Do not forget to change the reserved capacity, 100MB is only suitable for testing.
+If you want to try Garage on K8S, we currently only provide some basic [example files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/7e1ac51b580afa8e900206e7cc49791ed0a00d94/script/k8s). These files register a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/), a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding), and a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) with a [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/).
-*Note how we are mounting our ConfigMap: we need to set the `subpath` property to mount only the `garage.toml` file and not the whole `/etc` folder that would prevent K8S from writing its own files
-in `/etc` and fail the pod.*
-
-You can apply this file with:
-
-```bash
-kubectl apply -f service.yaml
-```
-
-Now, you are ready to interact with your cluster, each instance must have discovered the other ones:
+Once these files deployed, you will be able to interact with Garage as follow:
```bash
kubectl exec -it garage-0 --container garage -- /garage status
# ==== HEALTHY NODES ====
-# ID Hostname Address Tags Zone Capacity
-# e6284331c321a23c garage-0 172.17.0.5:3901 NO ROLE ASSIGNED
-# 570ff9b0ed3648a7 garage-2 [::ffff:172.17.0.7]:3901 NO ROLE ASSIGNED
-# e1990a2069429428 garage-1 [::ffff:172.17.0.6]:3901 NO ROLE ASSIGNED
+# ID Hostname Address Tags Zone Capacity
+# e628.. garage-0 172.17.0.5:3901 NO ROLE ASSIGNED
+# 570f.. garage-2 172.17.0.7:3901 NO ROLE ASSIGNED
+# e199.. garage-1 172.17.0.6:3901 NO ROLE ASSIGNED
```
-Of course, to have a full deployment, you will probably want to deploy a [Service](https://kubernetes.io/docs/concepts/services-networking/service/) in front of your cluster and/or a reverse proxy.
+You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster.
+
+If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful how you handle your data in Kubernetes to not lose them. In the near future, we plan to release a proper Helm chart and write "best practises" on our documentation.
If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster.
We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl).
@@ -265,5 +134,8 @@ In all cases, your feedback is welcome on our Matrix channel.
## Conclusion
+This is only the first iteration of the Kubernetes and OpenTelemetry into Garage, so things are still a bit rough.
+We plan to polish their integration in the coming months based on our experience and your feedback.
-
+You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will release a roadmap soon!
+In the mean time, we hope you will enjoy Garave v0.7!