diff options
Diffstat (limited to 'content/blog/2022-v0.7-released.md')
| -rw-r--r-- | content/blog/2022-v0.7-released.md | 178 |
1 files changed, 25 insertions, 153 deletions
diff --git a/content/blog/2022-v0.7-released.md b/content/blog/2022-v0.7-released.md index 2b90373..be3a8a3 100644 --- a/content/blog/2022-v0.7-released.md +++ b/content/blog/2022-v0.7-released.md @@ -9,7 +9,7 @@ date=2022-04-04 --- -Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our intial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute. +Two months ago, we were impressed by the success of our open beta launch at FOSDEM and on Hacker News: [our initial post](https://garagehq.deuxfleurs.fr/blog/2022-introducing-garage/) lead to more than 40k views in 10 days, going up to 100 views/minute, and all requests were served by Garage without cache! Since this event, we continued to improve Garage, and - 2 months after the initial release - we are happy to announce version 0.7.0. But first, we would like to thank the contributors that made this new release possible: Alex, Jill, Max Audron, Maximilien, Quentin, Rune Henrisken, Steam, and trinity-1686a. @@ -26,171 +26,40 @@ Besides bugfixes, there is two new features: a better Kubernetes integration and ## Kubernetes integration -Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on Kubernetes. -In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to simplify cluster discovery. -Garage can self-apply the [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD) to your cluster, or you can manage it manually. +Before Garage v0.7.0, you had to deploy a Consul cluster or spawn a "coordinating" pod to deploy Garage on [Kubernetes](https://kubernetes.io) (K8S). +In this new version, Garage integrates a method to discover other peers by using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CR) to simplify cluster discovery. -Let's see practically how it works with a minimalistic example (not secured nor suitable for production). -You can run it on [minikube](https://minikube.sigs.k8s.io) if you want a more interactive reading. +CR discovery can be quickly enabled by configuring the name of the desired service (`kubernetes_namespace`) and which namespace to look for (`kubernetes_service_name`) in your Garage config: -Start by creating a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containing Garage's configuration (let's name it `config.yaml`): - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: garage-config - namespace: default -data: - garage.toml: |- - metadata_dir = "/mnt/fast" - data_dir = "/mnt/slow" - - replication_mode = "3" - - rpc_bind_addr = "[::]:3901" - rpc_secret = "<secret>" - - bootstrap_peers = [] - - kubernetes_namespace = "default" - kubernetes_service_name = "garage-daemon" - kubernetes_skip_crd = false - - [s3_api] - s3_region = "garage" - api_bind_addr = "[::]:3900" - root_domain = ".s3.garage.tld" - - [s3_web] - bind_addr = "[::]:3902" - root_domain = ".web.garage.tld" - index = "index.html" -``` - -The 3 important parameters are `kubernetes_namespace`, `kubernetes_service_name`, and `kubernetes_skip_crd`. -Configure them according to your planned deployment. -The last one controls wether you want to create the CRD manually or allow Garage to create it automatically on startup. -In this example, we keep it to `false`, which means we allow Garage to automatically create the CRD. - -Apply this configuration on your cluster: - -```bash -kubectl apply -f config.yaml -``` - -Allowing Garage to create the CRD is not enough, the process must have enough permissions. -A quick unsecure way to add the permission is to create a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) to give admin rights to our local user, effectively breaking Kubernetes' security model (we name this file `admin.yml`): - -```yaml -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: garage-admin -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: -- apiGroup: rbac.authorization.k8s.io - kind: User - name: system:serviceaccount:default:default +```toml +kubernetes_namespace = "default" +kubernetes_service_name = "garage-daemon" ``` -Apply it: - -```bash -kubectl apply -f admin.yaml -``` +Custom Resources must be defined *a priori* with [Custom Resource Definition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRD). +If the CRD does not exist, Garage can create it for you. It is enabled by default but it requires some additional permissions. +If you prefer limiting accesses to your K8S cluster, you can create the resource manually and prevent Garage from automatically creating it: -Finally, we create a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) to run our service (`service.yaml`): - -```yaml -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: garage -spec: - selector: - matchLabels: - app: garage - serviceName: "garage" - replicas: 3 - template: - metadata: - labels: - app: garage - spec: - terminationGracePeriodSeconds: 10 - containers: - - name: garage - image: dxflrs/amd64_garage:v0.7.0 - ports: - - containerPort: 3900 - name: s3-api - - containerPort: 3902 - name: web-api - volumeMounts: - - name: fast - mountPath: /mnt/fast - - name: slow - mountPath: /mnt/slow - - name: etc - mountPath: /etc/garage.toml - subPath: garage.toml - volumes: - - name: etc - configMap: - name: garage-config - volumeClaimTemplates: - - metadata: - name: fast - spec: - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: 100Mi - - metadata: - name: slow - spec: - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: 100Mi +```toml +kubernetes_skip_crd = true ``` -Garage is a stateful program, so it needs a stable place to store its data and metadata. -This feature is provided by Kubernetes' [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used only from a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/), hence the choice of this K8S object to deploy our service. - -Kubernetes has many "drivers" for Persistent Volumes, for production uses we recommend **only** the `local` driver. -Using other drivers may lead to huge performance issues or data corruption, probably both in practice. - -In the example, we are claiming 2 volumes of 100MB. -We use 2 volumes instead of 1 because Garage separates its metadata from its data. -By having 2 volumes, you can reserve a smaller capacity on a SSD for the metadata and a larger capacity on a regular HDD for the data. -Do not forget to change the reserved capacity, 100MB is only suitable for testing. +If you want to try Garage on K8S, we currently only provide some basic [example files](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/7e1ac51b580afa8e900206e7cc49791ed0a00d94/script/k8s). These files register a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/), a [ClusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding), and a [StatefulSet](https://kubernetes.io/fr/docs/concepts/workloads/controllers/statefulset/) with a [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). -*Note how we are mounting our ConfigMap: we need to set the `subpath` property to mount only the `garage.toml` file and not the whole `/etc` folder that would prevent K8S from writing its own files -in `/etc` and fail the pod.* - -You can apply this file with: - -```bash -kubectl apply -f service.yaml -``` - -Now, you are ready to interact with your cluster, each instance must have discovered the other ones: +Once these files deployed, you will be able to interact with Garage as follow: ```bash kubectl exec -it garage-0 --container garage -- /garage status # ==== HEALTHY NODES ==== -# ID Hostname Address Tags Zone Capacity -# e6284331c321a23c garage-0 172.17.0.5:3901 NO ROLE ASSIGNED -# 570ff9b0ed3648a7 garage-2 [::ffff:172.17.0.7]:3901 NO ROLE ASSIGNED -# e1990a2069429428 garage-1 [::ffff:172.17.0.6]:3901 NO ROLE ASSIGNED +# ID Hostname Address Tags Zone Capacity +# e628.. garage-0 172.17.0.5:3901 NO ROLE ASSIGNED +# 570f.. garage-2 172.17.0.7:3901 NO ROLE ASSIGNED +# e199.. garage-1 172.17.0.6:3901 NO ROLE ASSIGNED ``` -Of course, to have a full deployment, you will probably want to deploy a [Service](https://kubernetes.io/docs/concepts/services-networking/service/) in front of your cluster and/or a reverse proxy. +You can then follow the [regular documentation](https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/#creating-a-cluster-layout) to complete the configuration of your cluster. + +If you target a production deployment, you should avoid binding admin rights to your cluster to create Garage's CRD. You will also need to expose some [Services](https://kubernetes.io/docs/concepts/services-networking/service/) to make your cluster reachable. Keep also in mind that Garage is a stateful service, so you must be very careful how you handle your data in Kubernetes to not lose them. In the near future, we plan to release a proper Helm chart and write "best practises" on our documentation. If Kubernetes is not your thing, know that we are running Garage on a Nomad+Consul cluster. We have not documented it yet but you can get a look at [our Nomad service](https://git.deuxfleurs.fr/Deuxfleurs/infrastructure/src/commit/1e5e4af35c073d04698bb10dd4ad1330d6c62a0d/app/garage/deploy/garage.hcl). @@ -265,5 +134,8 @@ In all cases, your feedback is welcome on our Matrix channel. ## Conclusion +This is only the first iteration of the Kubernetes and OpenTelemetry into Garage, so things are still a bit rough. +We plan to polish their integration in the coming months based on our experience and your feedback. - +You may also ask yourself what will be the other works we plan to conduct: stay tuned, we will release a roadmap soon! +In the mean time, we hope you will enjoy Garave v0.7! |