From 06caa12d4992ec31ff231c013a463ab07eeb793c Mon Sep 17 00:00:00 2001 From: Jakub Jirutka Date: Sun, 7 May 2023 19:25:57 +0200 Subject: doc: Add information about Alpine Linux package to Quick Start --- doc/book/quick-start/_index.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) (limited to 'doc/book') diff --git a/doc/book/quick-start/_index.md b/doc/book/quick-start/_index.md index f01789a3..5863c09b 100644 --- a/doc/book/quick-start/_index.md +++ b/doc/book/quick-start/_index.md @@ -40,6 +40,25 @@ or if you want a build customized for your system, you can [build Garage from source](@/documentation/cookbook/from-source.md). +### Alpine Linux + +If you use Alpine Linux, you can simply install the +[garage](https://pkgs.alpinelinux.org/packages?name=garage) package from the +Alpine Linux repositories (available since v3.17): + +```bash +apk add garage +``` + +The default configuration file is installed to `/etc/garage.toml`. You can run +Garage using: `rc-service garage start`. If you don't specify `rpc_secret`, it +will be automatically replaced with a random string on the first start. + +Please note that this package is built without Consul discovery, Kubernetes +discovery, OpenTelemetry exporter, and K2V features (K2V will be enabled once +it's stable). + + ## Configuring and starting Garage ### Generating a first configuration file -- cgit v1.2.3 From 97eb3892746bda3b814433f63c1448c20812520d Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Thu, 15 Jun 2023 12:59:04 +0100 Subject: docs/apps: Added ejabberd section. --- doc/book/connect/apps/index.md | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) (limited to 'doc/book') diff --git a/doc/book/connect/apps/index.md b/doc/book/connect/apps/index.md index 4d556ff8..5e4b9223 100644 --- a/doc/book/connect/apps/index.md +++ b/doc/book/connect/apps/index.md @@ -11,6 +11,7 @@ In this section, we cover the following web applications: | [Peertube](#peertube) | ✅ | Supported with the website endpoint, proxifying private videos unsupported | | [Mastodon](#mastodon) | ✅ | Natively supported | | [Matrix](#matrix) | ✅ | Tested with `synapse-s3-storage-provider` | +| [ejabberd](#ejabberd) | ✅ | `mod_s3_upload` | | [Pixelfed](#pixelfed) | ❓ | Not yet tested | | [Pleroma](#pleroma) | ❓ | Not yet tested | | [Lemmy](#lemmy) | ✅ | Supported with pict-rs | @@ -474,6 +475,52 @@ And add a new line. For example, to run it every 10 minutes: *External link:* [matrix-media-repo Documentation > S3](https://docs.t2bot.io/matrix-media-repo/configuration/s3-datastore.html) +## ejabberd + +ejabberd is an XMPP server implementation which, with the `mod_s3_upload` +module in the [ejabberd-contrib](https://github.com/processone/ejabberd-contrib) +repository, can be integrated to store chat media files in Garage. + +For uploads, this module leverages presigned URLs - this allows XMPP clients to +directly send media to Garage. Receiving clients then retrieve this media +through the [static website](@/documentation/cookbook/exposing-websites.md) +functionality. + +As the data itself is publicly accessible to someone with knowledge of the +object URL - users are recommended to use +[E2EE](@/documentation/cookbook/encryption.md) to protect this data-at-rest +from unauthorized access. + +Install the module with: + +```bash +ejabberdctl module_install mod_s3_upload +``` + +Create the required key and bucket with: + +```bash +garage key new --name ejabberd +garage bucket create objects.xmpp-server.fr +garage bucket allow objects.xmpp-server.fr --read --write --key ejabberd +garage bucket website --allow objects.xmpp-server.fr +``` + +The module can then be configured with: + +``` + mod_s3_upload: + #bucket_url: https://objects.xmpp-server.fr.my-garage-instance.mydomain.tld + bucket_url: https://my-garage-instance.mydomain.tld/objects.xmpp-server.fr + access_key_id: GK... + access_key_secret: ... + region: garage + download_url: https://objects.xmpp-server.fr +``` + +Other configuration options can be found in the +[configuration YAML file](https://github.com/processone/ejabberd-contrib/blob/master/mod_s3_upload/conf/mod_s3_upload.yml). + ## Pixelfed [Pixelfed Technical Documentation > Configuration](https://docs.pixelfed.org/technical-documentation/env.html#filesystem) -- cgit v1.2.3 From 6af2cde23f5229302f122453aa50bf15df7625b4 Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Thu, 15 Jun 2023 12:59:21 +0100 Subject: cookbook/encryption.md: Added note on XMPP. --- doc/book/cookbook/encryption.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) (limited to 'doc/book') diff --git a/doc/book/cookbook/encryption.md b/doc/book/cookbook/encryption.md index 8d45a0ee..09438549 100644 --- a/doc/book/cookbook/encryption.md +++ b/doc/book/cookbook/encryption.md @@ -104,5 +104,8 @@ Implementations are very specific to the various applications. Examples: in Matrix are probably encrypted using symmetric encryption, with a key that is distributed in the end-to-end encrypted message that contains the link to the object. -- Aerogramme: use the user's password as a key to decrypt data in the user's bucket +- XMPP: clients normally support either OMEMO / OpenPGP for the E2EE of user + messages. Media files are encrypted per + [XEP-0454](https://xmpp.org/extensions/xep-0454.html). +- Aerogramme: use the user's password as a key to decrypt data in the user's bucket -- cgit v1.2.3 From fb971a5f01547516e9850f0fd34e42ad1d67c777 Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Thu, 15 Jun 2023 15:42:12 +0100 Subject: cookbook/encryption.md: Added Cyberduck note. --- doc/book/cookbook/encryption.md | 5 +++++ 1 file changed, 5 insertions(+) (limited to 'doc/book') diff --git a/doc/book/cookbook/encryption.md b/doc/book/cookbook/encryption.md index 09438549..21a5cbc6 100644 --- a/doc/book/cookbook/encryption.md +++ b/doc/book/cookbook/encryption.md @@ -109,3 +109,8 @@ Implementations are very specific to the various applications. Examples: [XEP-0454](https://xmpp.org/extensions/xep-0454.html). - Aerogramme: use the user's password as a key to decrypt data in the user's bucket + +- Cyberduck: comes with support for + [Cryptomator](https://docs.cyberduck.io/cryptomator/) which allows users to + create client-side vaults to encrypt files in before they are uploaded to a + cloud storage endpoint. -- cgit v1.2.3 From 185f9e78f3c2ba80424e9e9c7c8ffc58a005c91b Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Thu, 15 Jun 2023 17:57:14 +0100 Subject: operations/durability-repairs.md: Added note about randomized scrub times. --- doc/book/operations/durability-repairs.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) (limited to 'doc/book') diff --git a/doc/book/operations/durability-repairs.md b/doc/book/operations/durability-repairs.md index b8992f85..79888a5a 100644 --- a/doc/book/operations/durability-repairs.md +++ b/doc/book/operations/durability-repairs.md @@ -26,8 +26,11 @@ their content is correct, by verifying their hash. Any block found to be corrupt (e.g. by bitrot or by an accidental manipulation of the datastore) will be restored from another node that holds a valid copy. -A scrub is run automatically by Garage every 30 days. It can also be launched -manually using `garage repair scrub start`. +Scrubs are automatically scheduled by Garage to run every 25-35 days (the +actual time is randomized to spread load across nodes). The next scheduled run +can be viewed with `garage worker get`. + +A scrub can also be launched manually using `garage repair scrub start`. To view the status of an ongoing scrub, first find the task ID of the scrub worker using `garage worker list`. Then, run `garage worker info ` to @@ -79,7 +82,7 @@ To help make the difference between cases 1 and cases 2 and 3, you may use the `garage block info` command to see which objects hold a reference to each block. In the second case (transient errors), Garage will try to fetch the block again -after a certain time, so the error should disappear natuarlly. You can also +after a certain time, so the error should disappear naturally. You can also request Garage to try to fetch the block immediately using `garage block retry-now` if you have fixed the transient issue. -- cgit v1.2.3 From a5ae566e0be487553839c843ffde4909a8146b4a Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Thu, 15 Jun 2023 18:06:22 +0100 Subject: apps/index.md: Fixed endpoint URL example. --- doc/book/connect/apps/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/book') diff --git a/doc/book/connect/apps/index.md b/doc/book/connect/apps/index.md index 5e4b9223..83aadec2 100644 --- a/doc/book/connect/apps/index.md +++ b/doc/book/connect/apps/index.md @@ -586,7 +586,7 @@ secret_key = 'abcdef0123456789...' ``` PICTRS__STORE__TYPE=object_storage -PICTRS__STORE__ENDPOINT=http:/my-garage-instance.mydomain.tld:3900 +PICTRS__STORE__ENDPOINT=http://my-garage-instance.mydomain.tld:3900 PICTRS__STORE__BUCKET_NAME=pictrs-data PICTRS__STORE__REGION=garage PICTRS__STORE__ACCESS_KEY=GK... -- cgit v1.2.3 From 194549ca46736f56662a438c9b43340a58d1dcc3 Mon Sep 17 00:00:00 2001 From: Florian Klink Date: Fri, 14 Jul 2023 14:24:40 +0300 Subject: doc/book: fix typo --- doc/book/reference-manual/configuration.md | 4 ++-- doc/book/reference-manual/k2v.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) (limited to 'doc/book') diff --git a/doc/book/reference-manual/configuration.md b/doc/book/reference-manual/configuration.md index 20a79aa6..b916bb61 100644 --- a/doc/book/reference-manual/configuration.md +++ b/doc/book/reference-manual/configuration.md @@ -409,7 +409,7 @@ message that redirects the client to the correct region. ### `root_domain` {#root_domain} -The optionnal suffix to access bucket using vhost-style in addition to path-style request. +The optional suffix to access bucket using vhost-style in addition to path-style request. Note path-style requests are always enabled, whether or not vhost-style is configured. Configuring vhost-style S3 required a wildcard DNS entry, and possibly a wildcard TLS certificate, but might be required by softwares not supporting path-style requests. @@ -432,7 +432,7 @@ This endpoint does not suport TLS: a reverse proxy should be used to provide it. ### `root_domain` -The optionnal suffix appended to bucket names for the corresponding HTTP Host. +The optional suffix appended to bucket names for the corresponding HTTP Host. For instance, if `root_domain` is `web.garage.eu`, a bucket called `deuxfleurs.fr` will be accessible either with hostname `deuxfleurs.fr.web.garage.eu` diff --git a/doc/book/reference-manual/k2v.md b/doc/book/reference-manual/k2v.md index ed069b27..c01f641e 100644 --- a/doc/book/reference-manual/k2v.md +++ b/doc/book/reference-manual/k2v.md @@ -3,7 +3,7 @@ title = "K2V" weight = 100 +++ -Starting with version 0.7.2, Garage introduces an optionnal feature, K2V, +Starting with version 0.7.2, Garage introduces an optional feature, K2V, which is an alternative storage API designed to help efficiently store many small values in buckets (in opposition to S3 which is more designed to store large blobs). -- cgit v1.2.3 From 4d7bbf7878870d4a13f8b99bff5d893b486b2fa8 Mon Sep 17 00:00:00 2001 From: Max Justus Spransy Date: Mon, 24 Jul 2023 10:01:48 -0700 Subject: operations/durability-repairs-md: Fix typo --- doc/book/operations/durability-repairs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/book') diff --git a/doc/book/operations/durability-repairs.md b/doc/book/operations/durability-repairs.md index 79888a5a..498c8fda 100644 --- a/doc/book/operations/durability-repairs.md +++ b/doc/book/operations/durability-repairs.md @@ -4,7 +4,7 @@ weight = 30 +++ To ensure the best durability of your data and to fix any inconsistencies that may -pop up in a distributed system, Garage provides a serires of repair operations. +pop up in a distributed system, Garage provides a series of repair operations. This guide will explain the meaning of each of them and when they should be applied. -- cgit v1.2.3 From 24e533f2623ac6ebbdac92efa9c08b6092c59daf Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Tue, 8 Aug 2023 11:05:42 +0200 Subject: support {s3,web}.root_domains in /check endpoint --- doc/book/cookbook/reverse-proxy.md | 3 ++ doc/book/reference-manual/admin-api.md | 90 ++++++++++++++++++++++++++++++++-- 2 files changed, 90 insertions(+), 3 deletions(-) (limited to 'doc/book') diff --git a/doc/book/cookbook/reverse-proxy.md b/doc/book/cookbook/reverse-proxy.md index 9c833ad0..5d7355a4 100644 --- a/doc/book/cookbook/reverse-proxy.md +++ b/doc/book/cookbook/reverse-proxy.md @@ -428,3 +428,6 @@ https:// { reverse_proxy localhost:3902 192.168.1.2:3902 example.tld:3902 } ``` + +More information on how this endpoint is implemented in Garage is available +in the [Admin API Reference](@/documentation/reference-manual/admin-api.md) page. diff --git a/doc/book/reference-manual/admin-api.md b/doc/book/reference-manual/admin-api.md index 363bc886..6932ac60 100644 --- a/doc/book/reference-manual/admin-api.md +++ b/doc/book/reference-manual/admin-api.md @@ -39,11 +39,95 @@ Authorization: Bearer ## Administration API endpoints -### Metrics-related endpoints - -#### Metrics `GET /metrics` +### Metrics `GET /metrics` Returns internal Garage metrics in Prometheus format. +The metrics are directly documented when returned by the API. + +**Example:** + +``` +$ curl -i http://localhost:3903/metrics +HTTP/1.1 200 OK +content-type: text/plain; version=0.0.4 +content-length: 12145 +date: Tue, 08 Aug 2023 07:25:05 GMT + +# HELP api_admin_error_counter Number of API calls to the various Admin API endpoints that resulted in errors +# TYPE api_admin_error_counter counter +api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="400"} 1 +api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="404"} 3 +# HELP api_admin_request_counter Number of API calls to the various Admin API endpoints +# TYPE api_admin_request_counter counter +api_admin_request_counter{api_endpoint="CheckWebsiteEnabled"} 7 +api_admin_request_counter{api_endpoint="Health"} 3 +# HELP api_admin_request_duration Duration of API calls to the various Admin API endpoints +... +``` + +### Health `GET /health` + +Returns `200 OK` if enough nodes are up to have a quorum (ie. serve requests), +otherwise returns `503 Service Unavailable`. + +**Example:** + +``` +$ curl -i http://localhost:3903/health +HTTP/1.1 200 OK +content-type: text/plain +content-length: 102 +date: Tue, 08 Aug 2023 07:22:38 GMT + +Garage is fully operational +Consult the full health check API endpoint at /v0/health for more details +``` + +### On-demand TLS `GET /check` + +To prevent abuses for on-demand TLS, Caddy developpers have specified an endpoint that can be queried by the reverse proxy +to know if a given domain is allowed to get a certificate. Garage implements this endpoints to tell if a given domain is handled by Garage or is garbage. + +Garage responds with the following logic: + - If the domain matches the pattern `.`, returns 200 OK + - If the domain matches the pattern `.` and website is configured for ``, returns 200 OK + - If the domain matches the pattern `` and website is configured for ``, returns 200 OK + - Otherwise, returns 404 Not Found, 400 Bad Request or 5xx requests. + +*Note 1: because in the path-style URL mode, there is only one domain that is not known by Garage, hence it is not supported by this API endpoint. +You must manually declare the domain in your reverse-proxy. Idem for K2V.* + +*Note 2: buckets in a user's namespace are not supported yet by this endpoint. This is a limitation of this endpoint currently.* + +**Example:** Suppose a Garage instance configured with `s3_api.root_domain = .s3.garage.localhost` and `s3_web.root_domain = .web.garage.localhost`. + +With a private `media` bucket (name in the global namespace, website is disabled), the endpoint will feature the following behavior: + +``` +$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media.s3.garage.localhost +200 +$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media +400 +$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=media.web.garage.localhost +400 +``` + +With a public `example.com` bucket (name in the global namespace, website is activated), the endpoint will feature the following behavior: + +``` +$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com.s3.garage.localhost +200 +$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com +200 +$ curl -so /dev/null -w "%{http_code}" http://localhost:3903/check?domain=example.com.web.garage.localhost +200 +``` + + +**References:** + - [Using On-Demand TLS](https://caddyserver.com/docs/automatic-https#using-on-demand-tls) + - [Add option for a backend check to approve use of on-demand TLS](https://github.com/caddyserver/caddy/pull/1939) + - [Serving tens of thousands of domains over HTTPS with Caddy](https://caddy.community/t/serving-tens-of-thousands-of-domains-over-https-with-caddy/11179) ### Cluster operations -- cgit v1.2.3 From 245a0882e18bb4ed2cb45e60ee13447a123922d6 Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Tue, 1 Aug 2023 14:06:37 +0100 Subject: reverse-proxy.md: Added caching section for Caddy. --- doc/book/cookbook/reverse-proxy.md | 41 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) (limited to 'doc/book') diff --git a/doc/book/cookbook/reverse-proxy.md b/doc/book/cookbook/reverse-proxy.md index 9c833ad0..bacac2ef 100644 --- a/doc/book/cookbook/reverse-proxy.md +++ b/doc/book/cookbook/reverse-proxy.md @@ -378,6 +378,47 @@ admin.garage.tld { But at the same time, the `reverse_proxy` is very flexible. For a production deployment, you should [read its documentation](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) as it supports features like DNS discovery of upstreams, load balancing with checks, streaming parameters, etc. +### Caching + +Caddy can compiled with a +[cache plugin](https://github.com/caddyserver/cache-handler) which can be used +to provide a hot-cache at the webserver-level for static websites hosted by +Garage. + +This can be configured as follows: + +```caddy +# Caddy global configuration section +{ + # Bare minimum configuration to enable cache. + order cache before rewrite + + cache + + #cache + # allowed_http_verbs GET + # default_cache_control public + # ttl 8h + #} +} + +# Site specific section +https:// { + cache + + #cache { + # timeout { + # backend 30s + # } + #} + + reverse_proxy ... +} +``` + +Caching is a complicated subject, and the reader is encouraged to study the +available options provided by the plugin. + ### On-demand TLS Caddy supports a technique called -- cgit v1.2.3 From 7f7d85654d51d0d1c7221f1d8c1d83a024c17198 Mon Sep 17 00:00:00 2001 From: Jonathan Davies Date: Fri, 18 Aug 2023 18:02:19 +0100 Subject: backup.md: Added restic-android note. --- doc/book/connect/backup.md | 1 + 1 file changed, 1 insertion(+) (limited to 'doc/book') diff --git a/doc/book/connect/backup.md b/doc/book/connect/backup.md index f51dda30..d20c3c96 100644 --- a/doc/book/connect/backup.md +++ b/doc/book/connect/backup.md @@ -105,6 +105,7 @@ restic restore 79766175 --target /var/lib/postgresql Restic has way more features than the ones presented here. You can discover all of them by accessing its documentation from the link below. +Files on Android devices can also be backed up with [restic-android](https://github.com/lhns/restic-android). *External links:* [Restic Documentation > Amazon S3](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3) -- cgit v1.2.3 From 51011e68b16efc2232606bee47fcdc9e8a11068e Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Mon, 28 Aug 2023 12:20:34 +0200 Subject: move alpine linux info to binary package page --- doc/book/cookbook/binary-packages.md | 15 ++++++++++++++- doc/book/quick-start/_index.md | 22 +++------------------- 2 files changed, 17 insertions(+), 20 deletions(-) (limited to 'doc/book') diff --git a/doc/book/cookbook/binary-packages.md b/doc/book/cookbook/binary-packages.md index 606de2b6..0a6ad8fc 100644 --- a/doc/book/cookbook/binary-packages.md +++ b/doc/book/cookbook/binary-packages.md @@ -7,10 +7,23 @@ Garage is also available in binary packages on: ## Alpine Linux +If you use Alpine Linux, you can simply install the +[garage](https://pkgs.alpinelinux.org/packages?name=garage) package from the +Alpine Linux repositories (available since v3.17): + ```bash -apk install garage +apk add garage ``` +The default configuration file is installed to `/etc/garage.toml`. You can run +Garage using: `rc-service garage start`. If you don't specify `rpc_secret`, it +will be automatically replaced with a random string on the first start. + +Please note that this package is built without Consul discovery, Kubernetes +discovery, OpenTelemetry exporter, and K2V features (K2V will be enabled once +it's stable). + + ## Arch Linux Garage is available in the [AUR](https://aur.archlinux.org/packages/garage). diff --git a/doc/book/quick-start/_index.md b/doc/book/quick-start/_index.md index 46aaa9bc..4f974ea5 100644 --- a/doc/book/quick-start/_index.md +++ b/doc/book/quick-start/_index.md @@ -35,30 +35,14 @@ Place this binary somewhere in your `$PATH` so that you can invoke the `garage` command directly (for instance you can copy the binary in `/usr/local/bin` or in `~/.local/bin`). +You may also check whether your distribution already includes a +[binary package for Garage](@/documentation/cookbook/binary-packages.md). + If a binary of the last version is not available for your architecture, or if you want a build customized for your system, you can [build Garage from source](@/documentation/cookbook/from-source.md). -### Alpine Linux - -If you use Alpine Linux, you can simply install the -[garage](https://pkgs.alpinelinux.org/packages?name=garage) package from the -Alpine Linux repositories (available since v3.17): - -```bash -apk add garage -``` - -The default configuration file is installed to `/etc/garage.toml`. You can run -Garage using: `rc-service garage start`. If you don't specify `rpc_secret`, it -will be automatically replaced with a random string on the first start. - -Please note that this package is built without Consul discovery, Kubernetes -discovery, OpenTelemetry exporter, and K2V features (K2V will be enabled once -it's stable). - - ## Configuring and starting Garage ### Generating a first configuration file -- cgit v1.2.3