aboutsummaryrefslogtreecommitdiff
path: root/doc/book
diff options
context:
space:
mode:
Diffstat (limited to 'doc/book')
-rw-r--r--doc/book/connect/apps/index.md4
-rw-r--r--doc/book/cookbook/real-world.md10
-rw-r--r--doc/book/design/goals.md2
-rw-r--r--doc/book/development/devenv.md2
-rw-r--r--doc/book/development/miscellaneous-notes.md8
-rw-r--r--doc/book/development/release-process.md55
-rw-r--r--doc/book/operations/durability-repairs.md4
-rw-r--r--doc/book/operations/layout.md12
-rw-r--r--doc/book/operations/upgrading.md2
-rw-r--r--doc/book/quick-start/_index.md7
-rw-r--r--doc/book/reference-manual/configuration.md22
11 files changed, 60 insertions, 68 deletions
diff --git a/doc/book/connect/apps/index.md b/doc/book/connect/apps/index.md
index f67a29c9..c8571fac 100644
--- a/doc/book/connect/apps/index.md
+++ b/doc/book/connect/apps/index.md
@@ -146,7 +146,7 @@ Keep the Key ID and the Secret key in a pad, they will be needed later.
We need two buckets, one for normal videos (named peertube-video) and one for webtorrent videos (named peertube-playlist).
```bash
-garage bucket create peertube-video
+garage bucket create peertube-videos
garage bucket create peertube-playlist
```
@@ -216,7 +216,7 @@ object_storage:
# Same settings but for webtorrent videos
videos:
- bucket_name: 'peertube-video'
+ bucket_name: 'peertube-videos'
prefix: ''
# You must fill this field to make Peertube use our reverse proxy/website logic
base_url: 'http://peertube-videos.web.garage.localhost'
diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md
index ea4ce1f9..ce0abddd 100644
--- a/doc/book/cookbook/real-world.md
+++ b/doc/book/cookbook/real-world.md
@@ -85,14 +85,14 @@ to store 2 TB of data in total.
## Get a Docker image
Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
-We encourage you to use a fixed tag (eg. `v0.9.0`) and not the `latest` tag.
-For this example, we will use the latest published version at the time of the writing which is `v0.9.0` but it's up to you
+We encourage you to use a fixed tag (eg. `v0.9.1`) and not the `latest` tag.
+For this example, we will use the latest published version at the time of the writing which is `v0.9.1` but it's up to you
to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
For example:
```
-sudo docker pull dxflrs/garage:v0.9.0
+sudo docker pull dxflrs/garage:v0.9.1
```
## Deploying and configuring Garage
@@ -157,7 +157,7 @@ docker run \
-v /etc/garage.toml:/etc/garage.toml \
-v /var/lib/garage/meta:/var/lib/garage/meta \
-v /var/lib/garage/data:/var/lib/garage/data \
- dxflrs/garage:v0.9.0
+ dxflrs/garage:v0.9.1
```
With this command line, Garage should be started automatically at each boot.
@@ -171,7 +171,7 @@ If you want to use `docker-compose`, you may use the following `docker-compose.y
version: "3"
services:
garage:
- image: dxflrs/garage:v0.9.0
+ image: dxflrs/garage:v0.9.1
network_mode: "host"
restart: unless-stopped
volumes:
diff --git a/doc/book/design/goals.md b/doc/book/design/goals.md
index 78ac7978..4efb6349 100644
--- a/doc/book/design/goals.md
+++ b/doc/book/design/goals.md
@@ -48,7 +48,5 @@ locations. They use Garage themselves for the following tasks:
- As a backup target using `rclone` and `restic`
-- In the Drone continuous integration platform to store task logs
-
The Deuxfleurs Garage cluster is a multi-site cluster currently composed of
9 nodes in 3 physical locations.
diff --git a/doc/book/development/devenv.md b/doc/book/development/devenv.md
index dd3bdec0..88f8ba06 100644
--- a/doc/book/development/devenv.md
+++ b/doc/book/development/devenv.md
@@ -80,7 +80,7 @@ nix-build \
--git_version $(git rev-parse HEAD)
```
-*The result is located in `result/bin`. You can pass arguments to cross compile: check `.drone.yml` for examples.*
+*The result is located in `result/bin`. You can pass arguments to cross compile: check `.woodpecker/release.yml` for examples.*
If you modify a `Cargo.toml` or regenerate any `Cargo.lock`, you must run `cargo2nix`:
diff --git a/doc/book/development/miscellaneous-notes.md b/doc/book/development/miscellaneous-notes.md
index f0083ae5..a421943f 100644
--- a/doc/book/development/miscellaneous-notes.md
+++ b/doc/book/development/miscellaneous-notes.md
@@ -81,12 +81,9 @@ Our cache will be checked.
- http://www.lpenz.org/articles/nixchannel/index.html
-## Drone
+## Woodpecker
-Do not try to set a build as trusted from the interface or the CLI tool,
-your request would be ignored. Instead, directly edit the database (table `repos`, column `repo_trusted`).
-
-Drone can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner.
+Woodpecker can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner.
## Building Docker containers
@@ -99,3 +96,4 @@ We were:
- Unable to use the kaniko container provided by Google as we can't run arbitrary logic: we need to put our secret in .docker/config.json.
Finally we chose to build kaniko through nix and use it in a `nix-shell`.
+We then switched to using kaniko from nixpkgs when it was packaged.
diff --git a/doc/book/development/release-process.md b/doc/book/development/release-process.md
index 3fed4add..0c6701c0 100644
--- a/doc/book/development/release-process.md
+++ b/doc/book/development/release-process.md
@@ -42,7 +42,7 @@ and the docker containers on Docker Hub.
## Automation
-We automated our release process with Nix and Drone to make it more reliable.
+We automated our release process with Nix and Woodpecker to make it more reliable.
Here we describe how we have done in case you want to debug or improve it.
### Caching build steps
@@ -62,52 +62,31 @@ Sending to the cache is done through `nix copy`, for example:
nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' result
```
-*Note that you need the signing key. In our case, it is stored as a secret in Drone.*
+*The signing key possessed by the Garage maintainers is required to update the Nix cache.*
-The previous command will only send the built packet and not its dependencies.
-To send its dependency, a tool named `nix-copy-closure` has been created but it is not compatible with the S3 protocol.
-
-Instead, you can use the following commands to list all the runtime dependencies:
+The previous command will only send the built package and not its dependencies.
+In the case of our CI pipeline, we want to cache all intermediate build steps
+as well. This can be done using this quite involved command (here as an example
+for the `pkgs.amd64.relase` package):
```bash
-nix copy \
- --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
- $(nix-store -qR result/)
+nix copy -j8 \
+ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/nix-signing-key.sec' \
+ $(nix path-info pkgs.amd64.release --file default.nix --derivation --recursive | sed 's/\.drv$/.drv^*/')
```
-*We could also write this expression with xargs but this tool is not available in our container.*
-
-But in certain cases, we want to cache compile time dependencies also.
-For example, the Nix project does not provide binaries for cross compiling to i686 and thus we need to compile gcc on our own.
-We do not want to compile gcc each time, so even if it is a compile time dependency, we want to cache it.
+This command will simultaneously build all of the required Nix paths (using at
+most 8 parallel Nix builder jobs) and send the resulting objects to the cache.
-This time, the command is a bit more involved:
-
-```bash
-nix copy --to \
- 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
- $(nix-store -qR --include-outputs \
- $(nix-instantiate))
-```
-
-This is the command we use in our CI as we expect the final binary to change, so we mainly focus on
-caching our development dependencies.
-
-*Currently there is no automatic garbage collection of the cache: we should monitor its growth.
-Hopefully, we can erase it totally without breaking any build, the next build will only be slower.*
-
-In practise, we concluded that we do not want to cache all the compilation dependencies.
-Instead, we want to cache the toolchain we use to build Garage each time we change it.
-So we removed from Drone any automatic update of the cache and instead handle them manually with:
+This can be run for all the Garage packages we build using the following command:
```
source ~/.awsrc
-nix-shell --run 'refresh_toolchain'
+nix-shell --attr cache --run 'refresh_cache'
```
-Internally, it will run `nix-build` on `nix/toolchain.nix` and send the output plus its depedencies to the cache.
-
-To erase the cache:
+We don't automate this step at each CI build, as *there is currently no automatic garbage collection of the cache.*
+This means we should also monitor the cache's size; if it ever becomes too big we can erase it with:
```
mc rm --recursive --force 'garage/nix/'
@@ -157,9 +136,9 @@ nix-shell --run refresh_index
If you want to compile for different architectures, you will need to repeat all these commands for each architecture.
-**In practise, and except for debugging, you will never directly run these commands. Release is handled by drone**
+**In practice, and except for debugging, you will never directly run these commands. Release is handled by Woodpecker.**
-### Drone
+### Drone (obsolete)
Our instance is available at [https://drone.deuxfleurs.fr](https://drone.deuxfleurs.fr).
You need an account on [https://git.deuxfleurs.fr](https://git.deuxfleurs.fr) to use it.
diff --git a/doc/book/operations/durability-repairs.md b/doc/book/operations/durability-repairs.md
index b0d2c78a..578899a8 100644
--- a/doc/book/operations/durability-repairs.md
+++ b/doc/book/operations/durability-repairs.md
@@ -49,7 +49,7 @@ verifications. Of course, scrubbing the entire data store will also take longer.
## Block check and resync
In some cases, nodes hold a reference to a block but do not actually have the block
-stored on disk. Conversely, they may also have on disk blocks that are not referenced
+stored on disk. Conversely, they may also have on-disk blocks that are not referenced
any more. To fix both cases, a block repair may be run with `garage repair blocks`.
This will scan the entire block reference counter table to check that the blocks
exist on disk, and will scan the entire disk store to check that stored blocks
@@ -95,7 +95,7 @@ using the `garage block purge` command.
In [multi-HDD setups](@/documentation/operations/multi-hdd.md), to ensure that
data blocks are well balanced between storage locations, you may run a
-rebalance operation using `garage repair rebalance`. This is usefull when
+rebalance operation using `garage repair rebalance`. This is useful when
adding storage locations or when capacities of the storage locations have been
changed. Once this is finished, Garage will know for each block of a single
possible location where it can be, which can increase access speed. This
diff --git a/doc/book/operations/layout.md b/doc/book/operations/layout.md
index ee05aba1..cf1372b0 100644
--- a/doc/book/operations/layout.md
+++ b/doc/book/operations/layout.md
@@ -13,7 +13,7 @@ In Garage, all of the data that can be stored in a given cluster is divided
into slices which we call *partitions*. Each partition is stored by
one or several nodes in the cluster
(see [`replication_mode`](@/documentation/reference-manual/configuration.md#replication_mode)).
-The layout determines the correspondence between these partition,
+The layout determines the correspondence between these partitions,
which exist on a logical level, and actual storage nodes.
## How cluster layouts work in Garage
@@ -94,10 +94,10 @@ follow the following recommendations:
## Understanding unexpected layout calculations
When adding, removing or modifying nodes in a cluster layout, sometimes
-unexpected assigntations of partitions to node can occur. These assignations
-are in fact normal and logical, given the objectives of the algorihtm. Indeed,
-**the layout algorithm prioritizes moving less data between nodes over the fact
-of achieving equal distribution of load. It also tries to use all links between
+unexpected assignations of partitions to node can occur. These assignations
+are in fact normal and logical, given the objectives of the algorithm. Indeed,
+**the layout algorithm prioritizes moving less data between nodes over
+achieving equal distribution of load. It also tries to use all links between
pairs of nodes in equal proportions when moving data.** This section presents
two examples and illustrates how one can control Garage's behavior to obtain
the desired results.
@@ -270,5 +270,5 @@ that is moved to node1).
This illustrates the second principle of the layout computation: **if there is
a choice in moving data out of some nodes, then all links between pairs of
nodes are used in equal proportions** (this is approximately true, there is
-randomness in the algorihtm to achieve this so there might be some small
+randomness in the algorithm to achieve this so there might be some small
fluctuations, as we see above).
diff --git a/doc/book/operations/upgrading.md b/doc/book/operations/upgrading.md
index 9a738282..6b6ea26d 100644
--- a/doc/book/operations/upgrading.md
+++ b/doc/book/operations/upgrading.md
@@ -9,7 +9,7 @@ On a new version release, there is 2 possibilities:
- protocols and data structures remained the same ➡️ this is a **minor upgrade**
- protocols or data structures changed ➡️ this is a **major upgrade**
-You can quickly now what type of update you will have to operate by looking at the version identifier:
+You can quickly know what type of update you will have to operate by looking at the version identifier:
when we require our users to do a major upgrade, we will always bump the first nonzero component of the version identifier
(e.g. from v0.7.2 to v0.8.0).
Conversely, for versions that only require a minor upgrade, the first nonzero component will always stay the same (e.g. from v0.8.0 to v0.8.1).
diff --git a/doc/book/quick-start/_index.md b/doc/book/quick-start/_index.md
index 1b129f36..cf6eabde 100644
--- a/doc/book/quick-start/_index.md
+++ b/doc/book/quick-start/_index.md
@@ -110,10 +110,11 @@ garage -c path/to/garage.toml server
If you have placed the `garage.toml` file in `/etc` (its default location), you can simply run `garage server`.
-You can tune Garage's verbosity as follows (from less verbose to more verbose):
+You can tune Garage's verbosity by setting the `RUST_LOG=` environment variable. \
+Available log levels are (from less verbose to more verbose): `error`, `warn`, `info` *(default)*, `debug` and `trace`.
-```
-RUST_LOG=garage=info garage server
+```bash
+RUST_LOG=garage=info garage server # default
RUST_LOG=garage=debug garage server
RUST_LOG=garage=trace garage server
```
diff --git a/doc/book/reference-manual/configuration.md b/doc/book/reference-manual/configuration.md
index 18d160bb..5e12a7da 100644
--- a/doc/book/reference-manual/configuration.md
+++ b/doc/book/reference-manual/configuration.md
@@ -394,7 +394,7 @@ Compression is done synchronously, setting a value too high will add latency to
This value can be different between nodes, compression is done by the node which receive the
API call.
-#### `rpc_secret`, `rpc_secret_file` or `GARAGE_RPC_SECRET` (env) {#rpc_secret}
+#### `rpc_secret`, `rpc_secret_file` or `GARAGE_RPC_SECRET`, `GARAGE_RPC_SECRET_FILE` (env) {#rpc_secret}
Garage uses a secret key, called an RPC secret, that is shared between all
nodes of the cluster in order to identify these nodes and allow them to
@@ -406,6 +406,9 @@ Since Garage `v0.8.2`, the RPC secret can also be stored in a file whose path is
given in the configuration variable `rpc_secret_file`, or specified as an
environment variable `GARAGE_RPC_SECRET`.
+Since Garage `v0.8.5` and `v0.9.1`, you can also specify the path of a file
+storing the secret as the `GARAGE_RPC_SECRET_FILE` environment variable.
+
#### `rpc_bind_addr` {#rpc_bind_addr}
The address and port on which to bind for inter-cluster communcations
@@ -438,6 +441,17 @@ be obtained by running `garage node id` and then included directly in the
key will be returned by `garage node id` and you will have to add the IP
yourself.
+### `allow_world_readable_secrets`
+
+Garage checks the permissions of your secret files to make sure they're not
+world-readable. In some cases, the check might fail and consider your files as
+world-readable even if they're not, for instance when using Posix ACLs.
+
+Setting `allow_world_readable_secrets` to `true` bypass this
+permission verification.
+
+Alternatively, you can set the `GARAGE_ALLOW_WORLD_READABLE_SECRETS`
+environment variable to `true` to bypass the permissions check.
### The `[consul_discovery]` section
@@ -583,7 +597,7 @@ See [administration API reference](@/documentation/reference-manual/admin-api.md
Alternatively, since `v0.8.5`, a path can be used to create a unix socket. Note that for security reasons,
the socket will have 0220 mode. Make sure to set user and group permissions accordingly.
-#### `metrics_token`, `metrics_token_file` or `GARAGE_METRICS_TOKEN` (env) {#admin_metrics_token}
+#### `metrics_token`, `metrics_token_file` or `GARAGE_METRICS_TOKEN`, `GARAGE_METRICS_TOKEN_FILE` (env) {#admin_metrics_token}
The token for accessing the Metrics endpoint. If this token is not set, the
Metrics endpoint can be accessed without access control.
@@ -593,8 +607,9 @@ You can use any random string for this value. We recommend generating a random t
`metrics_token` was introduced in Garage `v0.7.2`.
`metrics_token_file` and the `GARAGE_METRICS_TOKEN` environment variable are supported since Garage `v0.8.2`.
+`GARAGE_METRICS_TOKEN_FILE` is supported since `v0.8.5` / `v0.9.1`.
-#### `admin_token`, `admin_token_file` or `GARAGE_ADMIN_TOKEN` (env) {#admin_token}
+#### `admin_token`, `admin_token_file` or `GARAGE_ADMIN_TOKEN`, `GARAGE_ADMIN_TOKEN_FILE` (env) {#admin_token}
The token for accessing all of the other administration endpoints. If this
token is not set, access to these endpoints is disabled entirely.
@@ -604,6 +619,7 @@ You can use any random string for this value. We recommend generating a random t
`admin_token` was introduced in Garage `v0.7.2`.
`admin_token_file` and the `GARAGE_ADMIN_TOKEN` environment variable are supported since Garage `v0.8.2`.
+`GARAGE_ADMIN_TOKEN_FILE` is supported since `v0.8.5` / `v0.9.1`.
#### `trace_sink` {#admin_trace_sink}