aboutsummaryrefslogtreecommitdiff
path: root/doc/book
diff options
context:
space:
mode:
authorAlex <alex@adnab.me>2024-02-09 12:11:23 +0000
committerAlex <alex@adnab.me>2024-02-09 12:11:23 +0000
commit0c7ce001c907791518f93fd9d2db01eebc517c51 (patch)
tree08336bb7610c35b7707d3c87fd3f4eac7e2b21f7 /doc/book
parente4a43bfd592c149af8e3eac58ab317a518f0968a (diff)
parentf7ae966ed34d3ca8fb20afd2f20874dd42df7f7e (diff)
downloadgarage-0c7ce001c907791518f93fd9d2db01eebc517c51.tar.gz
garage-0c7ce001c907791518f93fd9d2db01eebc517c51.zip
Merge pull request 'Fix & simplify CI using Woodpecker' (#706) from nix-improvements into main
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/706
Diffstat (limited to 'doc/book')
-rw-r--r--doc/book/design/goals.md2
-rw-r--r--doc/book/development/devenv.md2
-rw-r--r--doc/book/development/miscellaneous-notes.md8
-rw-r--r--doc/book/development/release-process.md55
4 files changed, 21 insertions, 46 deletions
diff --git a/doc/book/design/goals.md b/doc/book/design/goals.md
index 78ac7978..4efb6349 100644
--- a/doc/book/design/goals.md
+++ b/doc/book/design/goals.md
@@ -48,7 +48,5 @@ locations. They use Garage themselves for the following tasks:
- As a backup target using `rclone` and `restic`
-- In the Drone continuous integration platform to store task logs
-
The Deuxfleurs Garage cluster is a multi-site cluster currently composed of
9 nodes in 3 physical locations.
diff --git a/doc/book/development/devenv.md b/doc/book/development/devenv.md
index dd3bdec0..88f8ba06 100644
--- a/doc/book/development/devenv.md
+++ b/doc/book/development/devenv.md
@@ -80,7 +80,7 @@ nix-build \
--git_version $(git rev-parse HEAD)
```
-*The result is located in `result/bin`. You can pass arguments to cross compile: check `.drone.yml` for examples.*
+*The result is located in `result/bin`. You can pass arguments to cross compile: check `.woodpecker/release.yml` for examples.*
If you modify a `Cargo.toml` or regenerate any `Cargo.lock`, you must run `cargo2nix`:
diff --git a/doc/book/development/miscellaneous-notes.md b/doc/book/development/miscellaneous-notes.md
index f0083ae5..a421943f 100644
--- a/doc/book/development/miscellaneous-notes.md
+++ b/doc/book/development/miscellaneous-notes.md
@@ -81,12 +81,9 @@ Our cache will be checked.
- http://www.lpenz.org/articles/nixchannel/index.html
-## Drone
+## Woodpecker
-Do not try to set a build as trusted from the interface or the CLI tool,
-your request would be ignored. Instead, directly edit the database (table `repos`, column `repo_trusted`).
-
-Drone can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner.
+Woodpecker can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner.
## Building Docker containers
@@ -99,3 +96,4 @@ We were:
- Unable to use the kaniko container provided by Google as we can't run arbitrary logic: we need to put our secret in .docker/config.json.
Finally we chose to build kaniko through nix and use it in a `nix-shell`.
+We then switched to using kaniko from nixpkgs when it was packaged.
diff --git a/doc/book/development/release-process.md b/doc/book/development/release-process.md
index 3fed4add..0c6701c0 100644
--- a/doc/book/development/release-process.md
+++ b/doc/book/development/release-process.md
@@ -42,7 +42,7 @@ and the docker containers on Docker Hub.
## Automation
-We automated our release process with Nix and Drone to make it more reliable.
+We automated our release process with Nix and Woodpecker to make it more reliable.
Here we describe how we have done in case you want to debug or improve it.
### Caching build steps
@@ -62,52 +62,31 @@ Sending to the cache is done through `nix copy`, for example:
nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' result
```
-*Note that you need the signing key. In our case, it is stored as a secret in Drone.*
+*The signing key possessed by the Garage maintainers is required to update the Nix cache.*
-The previous command will only send the built packet and not its dependencies.
-To send its dependency, a tool named `nix-copy-closure` has been created but it is not compatible with the S3 protocol.
-
-Instead, you can use the following commands to list all the runtime dependencies:
+The previous command will only send the built package and not its dependencies.
+In the case of our CI pipeline, we want to cache all intermediate build steps
+as well. This can be done using this quite involved command (here as an example
+for the `pkgs.amd64.relase` package):
```bash
-nix copy \
- --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
- $(nix-store -qR result/)
+nix copy -j8 \
+ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/nix-signing-key.sec' \
+ $(nix path-info pkgs.amd64.release --file default.nix --derivation --recursive | sed 's/\.drv$/.drv^*/')
```
-*We could also write this expression with xargs but this tool is not available in our container.*
-
-But in certain cases, we want to cache compile time dependencies also.
-For example, the Nix project does not provide binaries for cross compiling to i686 and thus we need to compile gcc on our own.
-We do not want to compile gcc each time, so even if it is a compile time dependency, we want to cache it.
+This command will simultaneously build all of the required Nix paths (using at
+most 8 parallel Nix builder jobs) and send the resulting objects to the cache.
-This time, the command is a bit more involved:
-
-```bash
-nix copy --to \
- 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
- $(nix-store -qR --include-outputs \
- $(nix-instantiate))
-```
-
-This is the command we use in our CI as we expect the final binary to change, so we mainly focus on
-caching our development dependencies.
-
-*Currently there is no automatic garbage collection of the cache: we should monitor its growth.
-Hopefully, we can erase it totally without breaking any build, the next build will only be slower.*
-
-In practise, we concluded that we do not want to cache all the compilation dependencies.
-Instead, we want to cache the toolchain we use to build Garage each time we change it.
-So we removed from Drone any automatic update of the cache and instead handle them manually with:
+This can be run for all the Garage packages we build using the following command:
```
source ~/.awsrc
-nix-shell --run 'refresh_toolchain'
+nix-shell --attr cache --run 'refresh_cache'
```
-Internally, it will run `nix-build` on `nix/toolchain.nix` and send the output plus its depedencies to the cache.
-
-To erase the cache:
+We don't automate this step at each CI build, as *there is currently no automatic garbage collection of the cache.*
+This means we should also monitor the cache's size; if it ever becomes too big we can erase it with:
```
mc rm --recursive --force 'garage/nix/'
@@ -157,9 +136,9 @@ nix-shell --run refresh_index
If you want to compile for different architectures, you will need to repeat all these commands for each architecture.
-**In practise, and except for debugging, you will never directly run these commands. Release is handled by drone**
+**In practice, and except for debugging, you will never directly run these commands. Release is handled by Woodpecker.**
-### Drone
+### Drone (obsolete)
Our instance is available at [https://drone.deuxfleurs.fr](https://drone.deuxfleurs.fr).
You need an account on [https://git.deuxfleurs.fr](https://git.deuxfleurs.fr) to use it.