Build Garage with Nixfeature/static

4 files changed, 570 insertions, 11 deletions

diff --git a/doc/book/src/development/devenv.md b/doc/book/src/development/devenv.md
index 6cb7c554..78affa48 100644
--- a/doc/book/src/development/devenv.md
+++ b/doc/book/src/development/devenv.md
@@ -1,17 +1,145 @@
 # Setup your development environment
 
-We propose the following quickstart to setup a full dev. environment as quickly as possible:
+Depending on your tastes, you can bootstrap your development environment in a traditional Rust way or through Nix.
 
-  1. Setup a rust/cargo environment. eg. `dnf install rust cargo`
-  2. Install awscli v2 by following the guide [here](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
-  3. Run `cargo build` to build the project
-  4. Run `./script/dev-cluster.sh` to launch a test cluster (feel free to read the script)
-  5. Run `./script/dev-configure.sh` to configure your test cluster with default values (same datacenter, 100 tokens)
-  6. Run `./script/dev-bucket.sh` to create a bucket named `eprouvette` and an API key that will be stored in `/tmp/garage.s3`
-  7. Run `source ./script/dev-env-aws.sh` to configure your CLI environment
-  8. You can use `garage` to manage the cluster. Try `garage --help`.
-  9. You can use the `awsgrg` alias to add, remove, and delete files. Try `awsgrg help`, `awsgrg cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt`, or `awsgrg ls s3://eprouvette`. `awsgrg` is a wrapper on the `aws s3` command pre-configured with the previously generated API key (the one in `/tmp/garage.s3`) and localhost as the endpoint.
+## The Nix way
 
-Now you should be ready to start hacking on garage!
+Nix is a generic package manager we use to precisely define our development environment.
+Instructions on how to install it are given on their [Download page](https://nixos.org/download.html).
 
+Check that your installation is working by running the following commands:
 
+```
+nix-shell --version
+nix-build --version
+nix-env   --version
+```
+
+Now, you can clone our git repository (run `nix-env -iA git` if you do not have git yet):
+
+```bash
+git clone https://git.deuxfleurs.fr/Deuxfleurs/garage
+cd garage
+```
+
+*Optionnaly, you can use our nix.conf file to speed up compilations:*
+
+```bash
+sudo mkdir -p /etc/nix
+sudo cp nix/nix.conf /etc/nix/nix.conf
+sudo killall nix-daemon
+```
+
+Now you can enter our nix-shell, all the required packages will be downloaded but they will not pollute your environment outside of the shell:
+
+```bash
+nix-shell
+```
+
+You can use the traditionnal Rust development workflow:
+
+```bash
+cargo build  # compile the project
+cargo run    # execute the project
+cargo test   # run the tests
+cargo fmt    # format the project, run it before any commit!
+cargo clippy # run the linter, run it before any commit!
+```
+
+You can build the project with Nix by running:
+
+```bash
+nix-build
+```
+
+You can parallelize the build (if you use our nix.conf file, it is already automatically done).
+To use all your cores when building a derivation use `-j`, and to build multiple derivations at once use `--max-jobs`.
+The special value `auto` will be replaced by the number of cores of your computer.
+An example:
+
+```bash
+nix-build -j $(nproc) --max-jobs auto
+```
+
+Our build has multiple parameters you might want to set:
+  - `release` build with release optimisations instead of debug
+  - `target allows` for cross compilation
+  - `compileMode` can be set to test or bench to build a unit test runner
+  - `git_version` to inject the hash to display when running `garage stats`
+
+An example:
+
+```bash
+nix-build \
+  --arg release true \
+  --argstr target x86_64-unknown-linux-musl \
+  --argstr compileMode build \
+  --git_version $(git rev-parse HEAD)
+```
+
+*The result is located in `result/bin`. You can pass arguments to cross compile: check `.drone.yml` for examples.*
+
+If you modify a `Cargo.toml` or regenerate any `Cargo.lock`, you must run `cargo2nix`:
+
+```
+cargo2nix -f
+```
+
+Many tools like rclone, `mc` (minio-client), or `aws` (awscliv2) will be available in your environment and will be useful to test Garage.
+
+**This is the recommended method.**
+
+## The Rust way
+
+You need a Rust distribution installed on your computer.
+The most simple way is to install it from [rustup](https://rustup.rs).
+Please avoid using your package manager to install Rust as some tools might be outdated or missing.
+
+Now, check your Rust distribution works by running the following commands:
+
+```bash
+rustc --version
+cargo --version
+rustfmt --version
+clippy-driver --version
+```
+
+Now, you need to clone our git repository ([how to install git](https://git-scm.com/downloads)):
+
+```bash
+git clone https://git.deuxfleurs.fr/Deuxfleurs/garage
+cd garage
+```
+
+You can now use the following commands:
+
+```bash
+cargo build  # compile the project
+cargo run    # execute the project
+cargo test   # run the tests
+cargo fmt    # format the project, run it before any commit!
+cargo clippy # run the linter, run it before any commit!
+```
+
+This is specific to our project, but you will need one last tool, `cargo2nix`.
+To install it, run:
+
+```bash
+cargo install --git https://github.com/superboum/cargo2nix --branch main cargo2nix
+```
+
+You must use it every time you modify a `Cargo.toml` or regenerate a `Cargo.lock` file as follow:
+
+```bash
+cargo build  # Rebuild Cargo.lock if needed
+cargo2nix -f
+```
+
+It will output a `Cargo.nix` file which is a specific `Cargo.lock` file dedicated to Nix that is required by our CI
+which means you must include it in your commits.
+
+Later, to use our scripts and integration tests, you might need additional tools.
+These tools are listed at the end of the `shell.nix` package in the `nativeBuildInputs` part.
+It is up to you to find a way to install the ones you need on your computer.
+
+**A global drawback of this method is that it is up to you to adapt your environment to the one defined in the Nix files.**
diff --git a/doc/book/src/development/miscellaneous_notes.md b/doc/book/src/development/miscellaneous_notes.md
new file mode 100644
index 00000000..c259cba1
--- /dev/null
+++ b/doc/book/src/development/miscellaneous_notes.md
@@ -0,0 +1,134 @@
+# Miscellaneous Notes
+
+## Quirks about cargo2nix/rust in Nix
+
+If you use submodules in your crate (like `crdt` and `replication` in `garage_table`), you must list them in `default.nix`
+
+The Windows target does not work. it might be solvable through [overrides](https://github.com/cargo2nix/cargo2nix/blob/master/overlay/overrides.nix). Indeed, we pass `x86_64-pc-windows-gnu` but mingw need `x86_64-w64-mingw32`
+
+We have a simple [PR on cargo2nix](https://github.com/cargo2nix/cargo2nix/pull/201) that fixes critical bugs but the project does not seem very active currently. We must use [my patched version of cargo2nix](https://github.com/superboum/cargo2nix) to enable i686 and armv6l compilation. We might need to contribute to cargo2nix in the future.
+
+
+## Nix
+
+Nix has no armv7 + musl toolchains but armv7l is backward compatible with armv6l.
+
+Signing keys are generated with: 
+
+```
+nix-store --generate-binary-cache-key nix.web.deuxfleurs.fr cache-priv-key.pem cache-pub-key.pem
+```
+
+We copy the secret key in our nix folder:
+
+```
+cp cache-priv-key.pem /etc/nix/signing-key.sec
+```
+
+Manually sign
+    
+We can sign the whole store with:
+
+```
+nix sign-paths --all -k /etc/nix/signing-key.sec
+```
+
+Or simply the current package and its dependencies with:
+
+```
+nix sign-paths --recursive -k /etc/nix/signing-key.sec
+```
+
+Setting a key in `nix.conf` will do the signature at build time automatically without additional commands, edit the `nix.conf` of your builder:
+
+```toml
+secret-key-files = /etc/nix/signing-key.sec
+max-jobs = auto
+cores = 8
+```
+
+Now you are ready to build your packages: 
+
+```bash
+cat > $HOME/.awsrc <<EOF
+export AWS_ACCESS_KEY_ID="xxx"
+export AWS_SECRET_ACCESS_KEY="xxx"
+EOF
+
+# source each time you want to send on the cache
+source ~/.awsrc
+
+# copy garage build dependencies (and not only the output)
+nix-build
+nix-store -qR --include-outputs $(nix-instantiate default.nix) 
+  | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'
+  
+# copy shell dependencies
+nix-build shell.nix -A inputDerivation
+nix copy $(nix-store -qR result/) --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' 
+```
+
+More example of nix-copy
+
+```
+# nix-build produces a result/ symlink
+nix copy result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'
+
+# alternative ways to use nix copy
+nix copy nixpkgs.garage --to ...
+nix copy /nix/store/3rbb9qsc2w6xl5xccz5ncfhy33nzv3dp-crate-garage-0.3.0 --to ...
+```
+    
+
+Clear the cache:
+
+```bash
+mc rm --recursive --force garage/nix/
+```
+
+---
+
+A desirable `nix.conf` for a consumer:
+
+```toml
+substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr
+trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs=
+```
+
+And now, whenever you run a command like:
+
+```
+nix-shell
+nix-build
+```
+
+Our cache will be checked.
+
+### Some references about Nix
+
+
+ - https://doc.rust-lang.org/nightly/rustc/platform-support.html
+ - https://nix.dev/tutorials/cross-compilation
+ - https://nixos.org/manual/nix/unstable/package-management/s3-substituter.html
+ - https://fzakaria.com/2020/09/28/nix-copy-closure-your-nix-shell.html
+ - http://www.lpenz.org/articles/nixchannel/index.html
+
+
+## Drone
+
+Do not try to set a build as trusted from the interface or the CLI tool,
+your request would be ignored. Instead, directly edit the database (table `repos`, column `repo_trusted`).
+
+Drone can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner.
+
+## Building Docker containers
+
+We were:
+  - Unable to use the official Docker plugin because
+    - it requires to mount docker socket in the container but it is not recommended
+    - you cant set the platform when building
+  - Unable to use buildah because it needs `CLONE_USERNS` capability
+  - Unable to use the kaniko plugin for Drone as we can't set the target platform
+  - Unable to use the kaniko container provided by Google as we can't run arbitrary logic: we need to put our secret in .docker/config.json.
+
+Finally we chose to build kaniko through nix and use it in a `nix-shell`.
diff --git a/doc/book/src/development/release_process.md b/doc/book/src/development/release_process.md
new file mode 100644
index 00000000..8591fd9f
--- /dev/null
+++ b/doc/book/src/development/release_process.md
@@ -0,0 +1,184 @@
+# Release process
+
+Before releasing a new version of Garage, our code pass through a succession of checks and transformations.
+We define them as our release process.
+
+## Trigger and classify a release
+
+While we run some tests on every commits, we do not make a release for all of them.
+
+A release can be triggered manually by "promoting" a successful build.
+Otherwise, every weeks, a release build is triggered on the `main` branch.
+
+If the build is from a tag following the regex: `v[0-9]+\.[0-9]+\.[0-9]+`, it will be listed as stable.
+If it is a tag but with a different format, it will be listed as Extra.
+Otherwise, if it is a commit, it will be listed as development.
+This logic is defined in `nix/build_index.nix`.
+
+## Testing
+
+For each commit, we first pass the code to a formatter (rustfmt) and a linter (clippy).
+Then we try to build it in debug mode and run both unit tests and our integration tests.
+
+Additionnaly, when releasing, our integration tests are run on the release build for amd64 and i686.
+
+## Generated Artifacts
+
+We generate the following binary artifacts for now:
+  - **architecture**: amd64, i686, aarch64, armv6
+  - **os**: linux
+  - **format**: static binary, docker container
+
+Additionnaly we also build two web pages:
+  - the documentation (this website)
+  - [the release page](https://garagehq.deuxfleurs.fr/releases.html)
+
+We publish the static binaries on our own garage cluster (you can access them through the releases page)
+and the docker containers on Docker Hub.
+
+## Automation
+
+We automated our release process with Nix and Drone to make it more reliable.
+Here we describe how we have done in case you want to debug or improve it.
+
+### Caching build steps
+
+To speed up the CI, we use the caching feature provided by Nix.
+
+You can benefit from it by using our provided `nix.conf` as recommended or by simply adding the following lines to your file:
+
+```toml
+substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr
+trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs=
+```
+
+Sending to the cache is done through `nix copy`, for example:
+
+```bash
+nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' result
+```
+
+*Note that you need the signing key. In our case, it is stored as a secret in Drone.*
+
+The previous command will only send the built packet and not its dependencies.
+To send its dependency, a tool named `nix-copy-closure` has been created but it is not compatible with the S3 protocol.
+
+Instead, you can use the following commands to list all the runtime dependencies:
+
+```bash
+nix copy \
+  --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
+  $(nix-store -qR result/)
+```
+
+*We could also write this expression with xargs but this tool is not available in our container.*
+
+But in certain cases, we want to cache compile time dependencies also.
+For example, the Nix project does not provide binaries for cross compiling to i686 and thus we need to compile gcc on our own.
+We do not want to compile gcc each time, so even if it is a compile time dependency, we want to cache it.
+
+This time, the command is a bit more involved:
+
+```bash
+nix copy --to \
+  's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/etc/nix/signing-key.sec' \
+   $(nix-store -qR --include-outputs \
+     $(nix-instantiate))
+```
+
+This is the command we use in our CI as we expect the final binary to change, so we mainly focus on
+caching our development dependencies.
+
+*Currently there is no automatic garbage collection of the cache: we should monitor its growth.
+Hopefully, we can erase it totally without breaking any build, the next build will only be slower.*
+
+To erase the cache:
+
+```
+mc rm --recursive --force 'garage/nix/*'
+```
+
+### Publishing Garage
+
+We defined our publishing logic in Nix, mostly as shell hooks.
+You can inspect them in `shell.nix` to see exactly how.
+Here, we will give a quick explanation on how to use them to manually publish a release.
+
+Supposing you just have built garage as follow:
+
+```bash
+nix-build --arg release true
+```
+
+To publish a static binary in `result/bin` on garagehq, run:
+
+```bash
+export AWS_ACCESS_KEY_ID=xxx
+export AWS_SECRET_ACCESS_KEY=xxx
+export DRONE_TAG=handcrafted-1.0.0 # or DRONE_COMMIT
+export TARGET=x86_64-unknown-linux-musl
+
+nix-shell --run to_s3
+```
+
+To create and publish a docker container, run:
+
+```bash
+export DOCKER_AUTH='{ "auths": { "https://index.docker.io/v1/": { "auth": "xxxx" }}}'
+export DOCKER_PLATFORM='linux/amd64' # check GOARCH and GOOS from golang.org
+export CONTAINER_NAME='me/amd64_garage'
+export CONTAINER_TAG='handcrafted-1.0.0'
+
+nix-shell --run to_docker
+```
+
+To rebuild the release page, run:
+```bash
+export AWS_ACCESS_KEY_ID=xxx
+export AWS_SECRET_ACCESS_KEY=xxx
+
+nix-shell --run refresh_index
+```
+
+If you want to compile for different architectures, you will need to repeat all these commands for each architecture.
+
+**In practise, and except for debugging, you will never directly run these commands. Release is handled by drone**
+
+### Drone
+
+Our instance is available at [https://drone.deuxfleurs.fr](https://drone.deuxfleurs.fr).  
+You need an account on [https://git.deuxfleurs.fr](https://git.deuxfleurs.fr) to use it.
+
+**Drone CLI** - Drone has a CLI tool to interact with.
+It can be downloaded from its Github [release page](https://github.com/drone/drone-cli/releases).
+
+To communicate with our instance, you must setup some environment variables.
+You can get them from your [Account Settings](https://drone.deuxfleurs.fr/account).
+
+To make drone easier to use, you could create a `~/.dronerc` that you could source each time you want to use it.
+
+```
+export DRONE_SERVER=https://drone.deuxfleurs.fr
+export DRONE_TOKEN=xxx
+drone info
+```
+
+The CLI tool is very self-discoverable, just append `--help` to each subcommands.
+Start with:
+
+```bash
+drone --help
+```
+
+**.drone.yml** - The builds steps are defined in `.drone.yml`.  
+You can not edit this file without resigning it.
+
+To sign it, you must be a maintainer and then run:
+
+```bash
+drone sign --save Deuxfleurs/garage
+```
+
+Looking at the file, you will see that most of the commands are `nix-shell` and `nix-build` commands with various parameters.
+
+
diff --git a/doc/book/src/development/scripts.md b/doc/book/src/development/scripts.md
new file mode 100644
index 00000000..387cb349
--- /dev/null
+++ b/doc/book/src/development/scripts.md
@@ -0,0 +1,113 @@
+# Development scripts
+
+We maintain a `script/` folder that contains some useful script to ease testing on Garage.
+
+A fully integrated script, `test-smoke.sh`, runs some basic tests on various tools such as minio client, awscli and rclone.
+To run it, enter a `nix-shell` (or install all required tools) and simply run:
+
+```bash
+nix-build # or cargo build
+./script/test-smoke.sh
+```
+
+If something fails, you can find useful logs in `/tmp/garage.log`.
+You can inspect the generated configuration and local data created by inspecting your `/tmp` directory:
+the script creates files and folder prefixed with the name "garage".
+
+## Bootstrapping a test cluster
+
+Under the hood `test-smoke.sh` uses multiple helpers scripts you can also run in case you want to manually test Garage.
+In this section, we introduce 3 scripts to quickly bootstrap a full test cluster with 3 instances.
+
+### 1. Start each daemon
+
+```bash
+./script/dev-cluster.sh
+```
+
+This script spawns 3 Garage instances with 3 configuration files.
+You can inspect the detailed configuration, including ports, by inspecting `/tmp/config.1` (change 1 by the instance number you want).
+
+This script also spawns a simple HTTPS reverse proxy through `socat` for the S3 endpoint that listens on port `4443`.
+Some libraries might require a TLS endpoint to work, refer to our issue [#64](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/64) for more detailed information on this subject.
+
+This script covers the [Launching the garage server](/quick_start/index.html#launching-the-garage-server) section of our Quick start page.
+
+### 2. Make them join the cluster
+
+```bash
+./script/dev-configure.sh
+```
+
+This script will configure each instance by assigning them a zone (`dc1`) and a weight (`1`).
+
+This script covers the [Configuring your Garage node](/quick_start/index.html#configuring-your-garage-node) section of our Quick start page.
+
+### 3. Create a key and a bucket
+
+```bash
+./script/dev-bucket.sh
+```
+
+This script will create a bucket named `eprouvette` with a key having read and write rights on this bucket.
+The key is stored in a filed named `/tmp/garage.s3` and can be used by the following tools to pre-configure them.
+
+This script covers the [Creating buckets and keys](/quick_start/index.html#creating-buckets-and-keys) section of our Quick start page.
+
+## Handlers for generic tools
+
+We provide wrappers for some CLI tools that configure themselves for your development cluster.
+They are meant to save you some configuration time as to use them, you are only required to source the right file.
+
+### awscli
+
+```bash
+source ./script/dev-env-aws.sh
+
+# some examples
+aws s3 ls s3://eprouvette
+aws s3 cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt
+```
+
+### minio-client
+
+
+```bash
+source ./script/dev-env-mc.sh
+
+# some examples
+mc ls garage/
+mc cp /proc/cpuinfo garage/eprouvette/cpuinfo.txt
+```
+
+### rclone
+
+```bash
+source ./script/dev-env-rclone.sh
+
+# some examples
+rclone lsd garage:
+rclone copy /proc/cpuinfo garage:eprouvette/cpuinfo.txt
+```
+
+### s3cmd
+
+```bash
+source ./script/dev-env-s3cmd.sh
+
+# some examples
+s3cmd ls
+s3cmd put /proc/cpuinfo s3://eprouvette/cpuinfo.txt
+```
+
+### duck
+
+*Warning! Duck is not yet provided by nix-shell.*
+
+```bash
+source ./script/dev-env-duck.sh
+
+# some examples
+duck --list garage:/
+duck --upload garage:/eprouvette/ /proc/cpuinfo
+```