diff options
Diffstat (limited to 'doc/book/development')
| -rw-r--r-- | doc/book/development/_index.md | 19 | ||||
| -rw-r--r-- | doc/book/development/devenv.md | 148 | ||||
| -rw-r--r-- | doc/book/development/miscellaneous_notes.md | 101 | ||||
| -rw-r--r-- | doc/book/development/release_process.md | 198 | ||||
| -rw-r--r-- | doc/book/development/scripts.md | 116 |
5 files changed, 582 insertions, 0 deletions
diff --git a/doc/book/development/_index.md b/doc/book/development/_index.md new file mode 100644 index 00000000..662ec358 --- /dev/null +++ b/doc/book/development/_index.md @@ -0,0 +1,19 @@ ++++ +title = "Development" +weight = 6 +sort_by = "weight" +template = "documentation.html" ++++ + +Now that you are a Garage expert, you want to enhance it, you are in the right place! +We discuss here how to hack on Garage, how we manage its development, etc. + +## Rust API (docs.rs) +If you encounter a specific bug in Garage or plan to patch it, you may jump directly to the source code's documentation! + + - [garage\_api](https://docs.rs/garage_api/latest/garage_api/) - contains the S3 standard API endpoint + - [garage\_model](https://docs.rs/garage_model/latest/garage_model/) - contains Garage's model built on the table abstraction + - [garage\_rpc](https://docs.rs/garage_rpc/latest/garage_rpc/) - contains Garage's federation protocol + - [garage\_table](https://docs.rs/garage_table/latest/garage_table/) - contains core Garage's CRDT datatypes + - [garage\_util](https://docs.rs/garage_util/latest/garage_util/) - contains garage helpers + - [garage\_web](https://docs.rs/garage_web/latest/garage_web/) - contains the S3 website endpoint diff --git a/doc/book/development/devenv.md b/doc/book/development/devenv.md new file mode 100644 index 00000000..c2ef4e7d --- /dev/null +++ b/doc/book/development/devenv.md @@ -0,0 +1,148 @@ ++++ +title = "Setup your environment" +weight = 5 ++++ + +Depending on your tastes, you can bootstrap your development environment in a traditional Rust way or through Nix. + +## The Nix way + +Nix is a generic package manager we use to precisely define our development environment. +Instructions on how to install it are given on their [Download page](https://nixos.org/download.html). + +Check that your installation is working by running the following commands: + +``` +nix-shell --version +nix-build --version +nix-env --version +``` + +Now, you can clone our git repository (run `nix-env -iA git` if you do not have git yet): + +```bash +git clone https://git.deuxfleurs.fr/Deuxfleurs/garage +cd garage +``` + +*Optionnaly, you can use our nix.conf file to speed up compilations:* + +```bash +sudo mkdir -p /etc/nix +sudo cp nix/nix.conf /etc/nix/nix.conf +sudo killall nix-daemon +``` + +Now you can enter our nix-shell, all the required packages will be downloaded but they will not pollute your environment outside of the shell: + +```bash +nix-shell +``` + +You can use the traditionnal Rust development workflow: + +```bash +cargo build # compile the project +cargo run # execute the project +cargo test # run the tests +cargo fmt # format the project, run it before any commit! +cargo clippy # run the linter, run it before any commit! +``` + +You can build the project with Nix by running: + +```bash +nix-build +``` + +You can parallelize the build (if you use our nix.conf file, it is already automatically done). +To use all your cores when building a derivation use `-j`, and to build multiple derivations at once use `--max-jobs`. +The special value `auto` will be replaced by the number of cores of your computer. +An example: + +```bash +nix-build -j $(nproc) --max-jobs auto +``` + +Our build has multiple parameters you might want to set: + - `release` build with release optimisations instead of debug + - `target allows` for cross compilation + - `compileMode` can be set to test or bench to build a unit test runner + - `git_version` to inject the hash to display when running `garage stats` + +An example: + +```bash +nix-build \ + --arg release true \ + --argstr target x86_64-unknown-linux-musl \ + --argstr compileMode build \ + --git_version $(git rev-parse HEAD) +``` + +*The result is located in `result/bin`. You can pass arguments to cross compile: check `.drone.yml` for examples.* + +If you modify a `Cargo.toml` or regenerate any `Cargo.lock`, you must run `cargo2nix`: + +``` +cargo2nix -f +``` + +Many tools like rclone, `mc` (minio-client), or `aws` (awscliv2) will be available in your environment and will be useful to test Garage. + +**This is the recommended method.** + +## The Rust way + +You need a Rust distribution installed on your computer. +The most simple way is to install it from [rustup](https://rustup.rs). +Please avoid using your package manager to install Rust as some tools might be outdated or missing. + +Now, check your Rust distribution works by running the following commands: + +```bash +rustc --version +cargo --version +rustfmt --version +clippy-driver --version +``` + +Now, you need to clone our git repository ([how to install git](https://git-scm.com/downloads)): + +```bash +git clone https://git.deuxfleurs.fr/Deuxfleurs/garage +cd garage +``` + +You can now use the following commands: + +```bash +cargo build # compile the project +cargo run # execute the project +cargo test # run the tests +cargo fmt # format the project, run it before any commit! +cargo clippy # run the linter, run it before any commit! +``` + +This is specific to our project, but you will need one last tool, `cargo2nix`. +To install it, run: + +```bash +cargo install --git https://github.com/superboum/cargo2nix --branch main cargo2nix +``` + +You must use it every time you modify a `Cargo.toml` or regenerate a `Cargo.lock` file as follow: + +```bash +cargo build # Rebuild Cargo.lock if needed +cargo2nix -f +``` + +It will output a `Cargo.nix` file which is a specific `Cargo.lock` file dedicated to Nix that is required by our CI +which means you must include it in your commits. + +Later, to use our scripts and integration tests, you might need additional tools. +These tools are listed at the end of the `shell.nix` package in the `nativeBuildInputs` part. +It is up to you to find a way to install the ones you need on your computer. + +**A global drawback of this method is that it is up to you to adapt your environment to the one defined in the Nix files.** diff --git a/doc/book/development/miscellaneous_notes.md b/doc/book/development/miscellaneous_notes.md new file mode 100644 index 00000000..f0083ae5 --- /dev/null +++ b/doc/book/development/miscellaneous_notes.md @@ -0,0 +1,101 @@ ++++ +title = "Miscellaneous notes" +weight = 20 ++++ + +## Quirks about cargo2nix/rust in Nix + +If you use submodules in your crate (like `crdt` and `replication` in `garage_table`), you must list them in `default.nix` + +The Windows target does not work. it might be solvable through [overrides](https://github.com/cargo2nix/cargo2nix/blob/master/overlay/overrides.nix). Indeed, we pass `x86_64-pc-windows-gnu` but mingw need `x86_64-w64-mingw32` + +We have a simple [PR on cargo2nix](https://github.com/cargo2nix/cargo2nix/pull/201) that fixes critical bugs but the project does not seem very active currently. We must use [my patched version of cargo2nix](https://github.com/superboum/cargo2nix) to enable i686 and armv6l compilation. We might need to contribute to cargo2nix in the future. + + +## Nix + +Nix has no armv7 + musl toolchains but armv7l is backward compatible with armv6l. + +```bash +cat > $HOME/.awsrc <<EOF +export AWS_ACCESS_KEY_ID="xxx" +export AWS_SECRET_ACCESS_KEY="xxx" +EOF + +# source each time you want to send on the cache +source ~/.awsrc + +# copy garage build dependencies (and not only the output) +nix-build +nix-store -qR --include-outputs $(nix-instantiate default.nix) + | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr®ion=garage' + +# copy shell dependencies +nix-build shell.nix -A inputDerivation +nix copy $(nix-store -qR result/) --to 's3://nix?endpoint=garage.deuxfleurs.fr®ion=garage' +``` + +More example of nix-copy + +``` +# nix-build produces a result/ symlink +nix copy result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr®ion=garage' + +# alternative ways to use nix copy +nix copy nixpkgs.garage --to ... +nix copy /nix/store/3rbb9qsc2w6xl5xccz5ncfhy33nzv3dp-crate-garage-0.3.0 --to ... +``` + + +Clear the cache: + +```bash +mc rm --recursive --force garage/nix/ +``` + +--- + +A desirable `nix.conf` for a consumer: + +```toml +substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr +trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs= +``` + +And now, whenever you run a command like: + +``` +nix-shell +nix-build +``` + +Our cache will be checked. + +### Some references about Nix + + + - https://doc.rust-lang.org/nightly/rustc/platform-support.html + - https://nix.dev/tutorials/cross-compilation + - https://nixos.org/manual/nix/unstable/package-management/s3-substituter.html + - https://fzakaria.com/2020/09/28/nix-copy-closure-your-nix-shell.html + - http://www.lpenz.org/articles/nixchannel/index.html + + +## Drone + +Do not try to set a build as trusted from the interface or the CLI tool, +your request would be ignored. Instead, directly edit the database (table `repos`, column `repo_trusted`). + +Drone can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner. + +## Building Docker containers + +We were: + - Unable to use the official Docker plugin because + - it requires to mount docker socket in the container but it is not recommended + - you cant set the platform when building + - Unable to use buildah because it needs `CLONE_USERNS` capability + - Unable to use the kaniko plugin for Drone as we can't set the target platform + - Unable to use the kaniko container provided by Google as we can't run arbitrary logic: we need to put our secret in .docker/config.json. + +Finally we chose to build kaniko through nix and use it in a `nix-shell`. diff --git a/doc/book/development/release_process.md b/doc/book/development/release_process.md new file mode 100644 index 00000000..e8d29ad8 --- /dev/null +++ b/doc/book/development/release_process.md @@ -0,0 +1,198 @@ ++++ +title = "Release process" +weight = 15 ++++ + +Before releasing a new version of Garage, our code pass through a succession of checks and transformations. +We define them as our release process. + +## Trigger and classify a release + +While we run some tests on every commits, we do not make a release for all of them. + +A release can be triggered manually by "promoting" a successful build. +Otherwise, every weeks, a release build is triggered on the `main` branch. + +If the build is from a tag following the regex: `v[0-9]+\.[0-9]+\.[0-9]+`, it will be listed as stable. +If it is a tag but with a different format, it will be listed as Extra. +Otherwise, if it is a commit, it will be listed as development. +This logic is defined in `nix/build_index.nix`. + +## Testing + +For each commit, we first pass the code to a formatter (rustfmt) and a linter (clippy). +Then we try to build it in debug mode and run both unit tests and our integration tests. + +Additionnaly, when releasing, our integration tests are run on the release build for amd64 and i686. + +## Generated Artifacts + +We generate the following binary artifacts for now: + - **architecture**: amd64, i686, aarch64, armv6 + - **os**: linux + - **format**: static binary, docker container + +Additionnaly we also build two web pages: + - the documentation (this website) + - [the release page](https://garagehq.deuxfleurs.fr/releases.html) + +We publish the static binaries on our own garage cluster (you can access them through the releases page) +and the docker containers on Docker Hub. + +## Automation + +We automated our release process with Nix and Drone to make it more reliable. +Here we describe how we have done in case you want to debug or improve it. + +### Caching build steps + +To speed up the CI, we use the caching feature provided by Nix. + +You can benefit from it by using our provided `nix.conf` as recommended or by simply adding the following lines to your file: + +```toml +substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr +trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs= +``` + +Sending to the cache is done through `nix copy`, for example: + +```bash +nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr®ion=garage&secret-key=/etc/nix/signing-key.sec' result +``` + +*Note that you need the signing key. In our case, it is stored as a secret in Drone.* + +The previous command will only send the built packet and not its dependencies. +To send its dependency, a tool named `nix-copy-closure` has been created but it is not compatible with the S3 protocol. + +Instead, you can use the following commands to list all the runtime dependencies: + +```bash +nix copy \ + --to 's3://nix?endpoint=garage.deuxfleurs.fr®ion=garage&secret-key=/etc/nix/signing-key.sec' \ + $(nix-store -qR result/) +``` + +*We could also write this expression with xargs but this tool is not available in our container.* + +But in certain cases, we want to cache compile time dependencies also. +For example, the Nix project does not provide binaries for cross compiling to i686 and thus we need to compile gcc on our own. +We do not want to compile gcc each time, so even if it is a compile time dependency, we want to cache it. + +This time, the command is a bit more involved: + +```bash +nix copy --to \ + 's3://nix?endpoint=garage.deuxfleurs.fr®ion=garage&secret-key=/etc/nix/signing-key.sec' \ + $(nix-store -qR --include-outputs \ + $(nix-instantiate)) +``` + +This is the command we use in our CI as we expect the final binary to change, so we mainly focus on +caching our development dependencies. + +*Currently there is no automatic garbage collection of the cache: we should monitor its growth. +Hopefully, we can erase it totally without breaking any build, the next build will only be slower.* + +In practise, we concluded that we do not want to cache all the compilation dependencies. +Instead, we want to cache the toolchain we use to build Garage each time we change it. +So we removed from Drone any automatic update of the cache and instead handle them manually with: + +``` +source ~/.awsrc +nix-shell --run 'refresh_toolchain' +``` + +Internally, it will run `nix-build` on `nix/toolchain.nix` and send the output plus its depedencies to the cache. + +To erase the cache: + +``` +mc rm --recursive --force 'garage/nix/' +``` + +### Publishing Garage + +We defined our publishing logic in Nix, mostly as shell hooks. +You can inspect them in `shell.nix` to see exactly how. +Here, we will give a quick explanation on how to use them to manually publish a release. + +Supposing you just have built garage as follow: + +```bash +nix-build --arg release true +``` + +To publish a static binary in `result/bin` on garagehq, run: + +```bash +export AWS_ACCESS_KEY_ID=xxx +export AWS_SECRET_ACCESS_KEY=xxx +export DRONE_TAG=handcrafted-1.0.0 # or DRONE_COMMIT +export TARGET=x86_64-unknown-linux-musl + +nix-shell --run to_s3 +``` + +To create and publish a docker container, run: + +```bash +export DOCKER_AUTH='{ "auths": { "https://index.docker.io/v1/": { "auth": "xxxx" }}}' +export DOCKER_PLATFORM='linux/amd64' # check GOARCH and GOOS from golang.org +export CONTAINER_NAME='me/amd64_garage' +export CONTAINER_TAG='handcrafted-1.0.0' + +nix-shell --run to_docker +``` + +To rebuild the release page, run: +```bash +export AWS_ACCESS_KEY_ID=xxx +export AWS_SECRET_ACCESS_KEY=xxx + +nix-shell --run refresh_index +``` + +If you want to compile for different architectures, you will need to repeat all these commands for each architecture. + +**In practise, and except for debugging, you will never directly run these commands. Release is handled by drone** + +### Drone + +Our instance is available at [https://drone.deuxfleurs.fr](https://drone.deuxfleurs.fr). +You need an account on [https://git.deuxfleurs.fr](https://git.deuxfleurs.fr) to use it. + +**Drone CLI** - Drone has a CLI tool to interact with. +It can be downloaded from its Github [release page](https://github.com/drone/drone-cli/releases). + +To communicate with our instance, you must setup some environment variables. +You can get them from your [Account Settings](https://drone.deuxfleurs.fr/account). + +To make drone easier to use, you could create a `~/.dronerc` that you could source each time you want to use it. + +``` +export DRONE_SERVER=https://drone.deuxfleurs.fr +export DRONE_TOKEN=xxx +drone info +``` + +The CLI tool is very self-discoverable, just append `--help` to each subcommands. +Start with: + +```bash +drone --help +``` + +**.drone.yml** - The builds steps are defined in `.drone.yml`. +You can not edit this file without resigning it. + +To sign it, you must be a maintainer and then run: + +```bash +drone sign --save Deuxfleurs/garage +``` + +Looking at the file, you will see that most of the commands are `nix-shell` and `nix-build` commands with various parameters. + + diff --git a/doc/book/development/scripts.md b/doc/book/development/scripts.md new file mode 100644 index 00000000..25f2ce0e --- /dev/null +++ b/doc/book/development/scripts.md @@ -0,0 +1,116 @@ ++++ +title = "Development scripts" +weight = 10 ++++ + +We maintain a `script/` folder that contains some useful script to ease testing on Garage. + +A fully integrated script, `test-smoke.sh`, runs some basic tests on various tools such as minio client, awscli and rclone. +To run it, enter a `nix-shell` (or install all required tools) and simply run: + +```bash +nix-build # or cargo build +./script/test-smoke.sh +``` + +If something fails, you can find useful logs in `/tmp/garage.log`. +You can inspect the generated configuration and local data created by inspecting your `/tmp` directory: +the script creates files and folder prefixed with the name "garage". + +## Bootstrapping a test cluster + +Under the hood `test-smoke.sh` uses multiple helpers scripts you can also run in case you want to manually test Garage. +In this section, we introduce 3 scripts to quickly bootstrap a full test cluster with 3 instances. + +### 1. Start each daemon + +```bash +./script/dev-cluster.sh +``` + +This script spawns 3 Garage instances with 3 configuration files. +You can inspect the detailed configuration, including ports, by inspecting `/tmp/config.1` (change 1 by the instance number you want). + +This script also spawns a simple HTTPS reverse proxy through `socat` for the S3 endpoint that listens on port `4443`. +Some libraries might require a TLS endpoint to work, refer to our issue [#64](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/64) for more detailed information on this subject. + +This script covers the [Launching the garage server](/quick_start/index.html#launching-the-garage-server) section of our Quick start page. + +### 2. Make them join the cluster + +```bash +./script/dev-configure.sh +``` + +This script will configure each instance by assigning them a zone (`dc1`) and a weight (`1`). + +This script covers the [Configuring your Garage node](/quick_start/index.html#configuring-your-garage-node) section of our Quick start page. + +### 3. Create a key and a bucket + +```bash +./script/dev-bucket.sh +``` + +This script will create a bucket named `eprouvette` with a key having read and write rights on this bucket. +The key is stored in a filed named `/tmp/garage.s3` and can be used by the following tools to pre-configure them. + +This script covers the [Creating buckets and keys](/quick_start/index.html#creating-buckets-and-keys) section of our Quick start page. + +## Handlers for generic tools + +We provide wrappers for some CLI tools that configure themselves for your development cluster. +They are meant to save you some configuration time as to use them, you are only required to source the right file. + +### awscli + +```bash +source ./script/dev-env-aws.sh + +# some examples +aws s3 ls s3://eprouvette +aws s3 cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt +``` + +### minio-client + + +```bash +source ./script/dev-env-mc.sh + +# some examples +mc ls garage/ +mc cp /proc/cpuinfo garage/eprouvette/cpuinfo.txt +``` + +### rclone + +```bash +source ./script/dev-env-rclone.sh + +# some examples +rclone lsd garage: +rclone copy /proc/cpuinfo garage:eprouvette/cpuinfo.txt +``` + +### s3cmd + +```bash +source ./script/dev-env-s3cmd.sh + +# some examples +s3cmd ls +s3cmd put /proc/cpuinfo s3://eprouvette/cpuinfo.txt +``` + +### duck + +*Warning! Duck is not yet provided by nix-shell.* + +```bash +source ./script/dev-env-duck.sh + +# some examples +duck --list garage:/ +duck --upload garage:/eprouvette/ /proc/cpuinfo +``` |