From a7af0c8af98b8009ef9c7047d9f710c3bcf4e41a Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Wed, 16 Nov 2022 13:27:24 +0100 Subject: Add best practices and doc of monitoring (fix #419) --- doc/book/cookbook/real-world.md | 46 ++++++++++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 7 deletions(-) (limited to 'doc/book/cookbook/real-world.md') diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md index 4fcb5cf7..6fe7e547 100644 --- a/doc/book/cookbook/real-world.md +++ b/doc/book/cookbook/real-world.md @@ -11,8 +11,9 @@ We recommend first following the [quick start guide](@/documentation/quick-start to get familiar with Garage's command line and usage patterns. +## Preparing your environment -## Prerequisites +### Prerequisites To run a real-world deployment, make sure the following conditions are met: @@ -21,10 +22,6 @@ To run a real-world deployment, make sure the following conditions are met: - Each machine has a public IP address which is reachable by other machines. Running behind a NAT is likely to be possible but hasn't been tested for the latest version (TODO). -- Ideally, each machine should have a SSD available in addition to the HDD you are dedicating - to Garage. This will allow for faster access to metadata and has the potential - to significantly reduce Garage's response times. - - This guide will assume you are using Docker containers to deploy Garage on each node. Garage can also be run independently, for instance as a [Systemd service](@/documentation/cookbook/systemd.md). You can also use an orchestrator such as Nomad or Kubernetes to automatically manage @@ -49,6 +46,42 @@ available in the different locations of your cluster is roughly the same. For instance, here, the Mercury node could be moved to Brussels; this would allow the cluster to store 2 TB of data in total. +### Best practices + +- If you have fast dedicated networking between all your nodes, and are planing to store + very large files, bump the `block_size` configuration parameter to 10 MB + (`block_size = 10485760`). + +- Garage stores its files in two locations: it uses a metadata directory to store frequently-accessed + small metadata items, and a data directory to store data blocks of uploaded objects. + Ideally, the metadata directory would be stored on an SSD (smaller but faster), + and the data directory would be stored on an HDD (larger but slower). + +- For the data directory, Garage already does checksumming and integrity verification, + so there is no need to use a filesystem such as BTRFS or ZFS that does it. + We recommend using XFS for the data partition, as it has the best performance. + Ext4 is not recommended as it has more strict limitations on the number of inodes, + which might cause issues with Garage when large numbers of objects are stored. + +- If you only have an HDD and no SSD, it's fine to put your metadata alongside the data + on the same drive. Having lots of RAM for your kernel to cache the metadata will + help a lot with performance. Make sure to use the LMDB database engine, + instead of Sled, which suffers from quite bad performance degradation on HDDs. + Sled is still the default for legacy reasons, but is not recommended anymore. + +- For the metadata storage, Garage does not do checksumming and integrity + verification on its own. If you are afraid of bitrot/data corruption, + put your metadata directory on a BTRFS partition. Otherwise, just use regular + EXT4 or XFS. + +- Having a single server with several storage drives is currently not very well + supported in Garage ([#218](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/218)). + For an easy setup, just put all your drives in a RAID0 or a ZFS RAIDZ array. + If you're adventurous, you can try to format each of your disk as + a separate XFS partition, and then run one `garage` daemon per disk drive, + or use something like [`mergerfs`](https://github.com/trapexit/mergerfs) to merge + all your disks in a single union filesystem that spreads load over them. + ## Get a Docker image Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated). @@ -81,6 +114,7 @@ A valid `/etc/garage/garage.toml` for our cluster would look as follows: ```toml metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" +db_engine = "lmdb" replication_mode = "3" @@ -90,8 +124,6 @@ rpc_bind_addr = "[::]:3901" rpc_public_addr = ":3901" rpc_secret = "" -bootstrap_peers = [] - [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" -- cgit v1.2.3 From fc8f795bba1e23c8932af322959e0e3db2ef1233 Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Wed, 16 Nov 2022 13:33:33 +0100 Subject: Rename subsections and add docker compose file --- doc/book/cookbook/real-world.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) (limited to 'doc/book/cookbook/real-world.md') diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md index 6fe7e547..f0fe3bd1 100644 --- a/doc/book/cookbook/real-world.md +++ b/doc/book/cookbook/real-world.md @@ -164,6 +164,21 @@ It should be restarted automatically at each reboot. Please note that we use host networking as otherwise Docker containers can not communicate with IPv6. +If you want to use `docker-compose`, you may use the following `docker-compose.yml` file as a reference: + +```yaml +version: "3" +services: + garage: + image: dxflrs/garage:v0.8.0 + network_mode: "host" + restart: unless-stopped + volumes: + - /etc/garage.toml:/etc/garage.toml + - /var/lib/garage/meta:/var/lib/garage/meta + - /var/lib/garage/data:/var/lib/garage/data +``` + Upgrading between Garage versions should be supported transparently, but please check the relase notes before doing so! To upgrade, simply stop and remove this container and -- cgit v1.2.3 From 555a54ec4063f253e2afdc54a07c43d6d3cb582d Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Wed, 16 Nov 2022 13:40:49 +0100 Subject: doc precisions and fixes --- doc/book/cookbook/real-world.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/book/cookbook/real-world.md') diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md index f0fe3bd1..0571969f 100644 --- a/doc/book/cookbook/real-world.md +++ b/doc/book/cookbook/real-world.md @@ -60,7 +60,7 @@ to store 2 TB of data in total. - For the data directory, Garage already does checksumming and integrity verification, so there is no need to use a filesystem such as BTRFS or ZFS that does it. We recommend using XFS for the data partition, as it has the best performance. - Ext4 is not recommended as it has more strict limitations on the number of inodes, + EXT4 is not recommended as it has more strict limitations on the number of inodes, which might cause issues with Garage when large numbers of objects are stored. - If you only have an HDD and no SSD, it's fine to put your metadata alongside the data -- cgit v1.2.3