diff options
Diffstat (limited to 'doc/book/cookbook/real-world.md')
| -rw-r--r-- | doc/book/cookbook/real-world.md | 88 |
1 files changed, 45 insertions, 43 deletions
diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md index 7061069f..ea4ce1f9 100644 --- a/doc/book/cookbook/real-world.md +++ b/doc/book/cookbook/real-world.md @@ -19,9 +19,10 @@ To run a real-world deployment, make sure the following conditions are met: - You have at least three machines with sufficient storage space available. -- Each machine has a public IP address which is reachable by other machines. It - is highly recommended that you use IPv6 for this end-to-end connectivity. If - IPv6 is not available, then using a mesh VPN such as +- Each machine has an IP address which makes it directly reachable by all other machines. + In many cases, nodes will be behind a NAT and will not each have a public + IPv4 addresses. In this case, is recommended that you use IPv6 for this + end-to-end connectivity if it is available. Otherwise, using a mesh VPN such as [Nebula](https://github.com/slackhq/nebula) or [Yggdrasil](https://yggdrasil-network.github.io/) are approaches to consider in addition to building out your own VPN tunneling. @@ -42,7 +43,7 @@ For our example, we will suppose the following infrastructure with IPv6 connecti | Brussels | Mars | fc00:F::1 | 1.5 TB | Note that Garage will **always** store the three copies of your data on nodes at different -locations. This means that in the case of this small example, the available capacity +locations. This means that in the case of this small example, the usable capacity of the cluster is in fact only 1.5 TB, because nodes in Brussels can't store more than that. This also means that nodes in Paris and London will be under-utilized. To make better use of the available hardware, you should ensure that the capacity @@ -75,28 +76,23 @@ to store 2 TB of data in total. - For the metadata storage, Garage does not do checksumming and integrity verification on its own. If you are afraid of bitrot/data corruption, - put your metadata directory on a BTRFS partition. Otherwise, just use regular + put your metadata directory on a ZFS or BTRFS partition. Otherwise, just use regular EXT4 or XFS. -- Having a single server with several storage drives is currently not very well - supported in Garage ([#218](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/218)). - For an easy setup, just put all your drives in a RAID0 or a ZFS RAIDZ array. - If you're adventurous, you can try to format each of your disk as - a separate XFS partition, and then run one `garage` daemon per disk drive, - or use something like [`mergerfs`](https://github.com/trapexit/mergerfs) to merge - all your disks in a single union filesystem that spreads load over them. +- Servers with multiple HDDs are supported natively by Garage without resorting + to RAID, see [our dedicated documentation page](@/documentation/operations/multi-hdd.md). ## Get a Docker image Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated). -We encourage you to use a fixed tag (eg. `v0.8.0`) and not the `latest` tag. -For this example, we will use the latest published version at the time of the writing which is `v0.8.0` but it's up to you +We encourage you to use a fixed tag (eg. `v0.9.0`) and not the `latest` tag. +For this example, we will use the latest published version at the time of the writing which is `v0.9.0` but it's up to you to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated). For example: ``` -sudo docker pull dxflrs/garage:v0.8.0 +sudo docker pull dxflrs/garage:v0.9.0 ``` ## Deploying and configuring Garage @@ -161,12 +157,13 @@ docker run \ -v /etc/garage.toml:/etc/garage.toml \ -v /var/lib/garage/meta:/var/lib/garage/meta \ -v /var/lib/garage/data:/var/lib/garage/data \ - dxflrs/garage:v0.8.0 + dxflrs/garage:v0.9.0 ``` -It should be restarted automatically at each reboot. -Please note that we use host networking as otherwise Docker containers -can not communicate with IPv6. +With this command line, Garage should be started automatically at each boot. +Please note that we use host networking as otherwise the network indirection +added by Docker would prevent Garage nodes from communicating with one another +(especially if using IPv6). If you want to use `docker-compose`, you may use the following `docker-compose.yml` file as a reference: @@ -174,7 +171,7 @@ If you want to use `docker-compose`, you may use the following `docker-compose.y version: "3" services: garage: - image: dxflrs/garage:v0.8.0 + image: dxflrs/garage:v0.9.0 network_mode: "host" restart: unless-stopped volumes: @@ -183,10 +180,12 @@ services: - /var/lib/garage/data:/var/lib/garage/data ``` -Upgrading between Garage versions should be supported transparently, -but please check the relase notes before doing so! -To upgrade, simply stop and remove this container and -start again the command with a new version of Garage. +If you wish to upgrade your cluster, make sure to read the corresponding +[documentation page](@/documentation/operations/upgrading.md) first, as well as +the documentation relevant to your version of Garage in the case of major +upgrades. With the containerized setup proposed here, the upgrade process +will require stopping and removing the existing container, and re-creating it +with the upgraded version. ## Controling the daemon @@ -270,12 +269,12 @@ of a role that is assigned to each active cluster node. For our example, we will suppose we have the following infrastructure (Capacity, Identifier and Zone are specific values to Garage described in the following): -| Location | Name | Disk Space | `Capacity` | `Identifier` | `Zone` | -|----------|---------|------------|------------|--------------|--------------| -| Paris | Mercury | 1 TB | `10` | `563e` | `par1` | -| Paris | Venus | 2 TB | `20` | `86f0` | `par1` | -| London | Earth | 2 TB | `20` | `6814` | `lon1` | -| Brussels | Mars | 1.5 TB | `15` | `212f` | `bru1` | +| Location | Name | Disk Space | Identifier | Zone (`-z`) | Capacity (`-c`) | +|----------|---------|------------|------------|-------------|-----------------| +| Paris | Mercury | 1 TB | `563e` | `par1` | `1T` | +| Paris | Venus | 2 TB | `86f0` | `par1` | `2T` | +| London | Earth | 2 TB | `6814` | `lon1` | `2T` | +| Brussels | Mars | 1.5 TB | `212f` | `bru1` | `1.5T` | #### Node identifiers @@ -297,6 +296,8 @@ garage status It will display the IP address associated with each node; from the IP address you will be able to recognize the node. +We will now use the `garage layout assign` command to configure the correct parameters for each node. + #### Zones Zones are simply a user-chosen identifier that identify a group of server that are grouped together logically. @@ -306,29 +307,29 @@ In most cases, a zone will correspond to a geographical location (i.e. a datacen Behind the scene, Garage will use zone definition to try to store the same data on different zones, in order to provide high availability despite failure of a zone. +Zones are passed to Garage using the `-z` flag of `garage layout assign` (see below). + #### Capacity -Garage reasons on an abstract metric about disk storage that is named the *capacity* of a node. -The capacity configured in Garage must be proportional to the disk space dedicated to the node. +Garage needs to know the storage capacity (disk space) it can/should use on +each node, to be able to correctly balance data. + +Capacity values are expressed in bytes and are passed to Garage using the `-c` flag of `garage layout assign` (see below). -Capacity values must be **integers** but can be given any signification. -Here we chose that 1 unit of capacity = 100 GB. +#### Tags -Note that the amount of data stored by Garage on each server may not be strictly proportional to -its capacity value, as Garage will priorize having 3 copies of data in different zones, -even if this means that capacities will not be strictly respected. For example in our above examples, -nodes Earth and Mars will always store a copy of everything each, and the third copy will -have 66% chance of being stored by Venus and 33% chance of being stored by Mercury. +You can add additional tags to nodes using the `-t` flag of `garage layout assign` (see below). +Tags have no specific meaning for Garage and can be used at your convenience. #### Injecting the topology Given the information above, we will configure our cluster as follow: ```bash -garage layout assign 563e -z par1 -c 10 -t mercury -garage layout assign 86f0 -z par1 -c 20 -t venus -garage layout assign 6814 -z lon1 -c 20 -t earth -garage layout assign 212f -z bru1 -c 15 -t mars +garage layout assign 563e -z par1 -c 1T -t mercury +garage layout assign 86f0 -z par1 -c 2T -t venus +garage layout assign 6814 -z lon1 -c 2T -t earth +garage layout assign 212f -z bru1 -c 1.5T -t mars ``` At this point, the changes in the cluster layout have not yet been applied. @@ -338,6 +339,7 @@ To show the new layout that will be applied, call: garage layout show ``` +Make sure to read carefully the output of `garage layout show`. Once you are satisfied with your new layout, apply it with: ```bash |