diff options
Diffstat (limited to 'doc/book')
26 files changed, 928 insertions, 709 deletions
diff --git a/doc/book/build/_index.md b/doc/book/build/_index.md new file mode 100644 index 00000000..9bb17086 --- /dev/null +++ b/doc/book/build/_index.md @@ -0,0 +1,54 @@ ++++ +title = "Build your own app" +weight = 4 +sort_by = "weight" +template = "documentation.html" ++++ + +Garage has many API that you can rely on to build complex applications. +In this section, we reference the existing SDKs and give some code examples. + + +## ⚠️ DISCLAIMER + +**K2V AND ADMIN SDK ARE TECHNICAL PREVIEWS**. The following limitations apply: + - The API is not complete, some actions are possible only through the `garage` binary + - The underlying admin API is not yet stable nor complete, it can breaks at any time + - The generator configuration is currently tweaked, the library might break at any time due to a generator change + - Because the API and the library are not stable, none of them are published in a package manager (npm, pypi, etc.) + - This code has not been extensively tested, some things might not work (please report!) + +To have the best experience possible, please consider: + - Make sure that the version of the library you are using is pinned (`go.sum`, `package-lock.json`, `requirements.txt`). + - Before upgrading your Garage cluster, make sure that you can find a version of this SDK that works with your targeted version and that you are able to update your own code to work with this new version of the library. + - Join our Matrix channel at `#garage:deuxfleurs.fr`, say that you are interested by this SDK, and report any friction. + - If stability is critical, mirror this repository on your own infrastructure, regenerate the SDKs and upgrade them at your own pace. + + +## About the APIs + +Code can interact with Garage through 3 different APIs: S3, K2V, and Admin. +Each of them has a specific scope. + +### S3 + +De-facto standard, introduced by Amazon, designed to store blobs of data. + +### K2V + +A simple database API similar to RiakKV or DynamoDB. +Think a key value store with some additional operations. +Its design is inspired by Distributed Hash Tables (DHT). + +More information: + - [In the reference manual](@/documentation/reference-manual/k2v.md) + + +### Administration + +Garage operations can also be automated through a REST API. +We are currently building this SDK for [Python](@/documentation/build/python.md#admin-api), [Javascript](@/documentation/build/javascript.md#administration) and [Golang](@/documentation/build/golang.md#administration). + +More information: + - [In the reference manual](@/documentation/reference-manual/admin-api.md) + - [Full specifiction](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.html) diff --git a/doc/book/build/golang.md b/doc/book/build/golang.md new file mode 100644 index 00000000..a508260e --- /dev/null +++ b/doc/book/build/golang.md @@ -0,0 +1,69 @@ ++++ +title = "Golang" +weight = 30 ++++ + +## S3 + +*Coming soon* + +Some refs: + - Minio minio-go-sdk + - [Reference](https://docs.min.io/docs/golang-client-api-reference.html) + + - Amazon aws-sdk-go-v2 + - [Installation](https://aws.github.io/aws-sdk-go-v2/docs/getting-started/) + - [Reference](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3) + - [Example](https://aws.github.io/aws-sdk-go-v2/docs/code-examples/s3/putobject/) + +## K2V + +*Coming soon* + +## Administration + +Install the SDK with: + +```bash +go get git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-golang +``` + +A short example: + +```go +package main + +import ( + "context" + "fmt" + "os" + garage "git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-golang" +) + +func main() { + // Set Host and other parameters + configuration := garage.NewConfiguration() + configuration.Host = "127.0.0.1:3903" + + + // We can now generate a client + client := garage.NewAPIClient(configuration) + + // Authentication is handled through the context pattern + ctx := context.WithValue(context.Background(), garage.ContextAccessToken, "s3cr3t") + + // Send a request + resp, r, err := client.NodesApi.GetNodes(ctx).Execute() + if err != nil { + fmt.Fprintf(os.Stderr, "Error when calling `NodesApi.GetNodes``: %v\n", err) + fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r) + } + + // Process the response + fmt.Fprintf(os.Stdout, "Target hostname: %v\n", resp.KnownNodes[resp.Node].Hostname) +} +``` + +See also: + - [generated doc](https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-golang) + - [examples](https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-generator/src/branch/main/example/golang) diff --git a/doc/book/build/javascript.md b/doc/book/build/javascript.md new file mode 100644 index 00000000..ff009ffe --- /dev/null +++ b/doc/book/build/javascript.md @@ -0,0 +1,55 @@ ++++ +title = "Javascript" +weight = 10 ++++ + +## S3 + +*Coming soon*. + +Some refs: + - Minio SDK + - [Reference](https://docs.min.io/docs/javascript-client-api-reference.html) + + - Amazon aws-sdk-js + - [Installation](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-started.html) + - [Reference](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html) + - [Example](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/s3-example-creating-buckets.html) + +## K2V + +*Coming soon* + +## Administration + +Install the SDK with: + +```bash +npm install --save git+https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-js.git +``` + +A short example: + +```javascript +const garage = require('garage_administration_api_v0garage_v0_8_0'); + +const api = new garage.ApiClient("http://127.0.0.1:3903/v0"); +api.authentications['bearerAuth'].accessToken = "s3cr3t"; + +const [node, layout, key, bucket] = [ + new garage.NodesApi(api), + new garage.LayoutApi(api), + new garage.KeyApi(api), + new garage.BucketApi(api), +]; + +node.getNodes().then((data) => { + console.log(`nodes: ${Object.values(data.knownNodes).map(n => n.hostname)}`) +}, (error) => { + console.error(error); +}); +``` + +See also: + - [sdk repository](https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-js) + - [examples](https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-generator/src/branch/main/example/javascript) diff --git a/doc/book/connect/code.md b/doc/book/build/others.md index 4b2c4cb0..341e82d5 100644 --- a/doc/book/connect/code.md +++ b/doc/book/build/others.md @@ -1,8 +1,10 @@ +++ -title = "Your code (PHP, JS, Go...)" -weight = 30 +title = "Others" +weight = 99 +++ +## S3 + If you are developping a new application, you may want to use Garage to store your user's media. The S3 API that Garage uses is a standard REST API, so as long as you can make HTTP requests, @@ -13,44 +15,14 @@ Instead, there are some libraries already avalaible. Some of them are maintained by Amazon, some by Minio, others by the community. -## PHP +### PHP - Amazon aws-sdk-php - [Installation](https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/getting-started_installation.html) - [Reference](https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-s3-2006-03-01.html) - [Example](https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-examples-creating-buckets.html) -## Javascript - - - Minio SDK - - [Reference](https://docs.min.io/docs/javascript-client-api-reference.html) - - - Amazon aws-sdk-js - - [Installation](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-started.html) - - [Reference](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html) - - [Example](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/s3-example-creating-buckets.html) - -## Golang - - - Minio minio-go-sdk - - [Reference](https://docs.min.io/docs/golang-client-api-reference.html) - - - Amazon aws-sdk-go-v2 - - [Installation](https://aws.github.io/aws-sdk-go-v2/docs/getting-started/) - - [Reference](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3) - - [Example](https://aws.github.io/aws-sdk-go-v2/docs/code-examples/s3/putobject/) - -## Python - - - Minio SDK - - [Reference](https://docs.min.io/docs/python-client-api-reference.html) - - - Amazon boto3 - - [Installation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html) - - [Reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html) - - [Example](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html) - -## Java +### Java - Minio SDK - [Reference](https://docs.min.io/docs/java-client-api-reference.html) @@ -60,23 +32,18 @@ Some of them are maintained by Amazon, some by Minio, others by the community. - [Reference](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html) - [Example](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-s3-objects.html) -## Rust - - - Amazon aws-rust-sdk - - [Github](https://github.com/awslabs/aws-sdk-rust) - -## .NET +### .NET - Minio SDK - [Reference](https://docs.min.io/docs/dotnet-client-api-reference.html) - Amazon aws-dotnet-sdk -## C++ +### C++ - Amazon aws-cpp-sdk -## Haskell +### Haskell - Minio SDK - [Reference](https://docs.min.io/docs/haskell-client-api-reference.html) diff --git a/doc/book/build/python.md b/doc/book/build/python.md new file mode 100644 index 00000000..5b797897 --- /dev/null +++ b/doc/book/build/python.md @@ -0,0 +1,138 @@ ++++ +title = "Python" +weight = 20 ++++ + +## S3 + +### Using Minio SDK + +First install the SDK: + +```bash +pip3 install minio +``` + +Then instantiate a client object using garage root domain, api key and secret: + +```python +import minio + +client = minio.Minio( + "your.domain.tld", + "GKyourapikey", + "abcd[...]1234", + # Force the region, this is specific to garage + region="region", +) +``` + +Then use all the standard S3 endpoints as implemented by the Minio SDK: + +``` +# List buckets +print(client.list_buckets()) + +# Put an object containing 'content' to /path in bucket named 'bucket': +content = b"content" +client.put_object( + "bucket", + "path", + io.BytesIO(content), + len(content), +) + +# Read the object back and check contents +data = client.get_object("bucket", "path").read() +assert data == content +``` + +For further documentation, see the Minio SDK +[Reference](https://docs.min.io/docs/python-client-api-reference.html) + +### Using Amazon boto3 + +*Coming soon* + +See the official documentation: + - [Installation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html) + - [Reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html) + - [Example](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html) + +## K2V + +*Coming soon* + +## Admin API + +You need at least Python 3.6, pip, and setuptools. +Because the python package is in a subfolder, the command is a bit more complicated than usual: + +```bash +pip3 install --user 'git+https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-python' +``` + +Now, let imagine you have a fresh Garage instance running on localhost, with the admin API configured on port 3903 with the bearer `s3cr3t`: + +```python +import garage_admin_sdk +from garage_admin_sdk.apis import * +from garage_admin_sdk.models import * + +configuration = garage_admin_sdk.Configuration( + host = "http://localhost:3903/v0", + access_token = "s3cr3t" +) + +# Init APIs +api = garage_admin_sdk.ApiClient(configuration) +nodes, layout, keys, buckets = NodesApi(api), LayoutApi(api), KeyApi(api), BucketApi(api) + +# Display some info on the node +status = nodes.get_nodes() +print(f"running garage {status.garage_version}, node_id {status.node}") + +# Change layout of this node +current = layout.get_layout() +layout.add_layout({ + status.node: NodeClusterInfo( + zone = "dc1", + capacity = 1, + tags = [ "dev" ], + ) +}) +layout.apply_layout(LayoutVersion( + version = current.version + 1 +)) + +# Create key, allow it to create buckets +kinfo = keys.add_key(AddKeyRequest(name="openapi")) + +allow_create = UpdateKeyRequestAllow(create_bucket=True) +keys.update_key(kinfo.access_key_id, UpdateKeyRequest(allow=allow_create)) + +# Create a bucket, allow key, set quotas +binfo = buckets.create_bucket(CreateBucketRequest(global_alias="documentation")) +binfo = buckets.allow_bucket_key(AllowBucketKeyRequest( + bucket_id=binfo.id, + access_key_id=kinfo.access_key_id, + permissions=AllowBucketKeyRequestPermissions(read=True, write=True, owner=True), +)) +binfo = buckets.update_bucket(binfo.id, UpdateBucketRequest( + quotas=UpdateBucketRequestQuotas(max_size=19029801,max_objects=1500))) + +# Display key +print(f""" +cluster ready +key id is {kinfo.access_key_id} +secret key is {kinfo.secret_access_key} +bucket {binfo.global_aliases[0]} contains {binfo.objects}/{binfo.quotas.max_objects} objects +""") +``` + +*This example is named `short.py` in the example folder. Other python examples are also available.* + +See also: + - [sdk repo](https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-python) + - [examples](https://git.deuxfleurs.fr/garage-sdk/garage-admin-sdk-generator/src/branch/main/example/python) + diff --git a/doc/book/build/rust.md b/doc/book/build/rust.md new file mode 100644 index 00000000..7101ba6e --- /dev/null +++ b/doc/book/build/rust.md @@ -0,0 +1,47 @@ ++++ +title = "Rust" +weight = 40 ++++ + +## S3 + +*Coming soon* + +Some refs: + - Amazon aws-rust-sdk + - [Github](https://github.com/awslabs/aws-sdk-rust) + +## K2V + +*Coming soon* + +Some refs: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/k2v-client + +```bash +# all these values can be provided on the cli instead +export AWS_ACCESS_KEY_ID=GK123456 +export AWS_SECRET_ACCESS_KEY=0123..789 +export AWS_REGION=garage +export K2V_ENDPOINT=http://172.30.2.1:3903 +export K2V_BUCKET=my-bucket + +cargo run --features=cli -- read-range my-partition-key --all + +cargo run --features=cli -- insert my-partition-key my-sort-key --text "my string1" +cargo run --features=cli -- insert my-partition-key my-sort-key --text "my string2" +cargo run --features=cli -- insert my-partition-key my-sort-key2 --text "my string" + +cargo run --features=cli -- read-range my-partition-key --all + +causality=$(cargo run --features=cli -- read my-partition-key my-sort-key2 -b | head -n1) +cargo run --features=cli -- delete my-partition-key my-sort-key2 -c $causality + +causality=$(cargo run --features=cli -- read my-partition-key my-sort-key -b | head -n1) +cargo run --features=cli -- insert my-partition-key my-sort-key --text "my string3" -c $causality + +cargo run --features=cli -- read-range my-partition-key --all +``` + +## Admin API + +*Coming soon* diff --git a/doc/book/connect/_index.md b/doc/book/connect/_index.md index 07c4e31f..ca44ac17 100644 --- a/doc/book/connect/_index.md +++ b/doc/book/connect/_index.md @@ -1,5 +1,5 @@ +++ -title = "Integrations" +title = "Existing integrations" weight = 3 sort_by = "weight" template = "documentation.html" @@ -14,7 +14,6 @@ In particular, you will find here instructions to connect it with: - [Applications](@/documentation/connect/apps/index.md) - [Website hosting](@/documentation/connect/websites.md) - [Software repositories](@/documentation/connect/repositories.md) - - [Your own code](@/documentation/connect/code.md) - [FUSE](@/documentation/connect/fs.md) ### Generic instructions diff --git a/doc/book/connect/apps/index.md b/doc/book/connect/apps/index.md index 05e7cad9..78d9310d 100644 --- a/doc/book/connect/apps/index.md +++ b/doc/book/connect/apps/index.md @@ -8,7 +8,7 @@ In this section, we cover the following web applications: | Name | Status | Note | |------|--------|------| | [Nextcloud](#nextcloud) | ✅ | Both Primary Storage and External Storage are supported | -| [Peertube](#peertube) | ✅ | Must be configured with the website endpoint | +| [Peertube](#peertube) | ✅ | Supported with the website endpoint, proxifying private videos unsupported | | [Mastodon](#mastodon) | ✅ | Natively supported | | [Matrix](#matrix) | ✅ | Tested with `synapse-s3-storage-provider` | | [Pixelfed](#pixelfed) | ❓ | Not yet tested | @@ -36,7 +36,7 @@ Second, we suppose you have created a key and a bucket. As a reminder, you can create a key for your nextcloud instance as follow: ```bash -garage key new --name nextcloud-key +garage key create nextcloud-key ``` Keep the Key ID and the Secret key in a pad, they will be needed later. @@ -128,13 +128,17 @@ In other words, Peertube is only responsible of the "control plane" and offload In return, this system is a bit harder to configure. We show how it is still possible to configure Garage with Peertube, allowing you to spread the load and the bandwidth usage on the Garage cluster. +Starting from version 5.0, Peertube also supports improving the security for private videos by not exposing them directly +but relying on a single control point in the Peertube instance. This is based on S3 per-object and prefix ACL, which are not currently supported +in Garage, so this feature is unsupported. While this technically impedes security for private videos, it is not a blocking issue and could be +a reasonable trade-off for some instances. ### Create resources in Garage Create a key for Peertube: ```bash -garage key new --name peertube-key +garage key create peertube-key ``` Keep the Key ID and the Secret key in a pad, they will be needed later. @@ -195,6 +199,11 @@ object_storage: max_upload_part: 2GB + proxy: + # You may enable this feature, yet it will not provide any security benefit, so + # you should rather benefit from Garage public endpoint for all videos + proxify_private_files: false + streaming_playlists: bucket_name: 'peertube-playlist' @@ -243,7 +252,7 @@ As such, your Garage cluster should be configured appropriately for good perform This is the usual Garage setup: ```bash -garage key new --name mastodon-key +garage key create mastodon-key garage bucket create mastodon-data garage bucket allow mastodon-data --read --write --key mastodon-key ``` @@ -369,7 +378,7 @@ Supposing you have a working synapse installation, you can add the module with p Now create a bucket and a key for your matrix instance (note your Key ID and Secret Key somewhere, they will be needed later): ```bash -garage key new --name matrix-key +garage key create matrix-key garage bucket create matrix garage bucket allow matrix --read --write --key matrix-key ``` diff --git a/doc/book/connect/backup.md b/doc/book/connect/backup.md index 48a2d7be..919e78c3 100644 --- a/doc/book/connect/backup.md +++ b/doc/book/connect/backup.md @@ -20,7 +20,7 @@ If you still want to use Borg, you can use it with `rclone mount`. Create your key and bucket: ```bash -garage key new my-key +garage key create my-key garage bucket create backup garage bucket allow backup --read --write --key my-key ``` diff --git a/doc/book/connect/repositories.md b/doc/book/connect/repositories.md index 4b14bb46..66365d64 100644 --- a/doc/book/connect/repositories.md +++ b/doc/book/connect/repositories.md @@ -23,7 +23,7 @@ You can configure a different target for each data type (check `[lfs]` and `[att Let's start by creating a key and a bucket (your key id and secret will be needed later, keep them somewhere): ```bash -garage key new --name gitea-key +garage key create gitea-key garage bucket create gitea garage bucket allow gitea --read --write --key gitea-key ``` @@ -118,7 +118,7 @@ through another support, like a git repository. As a first step, we will need to create a bucket on Garage and enabling website access on it: ```bash -garage key new --name nix-key +garage key create nix-key garage bucket create nix.example.com garage bucket allow nix.example.com --read --write --key nix-key garage bucket website nix.example.com --allow diff --git a/doc/book/cookbook/monitoring.md b/doc/book/cookbook/monitoring.md new file mode 100644 index 00000000..8206f645 --- /dev/null +++ b/doc/book/cookbook/monitoring.md @@ -0,0 +1,306 @@ ++++ +title = "Monitoring Garage" +weight = 40 ++++ + +Garage exposes some internal metrics in the Prometheus data format. +This page explains how to exploit these metrics. + +## Setting up monitoring + +### Enabling the Admin API endpoint + +If you have not already enabled the [administration API endpoint](@/documentation/reference-manual/admin-api.md), do so by adding the following lines to your configuration file: + +```toml +[admin] +api_bind_addr = "0.0.0.0:3903" +``` + +This will allow anyone to scrape Prometheus metrics by fetching +`http://localhost:3093/metrics`. If you want to restrict access +to the exported metrics, set the `metrics_token` configuration value +to a bearer token to be used when fetching the metrics endpoint. + +### Setting up Prometheus and Grafana + +Add a scrape config to your Prometheus daemon to scrape metrics from +all of your nodes: + +```yaml +scrape_configs: + - job_name: 'garage' + static_configs: + - targets: + - 'node1.mycluster:3903' + - 'node2.mycluster:3903' + - 'node3.mycluster:3903' +``` + +If you have set a metrics token in your Garage configuration file, +add the following lines in your Prometheus scrape config: + +```yaml + authorization: + type: Bearer + credentials: 'your metrics token' +``` + +To visualize the scraped data in Grafana, +you can either import our [Grafana dashboard for Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/branch/main/script/telemetry/grafana-garage-dashboard-prometheus.json) +or make your own. +We detail below the list of exposed metrics and their meaning. + + + +## List of exported metrics + + +### Metrics of the API endpoints + +#### `api_admin_request_counter` (counter) + +Counts the number of requests to a given endpoint of the administration API. Example: + +``` +api_admin_request_counter{api_endpoint="Metrics"} 127041 +``` + +#### `api_admin_request_duration` (histogram) + +Evaluates the duration of API calls to the various administration API endpoint. Example: + +``` +api_admin_request_duration_bucket{api_endpoint="Metrics",le="0.5"} 127041 +api_admin_request_duration_sum{api_endpoint="Metrics"} 605.250344830999 +api_admin_request_duration_count{api_endpoint="Metrics"} 127041 +``` + +#### `api_s3_request_counter` (counter) + +Counts the number of requests to a given endpoint of the S3 API. Example: + +``` +api_s3_request_counter{api_endpoint="CreateMultipartUpload"} 1 +``` + +#### `api_s3_error_counter` (counter) + +Counts the number of requests to a given endpoint of the S3 API that returned an error. Example: + +``` +api_s3_error_counter{api_endpoint="GetObject",status_code="404"} 39 +``` + +#### `api_s3_request_duration` (histogram) + +Evaluates the duration of API calls to the various S3 API endpoints. Example: + +``` +api_s3_request_duration_bucket{api_endpoint="CreateMultipartUpload",le="0.5"} 1 +api_s3_request_duration_sum{api_endpoint="CreateMultipartUpload"} 0.046340762 +api_s3_request_duration_count{api_endpoint="CreateMultipartUpload"} 1 +``` + +#### `api_k2v_request_counter` (counter), `api_k2v_error_counter` (counter), `api_k2v_error_duration` (histogram) + +Same as for S3, for the K2V API. + + +### Metrics of the Web endpoint + + +#### `web_request_counter` (counter) + +Number of requests to the web endpoint + +``` +web_request_counter{method="GET"} 80 +``` + +#### `web_request_duration` (histogram) + +Duration of requests to the web endpoint + +``` +web_request_duration_bucket{method="GET",le="0.5"} 80 +web_request_duration_sum{method="GET"} 1.0528433229999998 +web_request_duration_count{method="GET"} 80 +``` + +#### `web_error_counter` (counter) + +Number of requests to the web endpoint resulting in errors + +``` +web_error_counter{method="GET",status_code="404 Not Found"} 64 +``` + + +### Metrics of the data block manager + +#### `block_bytes_read`, `block_bytes_written` (counter) + +Number of bytes read/written to/from disk in the data storage directory. + +``` +block_bytes_read 120586322022 +block_bytes_written 3386618077 +``` + +#### `block_read_duration`, `block_write_duration` (histograms) + +Evaluates the duration of the reading/writing of individual data blocks in the data storage directory. + +``` +block_read_duration_bucket{le="0.5"} 169229 +block_read_duration_sum 2761.6902550310056 +block_read_duration_count 169240 +block_write_duration_bucket{le="0.5"} 3559 +block_write_duration_sum 195.59170078500006 +block_write_duration_count 3571 +``` + +#### `block_delete_counter` (counter) + +Counts the number of data blocks that have been deleted from storage. + +``` +block_delete_counter 122 +``` + +#### `block_resync_counter` (counter), `block_resync_duration` (histogram) + +Counts the number of resync operations the node has executed, and evaluates their duration. + +``` +block_resync_counter 308897 +block_resync_duration_bucket{le="0.5"} 308892 +block_resync_duration_sum 139.64204196100016 +block_resync_duration_count 308897 +``` + +#### `block_resync_queue_length` (gauge) + +The number of block hashes currently queued for a resync. +This is normal to be nonzero for long periods of time. + +``` +block_resync_queue_length 0 +``` + +#### `block_resync_errored_blocks` (gauge) + +The number of block hashes that we were unable to resync last time we tried. +**THIS SHOULD BE ZERO, OR FALL BACK TO ZERO RAPIDLY, IN A HEALTHY CLUSTER.** +Persistent nonzero values indicate that some data is likely to be lost. + +``` +block_resync_errored_blocks 0 +``` + + +### Metrics related to RPCs (remote procedure calls) between nodes + +#### `rpc_netapp_request_counter` (counter) + +Number of RPC requests emitted + +``` +rpc_request_counter{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 176 +``` + +#### `rpc_netapp_error_counter` (counter) + +Number of communication errors (errors in the Netapp library, generally due to disconnected nodes) + +``` +rpc_netapp_error_counter{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 354 +``` + +#### `rpc_timeout_counter` (counter) + +Number of RPC timeouts, should be close to zero in a healthy cluster. + +``` +rpc_timeout_counter{from="<this node>",rpc_endpoint="garage_rpc/membership.rs/SystemRpc",to="<remote node>"} 1 +``` + +#### `rpc_duration` (histogram) + +The duration of internal RPC calls between Garage nodes. + +``` +rpc_duration_bucket{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>",le="0.5"} 166 +rpc_duration_sum{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 35.172253716 +rpc_duration_count{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 174 +``` + + +### Metrics of the metadata table manager + +#### `table_gc_todo_queue_length` (gauge) + +Table garbage collector TODO queue length + +``` +table_gc_todo_queue_length{table_name="block_ref"} 0 +``` + +#### `table_get_request_counter` (counter), `table_get_request_duration` (histogram) + +Number of get/get_range requests internally made on each table, and their duration. + +``` +table_get_request_counter{table_name="bucket_alias"} 315 +table_get_request_duration_bucket{table_name="bucket_alias",le="0.5"} 315 +table_get_request_duration_sum{table_name="bucket_alias"} 0.048509778000000024 +table_get_request_duration_count{table_name="bucket_alias"} 315 +``` + + +#### `table_put_request_counter` (counter), `table_put_request_duration` (histogram) + +Number of insert/insert_many requests internally made on this table, and their duration + +``` +table_put_request_counter{table_name="block_ref"} 677 +table_put_request_duration_bucket{table_name="block_ref",le="0.5"} 677 +table_put_request_duration_sum{table_name="block_ref"} 61.617528636 +table_put_request_duration_count{table_name="block_ref"} 677 +``` + +#### `table_internal_delete_counter` (counter) + +Number of value deletions in the tree (due to GC or repartitioning) + +``` +table_internal_delete_counter{table_name="block_ref"} 2296 +``` + +#### `table_internal_update_counter` (counter) + +Number of value updates where the value actually changes (includes creation of new key and update of existing key) + +``` +table_internal_update_counter{table_name="block_ref"} 5996 +``` + +#### `table_merkle_updater_todo_queue_length` (gauge) + +Merkle tree updater TODO queue length (should fall to zero rapidly) + +``` +table_merkle_updater_todo_queue_length{table_name="block_ref"} 0 +``` + +#### `table_sync_items_received`, `table_sync_items_sent` (counters) + +Number of data items sent to/recieved from other nodes during resync procedures + +``` +table_sync_items_received{from="<remote node>",table_name="bucket_v2"} 3 +table_sync_items_sent{table_name="block_ref",to="<remote node>"} 2 +``` + + diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md index 4fcb5cf7..5423bbab 100644 --- a/doc/book/cookbook/real-world.md +++ b/doc/book/cookbook/real-world.md @@ -11,8 +11,9 @@ We recommend first following the [quick start guide](@/documentation/quick-start to get familiar with Garage's command line and usage patterns. +## Preparing your environment -## Prerequisites +### Prerequisites To run a real-world deployment, make sure the following conditions are met: @@ -21,10 +22,6 @@ To run a real-world deployment, make sure the following conditions are met: - Each machine has a public IP address which is reachable by other machines. Running behind a NAT is likely to be possible but hasn't been tested for the latest version (TODO). -- Ideally, each machine should have a SSD available in addition to the HDD you are dedicating - to Garage. This will allow for faster access to metadata and has the potential - to significantly reduce Garage's response times. - - This guide will assume you are using Docker containers to deploy Garage on each node. Garage can also be run independently, for instance as a [Systemd service](@/documentation/cookbook/systemd.md). You can also use an orchestrator such as Nomad or Kubernetes to automatically manage @@ -49,6 +46,42 @@ available in the different locations of your cluster is roughly the same. For instance, here, the Mercury node could be moved to Brussels; this would allow the cluster to store 2 TB of data in total. +### Best practices + +- If you have fast dedicated networking between all your nodes, and are planing to store + very large files, bump the `block_size` configuration parameter to 10 MB + (`block_size = 10485760`). + +- Garage stores its files in two locations: it uses a metadata directory to store frequently-accessed + small metadata items, and a data directory to store data blocks of uploaded objects. + Ideally, the metadata directory would be stored on an SSD (smaller but faster), + and the data directory would be stored on an HDD (larger but slower). + +- For the data directory, Garage already does checksumming and integrity verification, + so there is no need to use a filesystem such as BTRFS or ZFS that does it. + We recommend using XFS for the data partition, as it has the best performance. + EXT4 is not recommended as it has more strict limitations on the number of inodes, + which might cause issues with Garage when large numbers of objects are stored. + +- If you only have an HDD and no SSD, it's fine to put your metadata alongside the data + on the same drive. Having lots of RAM for your kernel to cache the metadata will + help a lot with performance. Make sure to use the LMDB database engine, + instead of Sled, which suffers from quite bad performance degradation on HDDs. + Sled is still the default for legacy reasons, but is not recommended anymore. + +- For the metadata storage, Garage does not do checksumming and integrity + verification on its own. If you are afraid of bitrot/data corruption, + put your metadata directory on a BTRFS partition. Otherwise, just use regular + EXT4 or XFS. + +- Having a single server with several storage drives is currently not very well + supported in Garage ([#218](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/218)). + For an easy setup, just put all your drives in a RAID0 or a ZFS RAIDZ array. + If you're adventurous, you can try to format each of your disk as + a separate XFS partition, and then run one `garage` daemon per disk drive, + or use something like [`mergerfs`](https://github.com/trapexit/mergerfs) to merge + all your disks in a single union filesystem that spreads load over them. + ## Get a Docker image Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated). @@ -76,11 +109,12 @@ especially you must consider the following folders/files: this folder will be your main data storage and must be on a large storage (e.g. large HDD) -A valid `/etc/garage/garage.toml` for our cluster would look as follows: +A valid `/etc/garage.toml` for our cluster would look as follows: ```toml metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" +db_engine = "lmdb" replication_mode = "3" @@ -90,8 +124,6 @@ rpc_bind_addr = "[::]:3901" rpc_public_addr = "<this node's public IP>:3901" rpc_secret = "<RPC secret>" -bootstrap_peers = [] - [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" @@ -132,6 +164,21 @@ It should be restarted automatically at each reboot. Please note that we use host networking as otherwise Docker containers can not communicate with IPv6. +If you want to use `docker-compose`, you may use the following `docker-compose.yml` file as a reference: + +```yaml +version: "3" +services: + garage: + image: dxflrs/garage:v0.8.0 + network_mode: "host" + restart: unless-stopped + volumes: + - /etc/garage.toml:/etc/garage.toml + - /var/lib/garage/meta:/var/lib/garage/meta + - /var/lib/garage/data:/var/lib/garage/data +``` + Upgrading between Garage versions should be supported transparently, but please check the relase notes before doing so! To upgrade, simply stop and remove this container and diff --git a/doc/book/cookbook/recovering.md b/doc/book/cookbook/recovering.md index 2424558c..2129a7f3 100644 --- a/doc/book/cookbook/recovering.md +++ b/doc/book/cookbook/recovering.md @@ -1,6 +1,6 @@ +++ title = "Recovering from failures" -weight = 35 +weight = 50 +++ Garage is meant to work on old, second-hand hardware. diff --git a/doc/book/cookbook/reverse-proxy.md b/doc/book/cookbook/reverse-proxy.md index fb918778..c8fde28d 100644 --- a/doc/book/cookbook/reverse-proxy.md +++ b/doc/book/cookbook/reverse-proxy.md @@ -70,14 +70,16 @@ A possible configuration: ```nginx upstream s3_backend { - # if you have a garage instance locally + # If you have a garage instance locally. server 127.0.0.1:3900; - # you can also put your other instances + # You can also put your other instances. server 192.168.1.3:3900; - # domain names also work + # Domain names also work. server garage1.example.com:3900; - # you can assign weights if you have some servers - # that are more powerful than others + # A "backup" server is only used if all others have failed. + server garage-remote.example.com:3900 backup; + # You can assign weights if you have some servers + # that can serve more requests than others. server garage2.example.com:3900 weight=2; } @@ -96,6 +98,8 @@ server { proxy_pass http://s3_backend; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; + # Disable buffering to a temporary file. + proxy_max_temp_file_size 0; } } ``` diff --git a/doc/book/cookbook/upgrading.md b/doc/book/cookbook/upgrading.md index f7659921..9f2ba73b 100644 --- a/doc/book/cookbook/upgrading.md +++ b/doc/book/cookbook/upgrading.md @@ -1,6 +1,6 @@ +++ title = "Upgrading Garage" -weight = 40 +weight = 60 +++ Garage is a stateful clustered application, where all nodes are communicating together and share data structures. diff --git a/doc/book/design/_index.md b/doc/book/design/_index.md index f3a08aaf..a3a6ac11 100644 --- a/doc/book/design/_index.md +++ b/doc/book/design/_index.md @@ -1,6 +1,6 @@ +++ title = "Design" -weight = 5 +weight = 6 sort_by = "weight" template = "documentation.html" +++ diff --git a/doc/book/design/goals.md b/doc/book/design/goals.md index 9c2d89f0..4e390ba6 100644 --- a/doc/book/design/goals.md +++ b/doc/book/design/goals.md @@ -12,7 +12,7 @@ as pictures, video, images, documents, etc., in a redundant multi-node setting. S3 is versatile enough to also be used to publish a static website. -Garage is an opinionated object storage solutoin, we focus on the following **desirable properties**: +Garage is an opinionated object storage solution, we focus on the following **desirable properties**: - **Internet enabled**: made for multi-sites (eg. datacenters, offices, households, etc.) interconnected through regular Internet connections. - **Self-contained & lightweight**: works everywhere and integrates well in existing environments to target [hyperconverged infrastructures](https://en.wikipedia.org/wiki/Hyper-converged_infrastructure). diff --git a/doc/book/development/_index.md b/doc/book/development/_index.md index 662ec358..8e730bf6 100644 --- a/doc/book/development/_index.md +++ b/doc/book/development/_index.md @@ -1,6 +1,6 @@ +++ title = "Development" -weight = 6 +weight = 7 sort_by = "weight" template = "documentation.html" +++ diff --git a/doc/book/quick-start/_index.md b/doc/book/quick-start/_index.md index 21331dcb..ab83b75a 100644 --- a/doc/book/quick-start/_index.md +++ b/doc/book/quick-start/_index.md @@ -42,25 +42,25 @@ you can [build Garage from source](@/documentation/cookbook/from-source.md). ## Configuring and starting Garage -### Writing a first configuration file +### Generating a first configuration file This first configuration file should allow you to get started easily with the simplest possible Garage deployment. -**Save it as `/etc/garage.toml`.** -You can also store it somewhere else, but you will have to specify `-c path/to/garage.toml` -at each invocation of the `garage` binary (for example: `garage -c ./garage.toml server`, `garage -c ./garage.toml status`). -```toml +We will create it with the following command line +to generate unique and private secrets for security reasons: + +```bash +cat > garage.toml <<EOF metadata_dir = "/tmp/meta" data_dir = "/tmp/data" +db_engine = "lmdb" replication_mode = "none" rpc_bind_addr = "[::]:3901" rpc_public_addr = "127.0.0.1:3901" -rpc_secret = "1799bccfd7411eddcf9ebd316bc1f5287ad12a68094e1c6ac6abde7e6feae1ec" - -bootstrap_peers = [] +rpc_secret = "$(openssl rand -hex 32)" [s3_api] s3_region = "garage" @@ -71,12 +71,26 @@ root_domain = ".s3.garage.localhost" bind_addr = "[::]:3902" root_domain = ".web.garage.localhost" index = "index.html" + +[k2v_api] +api_bind_addr = "[::]:3904" + +[admin] +api_bind_addr = "0.0.0.0:3903" +admin_token = "$(openssl rand -base64 32)" +EOF ``` -The `rpc_secret` value provided above is just an example. It will work, but in -order to secure your cluster you will need to use another one. You can generate -such a value with `openssl rand -hex 32`. +Now that your configuration file has been created, you can put +it in the right place. By default, garage looks at **`/etc/garage.toml`.** +You can also store it somewhere else, but you will have to specify `-c path/to/garage.toml` +at each invocation of the `garage` binary (for example: `garage -c ./garage.toml server`, `garage -c ./garage.toml status`). + +As you can see, the `rpc_secret` is a 32 bytes hexadecimal string. +You can regenerate it with `openssl rand -hex 32`. +If you target a cluster deployment with multiple nodes, make sure that +you use the same value for all nodes. As you can see in the `metadata_dir` and `data_dir` parameters, we are saving Garage's data in `/tmp` which gets erased when your system reboots. This means that data stored on this @@ -192,7 +206,7 @@ one key can access multiple buckets, multiple keys can access one bucket. Create an API key using the following command: ``` -garage key new --name nextcloud-app-key +garage key create nextcloud-app-key ``` The output should look as follows: @@ -219,6 +233,7 @@ Now that we have a bucket and a key, we need to give permissions to the key on t garage bucket allow \ --read \ --write \ + --owner \ nextcloud-bucket \ --key nextcloud-app-key ``` @@ -232,54 +247,73 @@ garage bucket info nextcloud-bucket ## Uploading and downlading from Garage -We recommend the use of MinIO Client to interact with Garage files (`mc`). -Instructions to install it and use it are provided on the -[MinIO website](https://docs.min.io/docs/minio-client-quickstart-guide.html). -Before reading the following, you need a working `mc` command on your path. +To download and upload files on garage, we can use a third-party tool named `awscli`. -Note that on certain Linux distributions such as Arch Linux, the Minio client binary -is called `mcli` instead of `mc` (to avoid name clashes with the Midnight Commander). -### Configure `mc` +### Install and configure `awscli` -You need your access key and secret key created above. -We will assume you are invoking `mc` on the same machine as the Garage server, -your S3 API endpoint is therefore `http://127.0.0.1:3900`. -For this whole configuration, you must set an alias name: we chose `my-garage`, that you will used for all commands. +If you have python on your system, you can install it with: -Adapt the following command accordingly and run it: +```bash +python -m pip install --user awscli +``` + +Now that `awscli` is installed, you must configure it to talk to your Garage instance, +with your key. There are multiple ways to do that, the simplest one is to create a file +named `~/.awsrc` with this content: ```bash -mc alias set \ - my-garage \ - http://127.0.0.1:3900 \ - <access key> \ - <secret key> \ - --api S3v4 +export AWS_ACCESS_KEY_ID=xxxx # put your Key ID here +export AWS_SECRET_ACCESS_KEY=xxxx # put your Secret key here +export AWS_DEFAULT_REGION='garage' +export AWS_ENDPOINT='http://localhost:3900' + +function aws { command aws --endpoint-url $AWS_ENDPOINT $@ ; } +aws --version ``` -### Use `mc` +Now, each time you want to use `awscli` on this target, run: + +```bash +source ~/.awsrc +``` -You can not list buckets from `mc` currently. +*You can create multiple files with different names if you +have multiple Garage clusters or different keys. +Switching from one cluster to another is as simple as +sourcing the right file.* -But the following commands and many more should work: +### Example usage of `awscli` ```bash -mc cp image.png my-garage/nextcloud-bucket -mc cp my-garage/nextcloud-bucket/image.png . -mc ls my-garage/nextcloud-bucket -mc mirror localdir/ my-garage/another-bucket +# list buckets +aws s3 ls + +# list objects of a bucket +aws s3 ls s3://my_files + +# copy from your filesystem to garage +aws s3 cp /proc/cpuinfo s3://my_files/cpuinfo.txt + +# copy from garage to your filesystem +aws s3 cp s3/my_files/cpuinfo.txt /tmp/cpuinfo.txt ``` +Note that you can use `awscli` for more advanced operations like +creating a bucket, pre-signing a request or managing your website. +[Read the full documentation to know more](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html). + +Some features are however not implemented like ACL or policy. +Check [our s3 compatibility list](@/documentation/reference-manual/s3-compatibility.md). ### Other tools for interacting with Garage The following tools can also be used to send and recieve files from/to Garage: -- the [AWS CLI](https://aws.amazon.com/cli/) -- [`rclone`](https://rclone.org/) -- [Cyberduck](https://cyberduck.io/) -- [`s3cmd`](https://s3tools.org/s3cmd) +- [minio-client](@/documentation/connect/cli.md#minio-client) +- [s3cmd](@/documentation/connect/cli.md#s3cmd) +- [rclone](@/documentation/connect/cli.md#rclone) +- [Cyberduck](@/documentation/connect/cli.md#cyberduck) +- [WinSCP](@/documentation/connect/cli.md#winscp) -Refer to the ["Integrations" section](@/documentation/connect/_index.md) to learn how to -configure application and command line utilities to integrate with Garage. +An exhaustive list is maintained in the ["Integrations" > "Browsing tools" section](@/documentation/connect/_index.md). diff --git a/doc/book/reference-manual/_index.md b/doc/book/reference-manual/_index.md index 62716df8..ab1de5e6 100644 --- a/doc/book/reference-manual/_index.md +++ b/doc/book/reference-manual/_index.md @@ -1,6 +1,6 @@ +++ title = "Reference Manual" -weight = 4 +weight = 5 sort_by = "weight" template = "documentation.html" +++ diff --git a/doc/book/reference-manual/admin-api.md b/doc/book/reference-manual/admin-api.md index 3a4a7aab..0b7e2e16 100644 --- a/doc/book/reference-manual/admin-api.md +++ b/doc/book/reference-manual/admin-api.md @@ -47,598 +47,13 @@ Returns internal Garage metrics in Prometheus format. ### Cluster operations -#### GetClusterStatus `GET /v0/status` +These endpoints are defined on a dedicated [Redocly page](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.html). You can also download its [OpenAPI specification](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.yml). -Returns the cluster's current status in JSON, including: +Requesting the API from the command line can be as simple as running: -- ID of the node being queried and its version of the Garage daemon -- Live nodes -- Currently configured cluster layout -- Staged changes to the cluster layout - -Example response body: - -```json -{ - "node": "ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f", - "garage_version": "git:v0.8.0", - "knownNodes": { - "ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f": { - "addr": "10.0.0.11:3901", - "is_up": true, - "last_seen_secs_ago": 9, - "hostname": "node1" - }, - "4a6ae5a1d0d33bf895f5bb4f0a418b7dc94c47c0dd2eb108d1158f3c8f60b0ff": { - "addr": "10.0.0.12:3901", - "is_up": true, - "last_seen_secs_ago": 1, - "hostname": "node2" - }, - "23ffd0cdd375ebff573b20cc5cef38996b51c1a7d6dbcf2c6e619876e507cf27": { - "addr": "10.0.0.21:3901", - "is_up": true, - "last_seen_secs_ago": 7, - "hostname": "node3" - }, - "e2ee7984ee65b260682086ec70026165903c86e601a4a5a501c1900afe28d84b": { - "addr": "10.0.0.22:3901", - "is_up": true, - "last_seen_secs_ago": 1, - "hostname": "node4" - } - }, - "layout": { - "version": 12, - "roles": { - "ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f": { - "zone": "dc1", - "capacity": 4, - "tags": [ - "node1" - ] - }, - "4a6ae5a1d0d33bf895f5bb4f0a418b7dc94c47c0dd2eb108d1158f3c8f60b0ff": { - "zone": "dc1", - "capacity": 6, - "tags": [ - "node2" - ] - }, - "23ffd0cdd375ebff573b20cc5cef38996b51c1a7d6dbcf2c6e619876e507cf27": { - "zone": "dc2", - "capacity": 10, - "tags": [ - "node3" - ] - } - }, - "stagedRoleChanges": { - "e2ee7984ee65b260682086ec70026165903c86e601a4a5a501c1900afe28d84b": { - "zone": "dc2", - "capacity": 5, - "tags": [ - "node4" - ] - } - } - } -} -``` - -#### ConnectClusterNodes `POST /v0/connect` - -Instructs this Garage node to connect to other Garage nodes at specified addresses. - -Example request body: - -```json -[ - "ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f@10.0.0.11:3901", - "4a6ae5a1d0d33bf895f5bb4f0a418b7dc94c47c0dd2eb108d1158f3c8f60b0ff@10.0.0.12:3901" -] -``` - -The format of the string for a node to connect to is: `<node ID>@<ip address>:<port>`, same as in the `garage node connect` CLI call. - -Example response: - -```json -[ - { - "success": true, - "error": null - }, - { - "success": false, - "error": "Handshake error" - } -] -``` - -#### GetClusterLayout `GET /v0/layout` - -Returns the cluster's current layout in JSON, including: - -- Currently configured cluster layout -- Staged changes to the cluster layout - -(the info returned by this endpoint is a subset of the info returned by GetClusterStatus) - -Example response body: - -```json -{ - "version": 12, - "roles": { - "ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f": { - "zone": "dc1", - "capacity": 4, - "tags": [ - "node1" - ] - }, - "4a6ae5a1d0d33bf895f5bb4f0a418b7dc94c47c0dd2eb108d1158f3c8f60b0ff": { - "zone": "dc1", - "capacity": 6, - "tags": [ - "node2" - ] - }, - "23ffd0cdd375ebff573b20cc5cef38996b51c1a7d6dbcf2c6e619876e507cf27": { - "zone": "dc2", - "capacity": 10, - "tags": [ - "node3" - ] - } - }, - "stagedRoleChanges": { - "e2ee7984ee65b260682086ec70026165903c86e601a4a5a501c1900afe28d84b": { - "zone": "dc2", - "capacity": 5, - "tags": [ - "node4" - ] - } - } -} -``` - -#### UpdateClusterLayout `POST /v0/layout` - -Send modifications to the cluster layout. These modifications will -be included in the staged role changes, visible in subsequent calls -of `GetClusterLayout`. Once the set of staged changes is satisfactory, -the user may call `ApplyClusterLayout` to apply the changed changes, -or `Revert ClusterLayout` to clear all of the staged changes in -the layout. - -Request body format: - -```json -{ - <node_id>: { - "capacity": <new_capacity>, - "zone": <new_zone>, - "tags": [ - <new_tag>, - ... - ] - }, - <node_id_to_remove>: null, - ... -} -``` - -Contrary to the CLI that may update only a subset of the fields -`capacity`, `zone` and `tags`, when calling this API all of these -values must be specified. - - -#### ApplyClusterLayout `POST /v0/layout/apply` - -Applies to the cluster the layout changes currently registered as -staged layout changes. - -Request body format: - -```json -{ - "version": 13 -} -``` - -Similarly to the CLI, the body must include the version of the new layout -that will be created, which MUST be 1 + the value of the currently -existing layout in the cluster. - -#### RevertClusterLayout `POST /v0/layout/revert` - -Clears all of the staged layout changes. - -Request body format: - -```json -{ - "version": 13 -} -``` - -Reverting the staged changes is done by incrementing the version number -and clearing the contents of the staged change list. -Similarly to the CLI, the body must include the incremented -version number, which MUST be 1 + the value of the currently -existing layout in the cluster. - - -### Access key operations - -#### ListKeys `GET /v0/key` - -Returns all API access keys in the cluster. - -Example response: - -```json -[ - { - "id": "GK31c2f218a2e44f485b94239e", - "name": "test" - }, - { - "id": "GKe10061ac9c2921f09e4c5540", - "name": "test2" - } -] -``` - -#### CreateKey `POST /v0/key` - -Creates a new API access key. - -Request body format: - -```json -{ - "name": "NameOfMyKey" -} -``` - -#### ImportKey `POST /v0/key/import` - -Imports an existing API key. - -Request body format: - -```json -{ - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "secretAccessKey": "b892c0665f0ada8a4755dae98baa3b133590e11dae3bcc1f9d769d67f16c3835", - "name": "NameOfMyKey" -} -``` - -#### GetKeyInfo `GET /v0/key?id=<acces key id>` -#### GetKeyInfo `GET /v0/key?search=<pattern>` - -Returns information about the requested API access key. - -If `id` is set, the key is looked up using its exact identifier (faster). -If `search` is set, the key is looked up using its name or prefix -of identifier (slower, all keys are enumerated to do this). - -Example response: - -```json -{ - "name": "test", - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "secretAccessKey": "b892c0665f0ada8a4755dae98baa3b133590e11dae3bcc1f9d769d67f16c3835", - "permissions": { - "createBucket": false - }, - "buckets": [ - { - "id": "70dc3bed7fe83a75e46b66e7ddef7d56e65f3c02f9f80b6749fb97eccb5e1033", - "globalAliases": [ - "test2" - ], - "localAliases": [], - "permissions": { - "read": true, - "write": true, - "owner": false - } - }, - { - "id": "d7452a935e663fc1914f3a5515163a6d3724010ce8dfd9e4743ca8be5974f995", - "globalAliases": [ - "test3" - ], - "localAliases": [], - "permissions": { - "read": true, - "write": true, - "owner": false - } - }, - { - "id": "e6a14cd6a27f48684579ec6b381c078ab11697e6bc8513b72b2f5307e25fff9b", - "globalAliases": [], - "localAliases": [ - "test" - ], - "permissions": { - "read": true, - "write": true, - "owner": true - } - }, - { - "id": "96470e0df00ec28807138daf01915cfda2bee8eccc91dea9558c0b4855b5bf95", - "globalAliases": [ - "alex" - ], - "localAliases": [], - "permissions": { - "read": true, - "write": true, - "owner": true - } - } - ] -} -``` - -#### DeleteKey `DELETE /v0/key?id=<acces key id>` - -Deletes an API access key. - -#### UpdateKey `POST /v0/key?id=<acces key id>` - -Updates information about the specified API access key. - -Request body format: - -```json -{ - "name": "NameOfMyKey", - "allow": { - "createBucket": true, - }, - "deny": {} -} -``` - -All fields (`name`, `allow` and `deny`) are optionnal. -If they are present, the corresponding modifications are applied to the key, otherwise nothing is changed. -The possible flags in `allow` and `deny` are: `createBucket`. - - -### Bucket operations - -#### ListBuckets `GET /v0/bucket` - -Returns all storage buckets in the cluster. - -Example response: - -```json -[ - { - "id": "70dc3bed7fe83a75e46b66e7ddef7d56e65f3c02f9f80b6749fb97eccb5e1033", - "globalAliases": [ - "test2" - ], - "localAliases": [] - }, - { - "id": "96470e0df00ec28807138daf01915cfda2bee8eccc91dea9558c0b4855b5bf95", - "globalAliases": [ - "alex" - ], - "localAliases": [] - }, - { - "id": "d7452a935e663fc1914f3a5515163a6d3724010ce8dfd9e4743ca8be5974f995", - "globalAliases": [ - "test3" - ], - "localAliases": [] - }, - { - "id": "e6a14cd6a27f48684579ec6b381c078ab11697e6bc8513b72b2f5307e25fff9b", - "globalAliases": [], - "localAliases": [ - { - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "alias": "test" - } - ] - } -] -``` - -#### GetBucketInfo `GET /v0/bucket?id=<bucket id>` -#### GetBucketInfo `GET /v0/bucket?globalAlias=<alias>` - -Returns information about the requested storage bucket. - -If `id` is set, the bucket is looked up using its exact identifier. -If `globalAlias` is set, the bucket is looked up using its global alias. -(both are fast) - -Example response: - -```json -{ - "id": "afa8f0a22b40b1247ccd0affb869b0af5cff980924a20e4b5e0720a44deb8d39", - "globalAliases": [], - "websiteAccess": false, - "websiteConfig": null, - "keys": [ - { - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "name": "Imported key", - "permissions": { - "read": true, - "write": true, - "owner": true - }, - "bucketLocalAliases": [ - "debug" - ] - } - ], - "objects": 14827, - "bytes": 13189855625, - "unfinshedUploads": 0, - "quotas": { - "maxSize": null, - "maxObjects": null - } -} -``` - -#### CreateBucket `POST /v0/bucket` - -Creates a new storage bucket. - -Request body format: - -```json -{ - "globalAlias": "NameOfMyBucket" -} -``` - -OR - -```json -{ - "localAlias": { - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "alias": "NameOfMyBucket", - "allow": { - "read": true, - "write": true, - "owner": false - } - } -} -``` - -OR - -```json -{} -``` - -Creates a new bucket, either with a global alias, a local one, -or no alias at all. - -Technically, you can also specify both `globalAlias` and `localAlias` and that would create -two aliases, but I don't see why you would want to do that. - -#### DeleteBucket `DELETE /v0/bucket?id=<bucket id>` - -Deletes a storage bucket. A bucket cannot be deleted if it is not empty. - -Warning: this will delete all aliases associated with the bucket! - -#### UpdateBucket `PUT /v0/bucket?id=<bucket id>` - -Updates configuration of the given bucket. - -Request body format: - -```json -{ - "websiteAccess": { - "enabled": true, - "indexDocument": "index.html", - "errorDocument": "404.html" - }, - "quotas": { - "maxSize": 19029801, - "maxObjects": null, - } -} -``` - -All fields (`websiteAccess` and `quotas`) are optionnal. -If they are present, the corresponding modifications are applied to the bucket, otherwise nothing is changed. - -In `websiteAccess`: if `enabled` is `true`, `indexDocument` must be specified. -The field `errorDocument` is optional, if no error document is set a generic -error message is displayed when errors happen. Conversely, if `enabled` is -`false`, neither `indexDocument` nor `errorDocument` must be specified. - -In `quotas`: new values of `maxSize` and `maxObjects` must both be specified, or set to `null` -to remove the quotas. An absent value will be considered the same as a `null`. It is not possible -to change only one of the two quotas. - -### Operations on permissions for keys on buckets - -#### BucketAllowKey `POST /v0/bucket/allow` - -Allows a key to do read/write/owner operations on a bucket. - -Request body format: - -```json -{ - "bucketId": "e6a14cd6a27f48684579ec6b381c078ab11697e6bc8513b72b2f5307e25fff9b", - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "permissions": { - "read": true, - "write": true, - "owner": true - }, -} -``` - -Flags in `permissions` which have the value `true` will be activated. -Other flags will remain unchanged. - -#### BucketDenyKey `POST /v0/bucket/deny` - -Denies a key from doing read/write/owner operations on a bucket. - -Request body format: - -```json -{ - "bucketId": "e6a14cd6a27f48684579ec6b381c078ab11697e6bc8513b72b2f5307e25fff9b", - "accessKeyId": "GK31c2f218a2e44f485b94239e", - "permissions": { - "read": false, - "write": false, - "owner": true - }, -} +```bash +curl -H 'Authorization: Bearer s3cr3t' http://localhost:3903/v0/status | jq ``` -Flags in `permissions` which have the value `true` will be deactivated. -Other flags will remain unchanged. - - -### Operations on bucket aliases - -#### GlobalAliasBucket `PUT /v0/bucket/alias/global?id=<bucket id>&alias=<global alias>` - -Empty body. Creates a global alias for a bucket. - -#### GlobalUnaliasBucket `DELETE /v0/bucket/alias/global?id=<bucket id>&alias=<global alias>` - -Removes a global alias for a bucket. - -#### LocalAliasBucket `PUT /v0/bucket/alias/local?id=<bucket id>&accessKeyId=<access key ID>&alias=<local alias>` - -Empty body. Creates a local alias for a bucket in the namespace of a specific access key. - -#### LocalUnaliasBucket `DELETE /v0/bucket/alias/local?id=<bucket id>&accessKeyId<access key ID>&alias=<local alias>` - -Removes a local alias for a bucket in the namespace of a specific access key. - +For more advanced use cases, we recommend using a SDK. +[Go to the "Build your own app" section to know how to use our SDKs](@/documentation/build/_index.md) diff --git a/doc/book/reference-manual/features.md b/doc/book/reference-manual/features.md index 1f21af6e..550504ff 100644 --- a/doc/book/reference-manual/features.md +++ b/doc/book/reference-manual/features.md @@ -83,7 +83,7 @@ This feature is totally invisible to S3 clients and does not break compatibility ### Cluster administration API Garage provides a fully-fledged REST API to administer your cluster programatically. -Functionnality included in the admin API include: setting up and monitoring +Functionality included in the admin API include: setting up and monitoring cluster nodes, managing access credentials, and managing storage buckets and bucket aliases. A full reference of the administration API is available [here](@/documentation/reference-manual/admin-api.md). diff --git a/doc/book/working-documents/_index.md b/doc/book/working-documents/_index.md index 9871d206..8fc170b7 100644 --- a/doc/book/working-documents/_index.md +++ b/doc/book/working-documents/_index.md @@ -1,6 +1,6 @@ +++ title = "Working Documents" -weight = 7 +weight = 8 sort_by = "weight" template = "documentation.html" +++ diff --git a/doc/book/working-documents/design-draft.md b/doc/book/working-documents/design-draft.md index 3c8298b0..6560dbed 100644 --- a/doc/book/working-documents/design-draft.md +++ b/doc/book/working-documents/design-draft.md @@ -1,6 +1,6 @@ +++ title = "Design draft (obsolete)" -weight = 50 +weight = 900 +++ **WARNING: this documentation is a design draft which was written before Garage's actual implementation. diff --git a/doc/book/working-documents/load-balancing.md b/doc/book/working-documents/load-balancing.md index bf6bdd95..1a65fdd2 100644 --- a/doc/book/working-documents/load-balancing.md +++ b/doc/book/working-documents/load-balancing.md @@ -1,6 +1,6 @@ +++ title = "Load balancing data (obsolete)" -weight = 60 +weight = 910 +++ **This is being yet improved in release 0.5. The working document has not been updated yet, it still only applies to Garage 0.2 through 0.4.** diff --git a/doc/book/working-documents/testing-strategy.md b/doc/book/working-documents/testing-strategy.md new file mode 100644 index 00000000..7d6be8ef --- /dev/null +++ b/doc/book/working-documents/testing-strategy.md @@ -0,0 +1,75 @@ ++++ +title = "Testing strategy" +weight = 30 ++++ + + +## Testing Garage + +Currently, we have the following tests: + +- some unit tests spread around the codebase +- integration tests written in Rust (`src/garage/test`) to check that Garage operations perform correctly +- integration test for compatibility with external tools (`script/test-smoke.sh`) + +We have also tried `minio/mint` but it fails a lot and for now we haven't gotten a lot from it. + +In the future: + +1. We'd like to have a systematic way of testing with `minio/mint`, + it would add value to Garage by providing a compatibility score and reference that can be trusted. +2. We'd also like to do testing with Jepsen in some way. + +## How to instrument Garagae + +We should try to test in least invasive ways, i.e. minimize the impact of the testing framework on Garage's source code. This means for example: + +- Not abstracting IO/nondeterminism in the source code +- Not making `garage` a shared library (launch using `execve`, it's perfectly fine) + +Instead, we should focus on building a clean outer interface for the `garage` binary, +for example loading configuration using environnement variables instead of the configuration file if that's helpfull for writing the tests. + +There are two reasons for this: + +- Keep the soure code clean and focused +- Test something that is as close as possible as the true garage that will actually be running + +Reminder: rules of simplicity, concerning changes to Garage's source code. +Always question what we are doing. +Never do anything just because it looks nice or because we "think" it might be usefull at some later point but without knowing precisely why/when. +Only do things that make perfect sense in the context of what we currently know. + +## References + +Testing is a research field on its own. +About testing distributed systems: + + - [Jepsen](https://jepsen.io/) is a testing framework designed to test distributed systems. It can mock some part of the system like the time and the network. + - [FoundationDB Testing Approach](https://www.micahlerner.com/2021/06/12/foundationdb-a-distributed-unbundled-transactional-key-value-store.html#what-is-unique-about-foundationdbs-testing-framework). They chose to abstract "all sources of nondeterminism and communication are abstracted, including network, disk, time, and pseudo random number generator" to be able to run tests by simulating faults. + - [Testing Distributed Systems](https://asatarin.github.io/testing-distributed-systems/) - Curated list of resources on testing distributed systems + +About S3 compatibility: + - [ceph/s3-tests](https://github.com/ceph/s3-tests) + - (deprecated) [minio/s3verify](https://blog.min.io/s3verify-a-simple-tool-to-verify-aws-s3-api-compatibility/) + - [minio/mint](https://github.com/minio/mint) + +About benchmarking S3 (I think it is not necessarily very relevant for this iteration): + - [minio/warp](https://github.com/minio/warp) + - [wasabi-tech/s3-benchmark](https://github.com/wasabi-tech/s3-benchmark) + - [dvassallo/s3-benchmark](https://github.com/dvassallo/s3-benchmark) + - [intel-cloud/cosbench](https://github.com/intel-cloud/cosbench) - used by Ceph + +Engineering blog posts: + - [Quincy @ Scale: A Tale of Three Large-Scale Clusters](https://ceph.io/en/news/blog/2022/three-large-scale-clusters/) + +Interesting blog posts on the blog of the Sled database: + +- <https://sled.rs/simulation.html> +- <https://sled.rs/perf.html> + +Misc: + - [mutagen](https://github.com/llogiq/mutagen) - mutation testing is a way to assert our test quality by mutating the code and see if the mutation makes the tests fail + - [fuzzing](https://rust-fuzz.github.io/book/) - cargo supports fuzzing, it could be a way to test our software reliability in presence of garbage data. + + |