aboutsummaryrefslogtreecommitdiff
path: root/src/util/config.rs
Commit message (Collapse)AuthorAgeFilesLines
* Merge tag 'v0.8.5' into sync-08-09Alex Auvolat2024-01-161-143/+12
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Garage v0.8.5 This minor release includes the following improvements and fixes: New features: - Configuration: make LMDB's `map_size` configurable and make `block_size` and `sled_cache_capacity` expressable as strings (such as `10M`) (#628, #630) - Add support for binding to Unix sockets for the S3, K2V, Admin and Web API servers (#640) - Move the `convert_db` command into the main Garage binary (#645) - Add support for specifying RPC secret and admin tokens as environment variables (#643) - Add `allow_world_readable_secrets` option to config file (#663, #685) Bug fixes: - Use `statvfs` instead of mount list to determine free space in metadata/data directories (#611, #631) - Add missing casts to fix 32-bit build (#632) - Fix error when none of the HTTP servers (S3/K2V/Admin/Web) is started and fix shutdown hang (#613, #633) - Add missing CORS headers to PostObject response (#609, #656) - Monitoring: finer histogram boundaries in Prometheus exported metrics (#531, #686) Other: - Documentation improvements (#641)
| * config: refactor secret sourcingAlex Auvolat2024-01-151-213/+7
| |
| * Add allow_world_readable_secrets option to config fileFélix Baylac Jacqué2023-10-261-5/+77
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Sometimes, the secret files permissions checks gets in the way. It's by no mean complete, it doesn't take the Posix ACLs into account among other things. Correctly checking the ACLs would be too involving (see https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/658#issuecomment-7102) and would likely still fail in some weird chmod settings. We're adding a new configuration file key allowing the user to disable this permission check altogether. The (already existing) env variable counterpart always take precedence to this config file option. That's useful in cases where the configuration file is static and cannot be easily altered. Fixes https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/658 Co-authored-by: Florian Klink <flokli@flokli.de>
| * util: move reading secret file into seperate helpernetworkException2023-10-191-16/+19
| | | | | | | | | | | | this patch moves the logic to read a secret file (and check for correct permissions) from `secret_from_file` into a new `read_secret_file` helper.
* | Merge branch 'main' into nextv0.9.0-rc1Alex Auvolat2023-10-031-4/+5
|\|
| * config: allow using paths for unix domain sockets in various placesnetworkException2023-09-291-4/+5
| | | | | | | | | | | | | | this patch updates the config format to also allow paths in bind addresses for unix domain sockets. this has been added to all apis except rpc.
* | Merge branch 'main' into nextAlex Auvolat2023-09-111-6/+59
|\|
| * config: make block_size and sled_cache_capacity expressable as stringsAlex Auvolat2023-09-111-8/+57
| |
| * make lmdb's map_size configurable (fix #628)Alex Auvolat2023-09-111-0/+4
| |
* | block manager: skeleton for multi-hdd supportAlex Auvolat2023-09-061-1/+21
| |
* | Merge branch 'main' into nextAlex Auvolat2023-06-131-3/+22
|\|
| * set default for [consul-services] apiRoberto Hidalgo2023-05-221-0/+1
| |
| * simplify code according to feedbackRoberto Hidalgo2023-05-221-10/+5
| |
| * rename mode to consul_http_apiRoberto Hidalgo2023-05-221-8/+8
| |
| * follow feedback, fold into existing featureRoberto Hidalgo2023-05-221-20/+19
| |
| * allow additional ServiceMeta, docsRoberto Hidalgo2023-05-221-0/+3
| |
| * register consul services against local agent instead of catalog apiRoberto Hidalgo2023-05-221-2/+23
| |
| * *: apply clippy recommendations.Jonathan Davies2023-05-091-1/+1
| |
* | Make fsync an option for meta and dataAlex Auvolat2023-06-091-0/+7
| |
* | Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaultsAlex Auvolat2023-05-171-1/+1
|/
* fixes for pr 499config-files-envAlex Auvolat2023-02-061-4/+4
|
* secrets can be passed directly in config, as file, or as envAlex Auvolat2023-02-031-46/+70
|
* Add tests for `rpc_secret_file`Felix Scheinost2023-01-071-0/+120
|
* Error on both `rpc_secret` and `rpc_secret_file`Felix Scheinost2023-01-071-1/+7
|
* Implement `rpc_secret_file`Felix Scheinost2023-01-041-2/+24
|
* Fix issue with 'http(s)://' prefixconsul-tlsAlex Auvolat2022-10-181-2/+2
|
* Add TLS support for Consul discovery + refactoringAlex Auvolat2022-10-181-10/+35
|
* RPC performance changesAlex Auvolat2022-09-191-0/+5
| | | | | | - configurable ping timeout - single, much higher, configurable RPC timeout - no more concurrency semaphore
* Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353)resolve-peer-namesAlex Auvolat2022-09-141-25/+3
|
* Merge branch 'main' into improve-depsAlex Auvolat2022-09-071-7/+0
|\
| * Ability to dynamically set resync tranquilityAlex Auvolat2022-09-021-7/+0
| |
* | Make all HTTP services optionnalAlex Auvolat2022-09-071-4/+2
|/
* Abstract database behind generic interface and implement alternative drivers ↵Alex2022-06-081-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#322) - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* First version of admin API (#298)Alex2022-05-241-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Spec:** - [x] Start writing - [x] Specify all layout endpoints - [x] Specify all endpoints for operations on keys - [x] Specify all endpoints for operations on key/bucket permissions - [x] Specify all endpoints for operations on buckets - [x] Specify all endpoints for operations on bucket aliases View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md> **Code:** - [x] Refactor code for admin api to use common api code that was created for K2V **General endpoints:** - [x] Metrics - [x] GetClusterStatus - [x] ConnectClusterNodes - [x] GetClusterLayout - [x] UpdateClusterLayout - [x] ApplyClusterLayout - [x] RevertClusterLayout **Key-related endpoints:** - [x] ListKeys - [x] CreateKey - [x] ImportKey - [x] GetKeyInfo - [x] UpdateKey - [x] DeleteKey **Bucket-related endpoints:** - [x] ListBuckets - [x] CreateBucket - [x] GetBucketInfo - [x] DeleteBucket - [x] PutBucketWebsite - [x] DeleteBucketWebsite **Operations on key/bucket permissions:** - [x] BucketAllowKey - [x] BucketDenyKey **Operations on bucket aliases:** - [x] GlobalAliasBucket - [x] GlobalUnaliasBucket - [x] LocalAliasBucket - [x] LocalUnaliasBucket **And also:** - [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling) - [x] Add management of website access - [ ] Check that nothing is missing wrt what can be done using the CLI - [ ] Improve formatting of the spec - [x] Make sure everyone is cool with the API design Fix #231 Fix #295 Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/298 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* First implementation of K2V (#293)Alex2022-05-101-2/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* Make background tranquility a configurable parameterAlex Auvolat2022-03-231-0/+7
|
* Make admin server optionalAlex Auvolat2022-03-141-2/+3
|
* Refactoring: rename config files, make modifications less invasiveAlex Auvolat2022-03-141-3/+3
|
* Add tracing integration with opentelemetryAlex Auvolat2022-03-141-0/+2
|
* Update dependencies and add admin module with metricsmricher2022-03-141-0/+10
| | | | | | | | | | - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet
* add support for kubernetes service discoveryMax Audron2022-03-121-0/+7
| | | | | | | | | | | | | | | | | | | | | This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually.
* Make use of website config, return error document on errorAlex Auvolat2022-01-131-2/+0
|
* Add compression using zstd (#173)trinity-1686a2021-12-151-0/+61
| | | | | | | | | fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
* add support for vhost-style s3 bucketTrinity Pointard2021-11-161-0/+3
|
* allow missing bootstrap_peers in garage.tomlTrinity Pointard2021-11-051-1/+1
|
* Improve CLI, adapt tests, update documentationAlex Auvolat2021-10-251-11/+5
|
* Improvements to CLI and various fixes for netapp versionAlex Auvolat2021-10-221-18/+8
| | | | Discovery via consul, persist peer list to file
* First port of Garage to NetappAlex Auvolat2021-10-221-33/+30
|
* Many improvements on ring/replication and its configuration:Alex Auvolat2021-05-281-28/+18
| | | | | | | | | | | | | | | | | | | | - Explicit "replication_mode" configuration parameters that takes either "none", "2" or "3" as values, instead of letting user configure replication factor themselves. These are presets whose corresponding replication/quorum values can be found in replication/mode.rs - Explicit support for single-node and two-node deployments (number of nodes must be at least "replication_mode", with "none" we can have only one node) - Ring is now stored much more compactly with 256*8 + n*32 bytes, instead of 256*32 bytes - Support for gateway-only nodes that do not store data (these nodes still need a metadata_directory to store the list of bucket and keys since those are stored on all nodes; it also technically needs a data_directory to start but it will stay empty unless we have bugs)
* Tune Sled configurationAlex Auvolat2021-05-031-0/+14
| | | | | | | - Make sled cache size and flush interval configurable - Set less agressive default values: - cache size 128MB instead of 1GB - Flush interval 2 seconds instead of .5 seconds
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-05 06:44:33 +0000