aboutsummaryrefslogtreecommitdiff
path: root/src/garage
Commit message (Collapse)AuthorAgeFilesLines
* Abstract database behind generic interface and implement alternative drivers ↵Alex2022-06-085-48/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#322) - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* First version of admin API (#298)Alex2022-05-246-116/+90
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Spec:** - [x] Start writing - [x] Specify all layout endpoints - [x] Specify all endpoints for operations on keys - [x] Specify all endpoints for operations on key/bucket permissions - [x] Specify all endpoints for operations on buckets - [x] Specify all endpoints for operations on bucket aliases View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md> **Code:** - [x] Refactor code for admin api to use common api code that was created for K2V **General endpoints:** - [x] Metrics - [x] GetClusterStatus - [x] ConnectClusterNodes - [x] GetClusterLayout - [x] UpdateClusterLayout - [x] ApplyClusterLayout - [x] RevertClusterLayout **Key-related endpoints:** - [x] ListKeys - [x] CreateKey - [x] ImportKey - [x] GetKeyInfo - [x] UpdateKey - [x] DeleteKey **Bucket-related endpoints:** - [x] ListBuckets - [x] CreateBucket - [x] GetBucketInfo - [x] DeleteBucket - [x] PutBucketWebsite - [x] DeleteBucketWebsite **Operations on key/bucket permissions:** - [x] BucketAllowKey - [x] BucketDenyKey **Operations on bucket aliases:** - [x] GlobalAliasBucket - [x] GlobalUnaliasBucket - [x] LocalAliasBucket - [x] LocalUnaliasBucket **And also:** - [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling) - [x] Add management of website access - [ ] Check that nothing is missing wrt what can be done using the CLI - [ ] Improve formatting of the spec - [x] Make sure everyone is cool with the API design Fix #231 Fix #295 Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/298 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* Add a K2V client library and CLI (#303)trinity-1686a2022-05-183-29/+3
| | | | | | | | | lib.rs could use getting split in modules, but I'm not sure how exactly Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/303 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
* K2V: Fix `end` parameter and add tests (fix #305)fix-k2v-305Alex Auvolat2022-05-171-1/+88
|
* First implementation of K2V (#293)Alex2022-05-1023-65/+1690
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* Fix `layout show` to not show changes when there are no changes (#297)Alex2022-05-092-5/+22
| | | | | | | | | fixes #295, partially Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/297 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* Update to netapp 0.4.2 (a tiny fix)Alex Auvolat2022-04-071-1/+1
|
* Add blocks in errored state to `garage stats`Alex Auvolat2022-03-281-0/+6
|
* Allow `garage layout assign` to assign to several nodes at onceAlex Auvolat2022-03-242-46/+56
|
* Small change to partition assignation algorithmAlex Auvolat2022-03-241-9/+9
| | | | | | This change helps ensure that nodes for each partition are spread over all datacenters, a property that wasn't ensured previously when going from a 2 DC deployment to a 3 DC deployment
* Move block manager to separate moduleAlex Auvolat2022-03-231-2/+1
|
* cleanuptrinity-1686a2022-03-231-19/+4
|
* add test for create bucket and put website with streaming signaturetrinity-1686a2022-03-231-0/+84
|
* add test framework for arbitraty S3 requeststrinity-1686a2022-03-2310-7/+401
| | | | and implement some basic test with it
* Update netapp to v0.4.1update-netappAlex Auvolat2022-03-151-1/+1
|
* Fix some new clippy lintsfix-resyncAlex Auvolat2022-03-144-7/+7
|
* Make admin server optionalAlex Auvolat2022-03-141-6/+12
|
* Bump version to 0.7 because of incompatible NetappAlex Auvolat2022-03-141-8/+8
|
* Refactoring: rename config files, make modifications less invasiveAlex Auvolat2022-03-142-4/+4
|
* Add metrics to API endpointAlex Auvolat2022-03-141-2/+3
|
* Refactor how durations are measuredAlex Auvolat2022-03-141-1/+1
|
* Update to Netapp 0.4 which supports distributed tracingAlex Auvolat2022-03-142-4/+5
|
* Add tracing integration with opentelemetryAlex Auvolat2022-03-144-4/+8
|
* Add many metrics in table/ and rpc/Alex Auvolat2022-03-141-0/+4
|
* Implement basic metrics in tableMaximilien R2022-03-141-1/+1
|
* Update dependencies and add admin module with metricsmricher2022-03-142-1/+13
| | | | | | | | | | - Global dependencies updated in Cargo.lock - New module created in src/admin to host: - the (future) admin REST API - the metric collection - add configuration block No metrics implemented yet
* Functional tests for admin commandstests/port-integrationQuentin Dufour2022-03-072-0/+75
|
* Functional tests for website endpointsQuentin Dufour2022-03-073-1/+343
|
* Functional test for multipart endpointsQuentin Dufour2022-03-071-0/+414
|
* Functional test for ListMultipartUploadsQuentin Dufour2022-03-072-7/+199
|
* Functional tests for bucket endpointsQuentin Dufour2022-03-071-0/+77
|
* Functional tests for object operationsQuentin Dufour2022-03-076-1/+274
|
* Move ListObjects tests to RustQuentin Dufour2022-03-072-0/+433
|
* garage(tests): Remove RNG stuffJill2022-02-113-16/+1
|
* garage(tests): Remove superfluous testJill2022-02-101-30/+0
|
* garage(tests): Add random suffix to created buckets.Jill2022-02-104-9/+31
|
* garage(tests): Add some unsafe-usage doc and tweaksJill2022-02-101-5/+8
|
* Add integration tests to DroneQuentin Dufour2022-02-101-1/+7
|
* tests: Fix garage integration testJill2022-02-102-7/+3
|
* tests: Add garage integration tests (base)Jill2022-02-109-0/+428
|
* Hide deleted key in bucket info (fix #211)Alex Auvolat2022-02-021-0/+3
|
* Allow setting index document and error document on the CLIAlex Auvolat2022-01-132-2/+10
|
* Implement ListMultipartUploads (#171)Quentin2022-01-121-1/+2
| | | | | | | | | | | | | | | | | | | | | | | Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2. It took me some times as I wanted to propose the following things: - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476). - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator) - An IO closure to make unit tests possibles. - Unit tests, to track regressions and document how to interact with the code - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK. Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications. Some points that we might want to discuss: - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation. - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token. - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now). Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/171 Co-authored-by: Quentin <quentin@dufour.io> Co-committed-by: Quentin <quentin@dufour.io>
* Fix trivial bug in CLIAlex Auvolat2022-01-101-2/+6
|
* Small CLI changesAlex Auvolat2022-01-051-5/+8
|
* Implement CreateBucketAlex Auvolat2022-01-051-8/+2
|
* Implement key allow|deny --create-bucketAlex Auvolat2022-01-052-0/+48
|
* Small changes in key model and refactoringnew-bucketsAlex Auvolat2022-01-042-20/+32
|
* New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updatesAlex Auvolat2022-01-044-22/+51
|
* New buckets for 0.6.0: small changesAlex Auvolat2022-01-043-9/+15
| | | | | | | | | | | | - Fix bucket delete - fix merge of bucket creation date - Replace deletable with option in aliases Rationale: if two aliases point to conflicting bucket, resolving by making an arbitrary choice risks making data accessible when it shouldn't be. We'd rather resolve to deleting the alias until someone puts it back.
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-14 21:45:46 +0000