aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| * Uniformize tracing::* imports (hopefully fixes 32-bit build)fix-32bit-buildAlex Auvolat2022-07-154-4/+3
| |
| * Background task manager (#332)Alex2022-07-0827-737/+2049
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - [x] New background worker trait - [x] Adapt all current workers to use new API - [x] Command to list currently running workers, and whether they are active, idle, or dead - [x] Error reporting - Optimizations - [x] Merkle updater: several items per iteration - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on - scrub: - [x] have only one worker with a channel to start/pause/cancel - [x] automatic scrub - [x] ability to view and change tranquility from CLI - [x] persistence of a few info - [ ] Testing Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/332 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * add delays in k2v test_items_and_indicesAlex Auvolat2022-07-081-0/+6
| |
| * Publish k2v-client crate to crates.io (#337)Alex2022-07-041-2/+7
| | | | | | | | | | | | | | Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/337 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * Fix poll item when item didn't changek2v-poll-fixAlex Auvolat2022-07-042-5/+12
| |
| * Add `poll` command to `k2v-cli` (#335)Alex2022-07-041-0/+28
| | | | | | | | | | | | | | Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/335 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * Fix typoAlex2022-06-291-1/+1
| |
| * Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333)Alex2022-06-2347-0/+8532
| | | | | | | | | | | | | | Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/333 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * improve internal item counter mechanisms and implement bucket quotas (#326)Alex2022-06-1532-347/+962
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * Send CORS headers for all requestsbug/corsQuentin Dufour2022-06-131-9/+2
| |
| * Fix garage_db build on 32-bit systemsfix-buildAlex Auvolat2022-06-093-14/+16
| |
| * Abstract database behind generic interface and implement alternative drivers ↵Alex2022-06-0842-645/+3090
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#322) - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * docs: Add Trafik reverse proxy documentationSimon C2022-06-071-2/+139
| |
| * docs: Fix title levelSimon C2022-06-071-1/+1
| |
| * docs: Add link to facilitate navigation in the documentationSimon C2022-06-071-1/+1
| |
| * Remove useless MC_REGION env variabledoc2Quentin Dufour2022-06-021-10/+0
| |
| * Fix garage_version() now that GIT_VERSION is read in crate garage_rpcv0.7.2.1Alex Auvolat2022-06-021-1/+1
| |
| * Fix Content-Type headers for {admin,k2v} errors and admin responsesfix/admin-api-content-typeAlex Auvolat2022-05-258-43/+30
| | | | | | | | Fix #315
| * Fixes to S3 compatibility page (#314)Alex2022-05-251-29/+48
| | | | | | | | | | | | | | | | | | Mention PostObject is implemented, fix english mistakes Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/314 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * Update Ceph S3 endpoints compatibilityQuentin Dufour2022-05-251-28/+36
| |
| * Shorter page titledoc/admin-apiAlex Auvolat2022-05-241-1/+1
| |
| * Add lost content for Restic with GarageAlex Auvolat2022-05-241-0/+55
| | | | | | | | Suggested-by: Quentin <quentin@deuxfleurs.fr>
| * Add documentation for new Admin API and a few infos on K2VAlex Auvolat2022-05-243-7/+116
| |
| * K2V client improvements (#307)v0.7.2Alex2022-05-246-6/+65
| | | | | | | | | | | | | | | | | | | | | | - [x] Better distinguish error types - [x] Parse error messages received from server - [x] Remove `src/` folder layer, we don't have that for other crates Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/307 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * First version of admin API (#298)Alex2022-05-2465-984/+3251
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Spec:** - [x] Start writing - [x] Specify all layout endpoints - [x] Specify all endpoints for operations on keys - [x] Specify all endpoints for operations on key/bucket permissions - [x] Specify all endpoints for operations on buckets - [x] Specify all endpoints for operations on bucket aliases View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md> **Code:** - [x] Refactor code for admin api to use common api code that was created for K2V **General endpoints:** - [x] Metrics - [x] GetClusterStatus - [x] ConnectClusterNodes - [x] GetClusterLayout - [x] UpdateClusterLayout - [x] ApplyClusterLayout - [x] RevertClusterLayout **Key-related endpoints:** - [x] ListKeys - [x] CreateKey - [x] ImportKey - [x] GetKeyInfo - [x] UpdateKey - [x] DeleteKey **Bucket-related endpoints:** - [x] ListBuckets - [x] CreateBucket - [x] GetBucketInfo - [x] DeleteBucket - [x] PutBucketWebsite - [x] DeleteBucketWebsite **Operations on key/bucket permissions:** - [x] BucketAllowKey - [x] BucketDenyKey **Operations on bucket aliases:** - [x] GlobalAliasBucket - [x] GlobalUnaliasBucket - [x] LocalAliasBucket - [x] LocalUnaliasBucket **And also:** - [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling) - [x] Add management of website access - [ ] Check that nothing is missing wrt what can be done using the CLI - [ ] Improve formatting of the spec - [x] Make sure everyone is cool with the API design Fix #231 Fix #295 Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/298 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * Add a K2V client library and CLI (#303)trinity-1686a2022-05-1814-199/+1721
| | | | | | | | | | | | | | | | | | lib.rs could use getting split in modules, but I'm not sure how exactly Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/303 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
| * K2V: Fix `end` parameter and add tests (fix #305)fix-k2v-305Alex Auvolat2022-05-172-2/+93
| |
| * Make background runner terminate correctlyAlex Auvolat2022-05-171-15/+22
| |
| * Fix ReadIndex spec and add JSON5 remark to docfix-k2v-docAlex Auvolat2022-05-161-7/+44
| |
| * First implementation of K2V (#293)Alex2022-05-1083-1228/+6493
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
| * Update netapp to 0.4.4, fix #300v0.7.1netapp-0.4.4Alex Auvolat2022-05-093-19/+19
| |
| * Fix `layout show` to not show changes when there are no changes (#297)Alex2022-05-092-5/+22
| | | | | | | | | | | | | | | | | | fixes #295, partially Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/297 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* | deleted zone_redundancy from System structMendes2022-10-041-2/+0
| |
* | Correction of a few bugs in the tests, modification of ClusterLayout::checkMendes2022-09-222-77/+137
| |
* | New version of the algorithm that calculate the layout.Mendes2022-09-219-698/+926
| | | | | | | | | | | | | | | | It takes as paramters the replication factor and the zone redundancy, computes the largest partition size reachable with these constraints, and among the possible assignation with this partition size, it computes the one that moves the least number of partitions compared to the previous assignation. This computation uses graph algorithms defined in graph_algo.rs
* | Added the section with description proofs of the parametric assignment ↵Mendes2022-09-104-12/+260
| | | | | | | | computation in the optimal layout report
* | ignore log files in commitMendes2022-09-085-333/+11
| |
* | Added a first draft version of the algorithm and analysis for the non-strict ↵Mendes2022-08-195-49/+120
| | | | | | | | mode.
* | Merge branch 'optimal-layout' of https://git.deuxfleurs.fr/Deuxfleurs/garage ↵Mendes2022-07-191-3/+1
|\ \ | | | | | | | | | into optimal-layout
| * | Correct small formatting issueAlex Auvolat2022-05-051-3/+1
| | |
* | | Added the latex report on the optimal layout algorithmMendes2022-07-1813-0/+12357
|/ /
* | Corrected the warnings and errors issued by cargo clippyMendes2022-05-012-80/+63
| |
* | updated cargo.lockAlex Auvolat2022-05-011-0/+1
| |
* | Apply cargo fmtAlex Auvolat2022-05-012-792/+842
| |
* | Change the way new layout assignations are computed.Alex Auvolat2022-05-014-434/+827
|/ | | | | | The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms. This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>
* fix clippyfix-multipart-fragmentationAlex Auvolat2022-04-191-1/+1
|
* Prettier code for defragmentationAlex Auvolat2022-04-191-104/+102
|
* Defragmentation in UploadPartCopy: first pass (not pretty but it compiles)Alex Auvolat2022-04-191-63/+122
|
* remove size limitation in UploadPartCopy (#280)withinboredom2022-04-191-12/+0
| | | | | | | | | | | This removes the >1mb s3_copy restriction. This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248). Co-authored-by: Rob Landers <landers.robert@gmail.com> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/280 Co-authored-by: withinboredom <landers.robert@gmail.com> Co-committed-by: withinboredom <landers.robert@gmail.com>
* Add documentation on running Kopia with GarageBaptiste Jonglez2022-04-101-0/+21
|
generated by cgit v1.2.3 (git 2.25.1) at 2024-12-19 14:20:31 +0000