aboutsummaryrefslogtreecommitdiff
path: root/src/model/s3
Commit message (Collapse)AuthorAgeFilesLines
* update Cargo dependenciesupdate-dependencies-1.1Alex Auvolat2024-08-251-2/+3
| | | | | | Some aws-* crates have been pinned because newer versions don't compile with rustc 1.73. These crates are only used for integration testing, no crates which are part of the final Garage version are concerned.
* [next-0.10] fixes to k2v rpc + comment fixesAlex Auvolat2024-03-271-1/+1
|
* [s3-checksum] implement x-amz-checksum-* headersAlex Auvolat2024-03-262-7/+55
|
* [block-ref-repair] Block refcount recalculation and repairAlex Auvolat2024-03-191-0/+39
| | | | | | | | | | - We always recalculate the reference count of a block before deleting it locally, to make sure that it is indeed zero. - If we had to fetch a remote block but we were not able to get it, check that refcount is indeed > 0. - Repair procedure that checks everything
* Merge pull request 'Remove Sled' (#767) from rm-sled into next-0.10Alex2024-03-121-7/+1
|\ | | | | | | Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/767
| * [rm-sled] Remove counted_tree_hackAlex Auvolat2024-03-081-7/+1
| |
* | [rm-migration] Remove migration path from Garage v0.5rm-migrationAlex Auvolat2024-03-082-94/+4
|/
* [sse-c] Remove special case for Content-Type headersse-cAlex Auvolat2024-03-071-3/+17
|
* [sse-c] Implement SSE-C encryptionAlex Auvolat2024-03-072-3/+162
|
* garage_db: refactor transactions and add on_commit mechanismAlex Auvolat2023-09-211-6/+2
|
* lifecycle: fix SkipBucket bugAlex Auvolat2023-09-061-4/+5
|
* lifecycle worker: correct small clippy lintsAlex Auvolat2023-09-041-3/+3
|
* lifecycle worker: don't get stuck on non-existent bucketAlex Auvolat2023-08-311-5/+16
|
* lifecycle worker: fix potential inifinite loopAlex Auvolat2023-08-311-1/+4
|
* lifecycle worker: add log message when startingAlex Auvolat2023-08-311-16/+16
|
* lifecycle worker: use queue_insert and process objects in batchesAlex Auvolat2023-08-311-36/+45
|
* lifecycle worker: mitigate potential bugs + refactoringAlex Auvolat2023-08-311-20/+31
|
* lifecycle worker: take into account disabled rulesAlex Auvolat2023-08-301-1/+5
|
* lifecycle worker: avoid building chrono's serde featureAlex Auvolat2023-08-301-10/+10
|
* lifecycle config: store date as given, try to debugAlex Auvolat2023-08-301-1/+8
|
* lifecycle worker: add loggingAlex Auvolat2023-08-301-0/+8
|
* lifecycle worker: skip entire bucket when no lifecycle config is setAlex Auvolat2023-08-301-6/+22
|
* lifecycle worker: implement main functionalityAlex Auvolat2023-08-301-5/+97
|
* lifecycle: skeleton for lifecycle workerAlex Auvolat2023-08-302-0/+254
|
* multipart uploads: save timestampAlex Auvolat2023-06-132-3/+17
|
* properly delete multipart uploads after completionnlnet-task1Alex Auvolat2023-06-091-20/+34
|
* comments and clippy lint fixesAlex Auvolat2023-06-092-9/+9
|
* fix mpu counter (add missing workers) and report info at appropriate placesAlex Auvolat2023-06-091-1/+1
|
* Adapt S3 API code to use new multipart upload modelsAlex Auvolat2023-06-092-0/+26
| | | | | | | - Create and PutPart - completemultipartupload - upload part copy - list_parts
* New multipart upload table layoutAlex Auvolat2023-06-094-31/+429
|
* cleanupAlex Auvolat2023-01-031-3/+0
|
* Refactor how things are migratedAlex Auvolat2023-01-033-219/+231
|
* Refactor background runner and get rid of job workerAlex Auvolat2022-12-142-4/+0
|
* Proper queueing for delayed inserts, now backed to diskAlex Auvolat2022-12-142-47/+40
|
* Fix clippy lint that says we should implement EqAlex Auvolat2022-09-133-5/+5
|
* Include code from v0.5.1 directly to remove dependenciesAlex Auvolat2022-09-062-2/+2
|
* improve internal item counter mechanisms and implement bucket quotas (#326)Alex2022-06-151-1/+60
| | | | | | | | | | | | | | | | - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* Abstract database behind generic interface and implement alternative drivers ↵Alex2022-06-083-13/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#322) - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* First implementation of K2V (#293)Alex2022-05-104-0/+621
**Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-12 00:40:56 +0000