aboutsummaryrefslogtreecommitdiff
path: root/src/db
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'main' into next-0.10Alex Auvolat2024-03-082-43/+157
|\
| * [factor-db-open] Combine logic for opening db enginesfactor-db-openAlex Auvolat2024-03-082-43/+157
| |
| * [rel-0.9.3] Bump version to 0.9.3v0.9.3Alex Auvolat2024-03-041-1/+1
| |
| * [rel-0.9.2] Bump version to v0.9.2v0.9.2rel-0.9.2Alex Auvolat2024-03-011-1/+1
| |
* | Merge branch 'main' into next-0.10Alex Auvolat2024-02-132-12/+49
|\|
| * [dep-upgrade-202402] refactor dependencies: move all as workspace depsAlex Auvolat2024-02-051-12/+7
| |
| * make all garage_db::Engine variants un-conditionalZdenek Crha2024-01-221-21/+16
| | | | | | | | | | | | | | | | | | | | | | Having all Engine enum variants conditional causes compilation errors when *none* of the DB engine features is enabled. This is not an issue for full garage build, but affects crates that use garage_db as dependency. Change all variants to be present at all times. It solves compilation errors and also allows us to better differentiate between invalid DB engine name and engine with support not compiled in current binary.
| * convert_db: prevent conversion between same input/output engineZdenek Crha2024-01-181-0/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use optional DB open overrides for both input and output database. Duplicating the same override flag for input/output would result in too many, too long flags. It would be too costly for very rare edge-case where converting between same DB engine, just with different flags. Because overrides flags for different engines are disjoint and we are preventing conversion between same input/ouput DB engine, we can have only one set. The override flag will be passed either to input or output, based on engine type it belongs to. It will never be passed to both of them and cause unwelcome surprise to user.
| * Bump version to 0.9.1v0.9.1rel-v0.9.1Alex Auvolat2024-01-161-1/+1
| |
* | bump crate versions to 0.10.0v0.10.0-beta1Alex Auvolat2024-01-111-1/+1
|/
* bump version to v0.9.0v0.9.0nextAlex Auvolat2023-10-101-1/+1
|
* Merge branch 'main' into nextAlex Auvolat2023-10-102-78/+0
|\
| * Move convert_db command into main garage binaryconvert-db-main-binaryAlex Auvolat2023-10-102-74/+0
| |
* | garage_db: refactor transactions and add on_commit mechanismAlex Auvolat2023-09-216-58/+63
| |
* | Merge branch 'main' into nextAlex Auvolat2023-09-111-1/+1
|\|
| * update version to 0.8.4v0.8.4hold-netapp-0.5.2Alex Auvolat2023-09-051-1/+1
| |
* | Merge branch 'main' into nextv0.9.0-beta1Alex Auvolat2023-08-291-4/+4
|\|
| * bump version to 0.8.3v0.8.3-rc1Alex Auvolat2023-08-281-1/+1
| |
| * cargo: Updated pretty_env_logger v0.4.0 -> v0.5.0.Jonathan Davies2023-08-281-1/+1
| |
| * garage/db: cargo: Updated rusqlite to 0.29.Jonathan Davies2023-08-281-1/+1
| |
| * don't build sqlite by defaulttrinity-1686a2023-06-261-1/+1
| | | | | | | | | | `bundled-libs` is enabled by default, and causes sqlite to be built too, even if the sqlite backend isn't enabled.
* | Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaultsAlex Auvolat2023-05-173-6/+18
| |
* | Merge branch 'main' into nextAlex Auvolat2023-05-091-1/+1
|\|
| * Fix undefined macro warn! on 32-bitJakub Jirutka2023-05-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Compiling garage_db v0.8.2 (garage-0.8.2/src/db) error: cannot find macro `warn` in this scope --> src/db/lmdb_adapter.rs:352:2 | 352 | warn!("LMDB is not recommended on 32-bit systems, database size will be limited"); | ^^^^ | = help: consider importing this macro: tracing::warn = note: `warn` is in scope, but it is an attribute: `#[warn]` error: could not compile `garage_db` due to previous error
* | Merge branch 'main' into nextAlex Auvolat2023-04-251-5/+5
|\|
| * Prepare for v0.8.2Alex Auvolat2023-03-131-1/+1
| |
| * db/Cargo.toml: Updated rusqlite from 0.27 to 0.28.Jonathan Davies2023-01-261-1/+1
| |
| * Cargo.toml: Loosen tracing dependency to just 0.1.Jonathan Davies2023-01-261-1/+1
| |
| * {db,util}/Cargo.toml: Updated mktemp from 0.4 to 0.5.Jonathan Davies2023-01-261-1/+1
| |
| * Cargo.toml: Updated clap from 3.1.18 to 4.1.Jonathan Davies2023-01-261-1/+1
| |
* | Merge branch 'main' into nextAlex Auvolat2023-01-044-1/+16
|\|
| * Bump everything to v0.8.1Alex Auvolat2023-01-021-1/+1
| |
| * cli: prettier table in garage statsAlex Auvolat2022-12-133-0/+15
| |
* | Merge commit 'ec12d6c' into nextAlex Auvolat2022-12-111-3/+0
|\ \ | |/ |/|
| * Return more info when layout's .check() fails, fix compilation, fix testAlex Auvolat2022-11-081-2/+0
| |
| * Merge branch 'main' into optimal-layoutAlex Auvolat2022-11-071-4/+9
| |\
| * | cargo fmtMendes2022-10-101-1/+0
| | |
| * | Tests written in layout.rsMendes2022-10-101-1/+1
| | | | | | | | | | | | | | | added staged_parameters to ClusterLayout removed the serde(default) -> will need a migration function
* | | put sled as default feature in garage_dbAlex Auvolat2022-11-211-0/+1
| |/ |/|
* | Fix conditional testing in garage_dbAlex Auvolat2022-10-141-4/+9
|/
* Add warnings when features are not included in buildAlex Auvolat2022-09-071-0/+3
|
* Make all DB engines optional build featuresAlex Auvolat2022-09-062-3/+9
|
* Remove Heed default featuresAlex Auvolat2022-09-051-1/+1
|
* Allow linking against system-provided libsqliteJakub Jirutka2022-09-031-1/+2
| | | | | | | | | | | | | | Unfortunately, rusqlite uses the opposite logic for enabling/disabling bundled libraries to others (libsodium-sys, zstd-sys). Cargo features are very limited and doesn't allow to enable feature A in a dependency iff feature B is disabled. Note, lmdb-rkv-sys doesn't need any special treatment because it automatically links against system liblmdb if found via pkgconf. Linux distros should build garage with `--no-default-features --features system-libs` to disable bundled-libs and enable system-libs.
* Uniformize tracing::* imports (hopefully fixes 32-bit build)fix-32bit-buildAlex Auvolat2022-07-153-3/+3
|
* Background task manager (#332)Alex2022-07-082-2/+2
| | | | | | | | | | | | | | | | | | | | | - [x] New background worker trait - [x] Adapt all current workers to use new API - [x] Command to list currently running workers, and whether they are active, idle, or dead - [x] Error reporting - Optimizations - [x] Merkle updater: several items per iteration - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on - scrub: - [x] have only one worker with a channel to start/pause/cancel - [x] automatic scrub - [x] ability to view and change tranquility from CLI - [x] persistence of a few info - [ ] Testing Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/332 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* improve internal item counter mechanisms and implement bucket quotas (#326)Alex2022-06-154-0/+30
| | | | | | | | | | | | | | | | - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
* Fix garage_db build on 32-bit systemsfix-buildAlex Auvolat2022-06-092-8/+15
|
* Abstract database behind generic interface and implement alternative drivers ↵Alex2022-06-088-0/+1834
(#322) - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-14 05:58:33 +0000