diff options
Diffstat (limited to 'doc')
28 files changed, 774 insertions, 12 deletions
diff --git a/doc/book/cookbook/monitoring.md b/doc/book/cookbook/monitoring.md new file mode 100644 index 00000000..8206f645 --- /dev/null +++ b/doc/book/cookbook/monitoring.md @@ -0,0 +1,306 @@ ++++ +title = "Monitoring Garage" +weight = 40 ++++ + +Garage exposes some internal metrics in the Prometheus data format. +This page explains how to exploit these metrics. + +## Setting up monitoring + +### Enabling the Admin API endpoint + +If you have not already enabled the [administration API endpoint](@/documentation/reference-manual/admin-api.md), do so by adding the following lines to your configuration file: + +```toml +[admin] +api_bind_addr = "0.0.0.0:3903" +``` + +This will allow anyone to scrape Prometheus metrics by fetching +`http://localhost:3093/metrics`. If you want to restrict access +to the exported metrics, set the `metrics_token` configuration value +to a bearer token to be used when fetching the metrics endpoint. + +### Setting up Prometheus and Grafana + +Add a scrape config to your Prometheus daemon to scrape metrics from +all of your nodes: + +```yaml +scrape_configs: + - job_name: 'garage' + static_configs: + - targets: + - 'node1.mycluster:3903' + - 'node2.mycluster:3903' + - 'node3.mycluster:3903' +``` + +If you have set a metrics token in your Garage configuration file, +add the following lines in your Prometheus scrape config: + +```yaml + authorization: + type: Bearer + credentials: 'your metrics token' +``` + +To visualize the scraped data in Grafana, +you can either import our [Grafana dashboard for Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/branch/main/script/telemetry/grafana-garage-dashboard-prometheus.json) +or make your own. +We detail below the list of exposed metrics and their meaning. + + + +## List of exported metrics + + +### Metrics of the API endpoints + +#### `api_admin_request_counter` (counter) + +Counts the number of requests to a given endpoint of the administration API. Example: + +``` +api_admin_request_counter{api_endpoint="Metrics"} 127041 +``` + +#### `api_admin_request_duration` (histogram) + +Evaluates the duration of API calls to the various administration API endpoint. Example: + +``` +api_admin_request_duration_bucket{api_endpoint="Metrics",le="0.5"} 127041 +api_admin_request_duration_sum{api_endpoint="Metrics"} 605.250344830999 +api_admin_request_duration_count{api_endpoint="Metrics"} 127041 +``` + +#### `api_s3_request_counter` (counter) + +Counts the number of requests to a given endpoint of the S3 API. Example: + +``` +api_s3_request_counter{api_endpoint="CreateMultipartUpload"} 1 +``` + +#### `api_s3_error_counter` (counter) + +Counts the number of requests to a given endpoint of the S3 API that returned an error. Example: + +``` +api_s3_error_counter{api_endpoint="GetObject",status_code="404"} 39 +``` + +#### `api_s3_request_duration` (histogram) + +Evaluates the duration of API calls to the various S3 API endpoints. Example: + +``` +api_s3_request_duration_bucket{api_endpoint="CreateMultipartUpload",le="0.5"} 1 +api_s3_request_duration_sum{api_endpoint="CreateMultipartUpload"} 0.046340762 +api_s3_request_duration_count{api_endpoint="CreateMultipartUpload"} 1 +``` + +#### `api_k2v_request_counter` (counter), `api_k2v_error_counter` (counter), `api_k2v_error_duration` (histogram) + +Same as for S3, for the K2V API. + + +### Metrics of the Web endpoint + + +#### `web_request_counter` (counter) + +Number of requests to the web endpoint + +``` +web_request_counter{method="GET"} 80 +``` + +#### `web_request_duration` (histogram) + +Duration of requests to the web endpoint + +``` +web_request_duration_bucket{method="GET",le="0.5"} 80 +web_request_duration_sum{method="GET"} 1.0528433229999998 +web_request_duration_count{method="GET"} 80 +``` + +#### `web_error_counter` (counter) + +Number of requests to the web endpoint resulting in errors + +``` +web_error_counter{method="GET",status_code="404 Not Found"} 64 +``` + + +### Metrics of the data block manager + +#### `block_bytes_read`, `block_bytes_written` (counter) + +Number of bytes read/written to/from disk in the data storage directory. + +``` +block_bytes_read 120586322022 +block_bytes_written 3386618077 +``` + +#### `block_read_duration`, `block_write_duration` (histograms) + +Evaluates the duration of the reading/writing of individual data blocks in the data storage directory. + +``` +block_read_duration_bucket{le="0.5"} 169229 +block_read_duration_sum 2761.6902550310056 +block_read_duration_count 169240 +block_write_duration_bucket{le="0.5"} 3559 +block_write_duration_sum 195.59170078500006 +block_write_duration_count 3571 +``` + +#### `block_delete_counter` (counter) + +Counts the number of data blocks that have been deleted from storage. + +``` +block_delete_counter 122 +``` + +#### `block_resync_counter` (counter), `block_resync_duration` (histogram) + +Counts the number of resync operations the node has executed, and evaluates their duration. + +``` +block_resync_counter 308897 +block_resync_duration_bucket{le="0.5"} 308892 +block_resync_duration_sum 139.64204196100016 +block_resync_duration_count 308897 +``` + +#### `block_resync_queue_length` (gauge) + +The number of block hashes currently queued for a resync. +This is normal to be nonzero for long periods of time. + +``` +block_resync_queue_length 0 +``` + +#### `block_resync_errored_blocks` (gauge) + +The number of block hashes that we were unable to resync last time we tried. +**THIS SHOULD BE ZERO, OR FALL BACK TO ZERO RAPIDLY, IN A HEALTHY CLUSTER.** +Persistent nonzero values indicate that some data is likely to be lost. + +``` +block_resync_errored_blocks 0 +``` + + +### Metrics related to RPCs (remote procedure calls) between nodes + +#### `rpc_netapp_request_counter` (counter) + +Number of RPC requests emitted + +``` +rpc_request_counter{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 176 +``` + +#### `rpc_netapp_error_counter` (counter) + +Number of communication errors (errors in the Netapp library, generally due to disconnected nodes) + +``` +rpc_netapp_error_counter{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 354 +``` + +#### `rpc_timeout_counter` (counter) + +Number of RPC timeouts, should be close to zero in a healthy cluster. + +``` +rpc_timeout_counter{from="<this node>",rpc_endpoint="garage_rpc/membership.rs/SystemRpc",to="<remote node>"} 1 +``` + +#### `rpc_duration` (histogram) + +The duration of internal RPC calls between Garage nodes. + +``` +rpc_duration_bucket{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>",le="0.5"} 166 +rpc_duration_sum{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 35.172253716 +rpc_duration_count{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 174 +``` + + +### Metrics of the metadata table manager + +#### `table_gc_todo_queue_length` (gauge) + +Table garbage collector TODO queue length + +``` +table_gc_todo_queue_length{table_name="block_ref"} 0 +``` + +#### `table_get_request_counter` (counter), `table_get_request_duration` (histogram) + +Number of get/get_range requests internally made on each table, and their duration. + +``` +table_get_request_counter{table_name="bucket_alias"} 315 +table_get_request_duration_bucket{table_name="bucket_alias",le="0.5"} 315 +table_get_request_duration_sum{table_name="bucket_alias"} 0.048509778000000024 +table_get_request_duration_count{table_name="bucket_alias"} 315 +``` + + +#### `table_put_request_counter` (counter), `table_put_request_duration` (histogram) + +Number of insert/insert_many requests internally made on this table, and their duration + +``` +table_put_request_counter{table_name="block_ref"} 677 +table_put_request_duration_bucket{table_name="block_ref",le="0.5"} 677 +table_put_request_duration_sum{table_name="block_ref"} 61.617528636 +table_put_request_duration_count{table_name="block_ref"} 677 +``` + +#### `table_internal_delete_counter` (counter) + +Number of value deletions in the tree (due to GC or repartitioning) + +``` +table_internal_delete_counter{table_name="block_ref"} 2296 +``` + +#### `table_internal_update_counter` (counter) + +Number of value updates where the value actually changes (includes creation of new key and update of existing key) + +``` +table_internal_update_counter{table_name="block_ref"} 5996 +``` + +#### `table_merkle_updater_todo_queue_length` (gauge) + +Merkle tree updater TODO queue length (should fall to zero rapidly) + +``` +table_merkle_updater_todo_queue_length{table_name="block_ref"} 0 +``` + +#### `table_sync_items_received`, `table_sync_items_sent` (counters) + +Number of data items sent to/recieved from other nodes during resync procedures + +``` +table_sync_items_received{from="<remote node>",table_name="bucket_v2"} 3 +table_sync_items_sent{table_name="block_ref",to="<remote node>"} 2 +``` + + diff --git a/doc/book/cookbook/real-world.md b/doc/book/cookbook/real-world.md index 4fcb5cf7..5423bbab 100644 --- a/doc/book/cookbook/real-world.md +++ b/doc/book/cookbook/real-world.md @@ -11,8 +11,9 @@ We recommend first following the [quick start guide](@/documentation/quick-start to get familiar with Garage's command line and usage patterns. +## Preparing your environment -## Prerequisites +### Prerequisites To run a real-world deployment, make sure the following conditions are met: @@ -21,10 +22,6 @@ To run a real-world deployment, make sure the following conditions are met: - Each machine has a public IP address which is reachable by other machines. Running behind a NAT is likely to be possible but hasn't been tested for the latest version (TODO). -- Ideally, each machine should have a SSD available in addition to the HDD you are dedicating - to Garage. This will allow for faster access to metadata and has the potential - to significantly reduce Garage's response times. - - This guide will assume you are using Docker containers to deploy Garage on each node. Garage can also be run independently, for instance as a [Systemd service](@/documentation/cookbook/systemd.md). You can also use an orchestrator such as Nomad or Kubernetes to automatically manage @@ -49,6 +46,42 @@ available in the different locations of your cluster is roughly the same. For instance, here, the Mercury node could be moved to Brussels; this would allow the cluster to store 2 TB of data in total. +### Best practices + +- If you have fast dedicated networking between all your nodes, and are planing to store + very large files, bump the `block_size` configuration parameter to 10 MB + (`block_size = 10485760`). + +- Garage stores its files in two locations: it uses a metadata directory to store frequently-accessed + small metadata items, and a data directory to store data blocks of uploaded objects. + Ideally, the metadata directory would be stored on an SSD (smaller but faster), + and the data directory would be stored on an HDD (larger but slower). + +- For the data directory, Garage already does checksumming and integrity verification, + so there is no need to use a filesystem such as BTRFS or ZFS that does it. + We recommend using XFS for the data partition, as it has the best performance. + EXT4 is not recommended as it has more strict limitations on the number of inodes, + which might cause issues with Garage when large numbers of objects are stored. + +- If you only have an HDD and no SSD, it's fine to put your metadata alongside the data + on the same drive. Having lots of RAM for your kernel to cache the metadata will + help a lot with performance. Make sure to use the LMDB database engine, + instead of Sled, which suffers from quite bad performance degradation on HDDs. + Sled is still the default for legacy reasons, but is not recommended anymore. + +- For the metadata storage, Garage does not do checksumming and integrity + verification on its own. If you are afraid of bitrot/data corruption, + put your metadata directory on a BTRFS partition. Otherwise, just use regular + EXT4 or XFS. + +- Having a single server with several storage drives is currently not very well + supported in Garage ([#218](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/218)). + For an easy setup, just put all your drives in a RAID0 or a ZFS RAIDZ array. + If you're adventurous, you can try to format each of your disk as + a separate XFS partition, and then run one `garage` daemon per disk drive, + or use something like [`mergerfs`](https://github.com/trapexit/mergerfs) to merge + all your disks in a single union filesystem that spreads load over them. + ## Get a Docker image Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated). @@ -76,11 +109,12 @@ especially you must consider the following folders/files: this folder will be your main data storage and must be on a large storage (e.g. large HDD) -A valid `/etc/garage/garage.toml` for our cluster would look as follows: +A valid `/etc/garage.toml` for our cluster would look as follows: ```toml metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" +db_engine = "lmdb" replication_mode = "3" @@ -90,8 +124,6 @@ rpc_bind_addr = "[::]:3901" rpc_public_addr = "<this node's public IP>:3901" rpc_secret = "<RPC secret>" -bootstrap_peers = [] - [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" @@ -132,6 +164,21 @@ It should be restarted automatically at each reboot. Please note that we use host networking as otherwise Docker containers can not communicate with IPv6. +If you want to use `docker-compose`, you may use the following `docker-compose.yml` file as a reference: + +```yaml +version: "3" +services: + garage: + image: dxflrs/garage:v0.8.0 + network_mode: "host" + restart: unless-stopped + volumes: + - /etc/garage.toml:/etc/garage.toml + - /var/lib/garage/meta:/var/lib/garage/meta + - /var/lib/garage/data:/var/lib/garage/data +``` + Upgrading between Garage versions should be supported transparently, but please check the relase notes before doing so! To upgrade, simply stop and remove this container and diff --git a/doc/book/cookbook/recovering.md b/doc/book/cookbook/recovering.md index 2424558c..2129a7f3 100644 --- a/doc/book/cookbook/recovering.md +++ b/doc/book/cookbook/recovering.md @@ -1,6 +1,6 @@ +++ title = "Recovering from failures" -weight = 35 +weight = 50 +++ Garage is meant to work on old, second-hand hardware. diff --git a/doc/book/cookbook/upgrading.md b/doc/book/cookbook/upgrading.md index f7659921..9f2ba73b 100644 --- a/doc/book/cookbook/upgrading.md +++ b/doc/book/cookbook/upgrading.md @@ -1,6 +1,6 @@ +++ title = "Upgrading Garage" -weight = 40 +weight = 60 +++ Garage is a stateful clustered application, where all nodes are communicating together and share data structures. diff --git a/doc/book/quick-start/_index.md b/doc/book/quick-start/_index.md index 03c542f9..ac55d2f7 100644 --- a/doc/book/quick-start/_index.md +++ b/doc/book/quick-start/_index.md @@ -54,6 +54,7 @@ to generate unique and private secrets for security reasons: cat > garage.toml <<EOF metadata_dir = "/tmp/meta" data_dir = "/tmp/data" +db_engine = "lmdb" replication_mode = "none" @@ -61,8 +62,6 @@ rpc_bind_addr = "[::]:3901" rpc_public_addr = "127.0.0.1:3901" rpc_secret = "$(openssl rand -hex 32)" -bootstrap_peers = [] - [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/.gitignore b/doc/talks/2022-11-19-Capitole-du-Libre/.gitignore new file mode 100644 index 00000000..b1e713dd --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/.gitignore @@ -0,0 +1,10 @@ +*.aux +*.bbl +*.blg +*.log +*.nav +*.out +*.snm +*.synctex.gz +*.toc +*.dvi diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/Makefile b/doc/talks/2022-11-19-Capitole-du-Libre/Makefile new file mode 100644 index 00000000..c5039140 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/Makefile @@ -0,0 +1,8 @@ +all: + pdflatex présentation.tex + +clean: + rm -f *.aux *.bbl *.blg *.log *.nav *.out *.snm *.synctex.gz *.toc *.dvi présentation.pdf + +clean_sauf_pdf: + rm -f *.aux *.bbl *.blg *.log *.nav *.out *.snm *.synctex.gz *.toc *.dvi diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/NGI.png b/doc/talks/2022-11-19-Capitole-du-Libre/NGI.png Binary files differnew file mode 100644 index 00000000..39031afc --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/NGI.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/agpl-v3-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/agpl-v3-logo.png Binary files differnew file mode 100644 index 00000000..83e1263a --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/agpl-v3-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/carte-Europe.pdf b/doc/talks/2022-11-19-Capitole-du-Libre/carte-Europe.pdf Binary files differnew file mode 100644 index 00000000..89ca67e7 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/carte-Europe.pdf diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/deuxfleurs-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/deuxfleurs-logo.png Binary files differnew file mode 100644 index 00000000..4a5e0865 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/deuxfleurs-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/drapeau_européen.png b/doc/talks/2022-11-19-Capitole-du-Libre/drapeau_européen.png Binary files differnew file mode 100644 index 00000000..dfb05834 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/drapeau_européen.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/garage-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/garage-logo.png Binary files differnew file mode 100644 index 00000000..6ae568d7 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/garage-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/logo_chatons.png b/doc/talks/2022-11-19-Capitole-du-Libre/logo_chatons.png Binary files differnew file mode 100644 index 00000000..890cf17e --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/logo_chatons.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/mastodon-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/mastodon-logo.png Binary files differnew file mode 100644 index 00000000..fbb35d75 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/mastodon-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/matrix-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/matrix-logo.png Binary files differnew file mode 100644 index 00000000..b62de22b --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/matrix-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/neptune.jpg b/doc/talks/2022-11-19-Capitole-du-Libre/neptune.jpg Binary files differnew file mode 100644 index 00000000..e59f0bfa --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/neptune.jpg diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/nextcloud-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/nextcloud-logo.png Binary files differnew file mode 100644 index 00000000..f5a480c5 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/nextcloud-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/peertube-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/peertube-logo.png Binary files differnew file mode 100644 index 00000000..eba8d79f --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/peertube-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/présentation.pdf b/doc/talks/2022-11-19-Capitole-du-Libre/présentation.pdf Binary files differnew file mode 100644 index 00000000..ea6f8287 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/présentation.pdf diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/présentation.tex b/doc/talks/2022-11-19-Capitole-du-Libre/présentation.tex new file mode 100644 index 00000000..a69df8a6 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/présentation.tex @@ -0,0 +1,340 @@ +\documentclass[11pt, aspectratio=1610]{beamer} +\usetheme{Warsaw} +\usepackage[utf8]{inputenc} +\usepackage[french]{babel} +\usepackage{amsmath} +\usepackage{amsfonts} +\usepackage{amssymb} +\usepackage{tikz} +\usepackage{graphicx} +\usepackage{xcolor} +\usepackage{setspace} +\usepackage{todonotes} +\presetkeys{todonotes}{inline}{} +\renewcommand{\baselinestretch}{1.25} + +\definecolor{orange_garage}{RGB}{255,147,41} +\definecolor{gris_garage}{RGB}{78,78,78} + +\author[Association Deuxfleurs]{~\linebreak Vincent Giraud} +\title[De l'auto-hébergement à l'entre-hébergement avec Garage]{De l'auto-hébergement à l'entre-hébergement :\\Garage, pour conserver ses données ensemble} +%\setbeamercovered{transparent} +%\setbeamertemplate{navigation symbols}{} +\date{Capitole du Libre 2022\linebreak + +\scriptsize Samedi 19 novembre 2022\linebreak +} + +\setbeamercolor{palette primary}{fg=gris_garage,bg=orange_garage} +\setbeamercolor{palette secondary}{fg=gris_garage,bg=gris_garage} +\setbeamercolor{palette tiertary}{fg=white,bg=gris_garage} +\setbeamercolor{palette quaternary}{fg=white,bg=gris_garage} +\setbeamercolor{navigation symbols}{fg=black, bg=white} +\setbeamercolor{navigation symbols dimmed}{fg=darkgray, bg=white} +\setbeamercolor{itemize item}{fg=gris_garage} +\setbeamertemplate{itemize item}[circle] + +\addtobeamertemplate{navigation symbols}{}{% + \usebeamerfont{footline}% + \usebeamercolor[fg]{footline}% + \hspace{1em}% + \insertframenumber/\inserttotalframenumber +} + +\setbeamertemplate{headline} +{% + \leavevmode% + \begin{beamercolorbox}[wd=.5\paperwidth,ht=2.5ex,dp=1.125ex]{section in head/foot}% + \hbox to .5\paperwidth{\hfil\insertsectionhead\hfil} + \end{beamercolorbox}% + \begin{beamercolorbox}[wd=.5\paperwidth,ht=2.5ex,dp=1.125ex]{subsection in head/foot}% + \hbox to .5\paperwidth{\hfil\insertsubsectionhead\hfil} + \end{beamercolorbox}% +} +\addtobeamertemplate{footnote}{}{\vspace{2ex}} + +\begin{document} +\begin{frame} +\titlepage +\end{frame} + +\section{Introduction} +\subsection{Présentation} +\begin{frame} +\begin{columns} +\column{0.5 \linewidth} +\begin{center} +\includegraphics[width=3.5cm]{deuxfleurs-logo.png}\linebreak + +\texttt{https://deuxfleurs.fr} +\end{center} +\column{0.4 \linewidth} +\begin{center} +Deuxfleurs est une association militant en faveur d'un internet plus convivial, avec une organisation et des rapports de force repensés.\linebreak + +Nous faisons partie du CHATONS\footnote[frame]{Collectif des Hébergeurs Alternatifs, Transparents, Ouverts, Neutres et Solidaires} depuis avril 2022. + +\includegraphics[width=2cm]{logo_chatons.png} +\end{center} +\end{columns} +\end{frame} + +\subsection{Héberger à la maison} +\begin{frame} +\begin{columns} +\begin{column}{0.5 \linewidth} +\begin{center} +Pour échapper au contrôle et au giron des opérateurs de clouds, héberger ses données à la maison présente de nombreux avantages... +\end{center} + +\vspace{0.5cm} + +\begin{itemize}[<+(1)->] +\item On récupère la souveraineté sur ses données +\item On gagne en vie privée +\item On gagne en libertés +\item On est responsabilisé face à ses besoins +\end{itemize} +\end{column} +\vrule{} +\begin{column}{0.5 \linewidth} +\begin{center} +\onslide<6->{... mais aussi bien des contraintes...} +\end{center} + +\vspace{0.5cm} + +\begin{itemize}[<+(2)->] +\item On repose sur une connexion internet pour particulier +\item Un certain savoir-faire et moultes compétences sont requis +\item Assurer la résilience de ses services est difficile +\item Bien sauvegarder ses données, et ceci au-delà de son site géographique, n'est pas évident +\end{itemize} +\end{column} +\end{columns} +\end{frame} + +\subsection{Sauvegarder pour se parer à tout imprévu} +\begin{frame} +\begin{center} +Sauvegarder pour se parer contre les pannes matérielles est une chose... + +Sauvegarder pour se parer contre les cambriolages et les incendies en est une autre !\linebreak + +\vspace{1cm} +\onslide<2->{Répartir géographiquement ses données devient alors nécessaire.} +\end{center} +\end{frame} + +\section{Les solutions à explorer} +\subsection{L'entre-hébergement} +\begin{frame} +\begin{center} +On a vu récemment se développer au sein du CHATONS la notion d'entre-hébergement : en plus de renforcer l'intégrité des sauvegardes, on va améliorer la disponibilité pendant les coupures de liaison internet, de courant, ou pendant les déménagements d'administrateurs par exemple.\linebreak + +\vspace{1cm} +\onslide<2-> +{ +Dans le cadre du collectif, il s'agit de partager ses volumes de données entre hébergeurs.\linebreak + +Pour assurer la confidentialité, on peut chiffrer les données au niveau applicatif. +} +\end{center} +\end{frame} + +\subsection{S3 contre les systèmes de fichiers} +\begin{frame} +\begin{center} +Dans le cadre de l'administration de services en ligne, les systèmes de fichiers recèlent certaines difficultés.\linebreak + +\vspace{1cm} +Le standard S3 apporte des facilités; on réduit le stockage à un paradigme de clé-valeur basé essentiellement sur deux opérations seulement: lire ou écrire une clé. +\end{center} +\end{frame} + +\section{Garage} +\subsection{Présentation} +\begin{frame} +\begin{columns} +\column{0.5 \linewidth} +\begin{center} +Garage essaye de répondre à l'ensemble de ces besoins.\linebreak + +\vspace{0.5cm} +Il s'agit d'un logiciel libre permettant de distribuer un service S3 sur diverses machines éloignées. +\end{center} +\column{0.5 \linewidth} +\begin{center} +\includegraphics[width=4cm]{garage-logo.png}\linebreak + +\texttt{https://garagehq.deuxfleurs.fr/} +\end{center} +\end{columns} +\end{frame} + +\subsection{Gestion des zones} +\begin{frame} +\begin{center} +Garage va prendre en compte les zones géographiques au moment de répliquer les données.\linebreak + +\vspace{1cm} +\includegraphics[width=13.25cm]{zones.png} +\end{center} +\end{frame} + +\subsection{Comment ça marche ?} +\begin{frame} +\begin{columns} +\column{0.5 \linewidth} +\input{schéma europe} +\column{0.5 \linewidth} +\begin{center} +Chaque objet est dupliqué sur plusieurs zones différentes.\linebreak + +\onslide<5->{Lorsqu'un nouvel hébergeur rejoint le réseau, la charge se voit équilibrée.}\linebreak + +\onslide<12->{Si une zone devient indisponible, les autres continuent d'assurer le service.}\linebreak +\end{center} +\end{columns} +\end{frame} + +\subsection{Financement} +\begin{frame} +\begin{center} +Dans le cadre du programme \textit{Horizon 2021} de l'Union Européenne, nous avons reçu une subvention de la part de l'initiative NGI Pointer\footnote[frame]{Next Generation Internet Program for Open Internet Renovation}.\linebreak + +\includegraphics[width=3cm]{drapeau_européen.png}\hspace{1cm} +\includegraphics[width=3cm]{NGI.png}\linebreak + +Nous avons ainsi pu financer le développement de Garage pendant 1 an. +\end{center} +\end{frame} + +\subsection{Licence} +\begin{frame} +\begin{center} +De par nos valeurs, nous avons attribué la licence AGPL version 3 à Garage, notamment afin qu'il reste parmi les biens communs.\linebreak + +\vspace{0.5cm} +\includegraphics[width=5cm]{agpl-v3-logo.png}\linebreak +\end{center} +\end{frame} + +\subsection{Langage utilisé} +\begin{frame} +\begin{center} +Nous avons décidé d'écrire Garage à l'aide du langage Rust, afin d'obtenir une compilation vers des binaires natifs et efficaces.\linebreak + +\includegraphics[width=3.5cm]{rust-logo.png}\linebreak + +Ce choix permet également de bénéficier des avantages reconnus de Rust en termes de sécurité. +\end{center} +\end{frame} + +\subsection{Matériel requis} +\begin{frame} +\begin{center} +Garage peut ainsi être performant sur des machines limitées. Les prérequis sont minimes : n'importe quelle machine avec un processeur qui a moins d'une décennie, 1~gigaoctet de mémoire vive, et 16~gigaoctets de stockage suffit.\linebreak + +\vspace{1cm} + +Cet aspect est déterminant : il permet en effet d'héberger sur du matériel acheté d'occasion, pour réduire l'impact écologique de nos infrastructures. +\end{center} +\end{frame} + +\subsection{Performances} +\begin{frame} +\begin{center} +\includegraphics[width=13.25cm]{rpc-amplification.png} +\end{center} +\end{frame} + +\begin{frame} +\begin{center} +\includegraphics[width=11cm]{rpc-complexity.png} +\end{center} +\end{frame} + +\subsection{Services} +\begin{frame} +\begin{center} +Puisqu'il suit le standard S3, beaucoup de services populaires sont par conséquence compatibles avec Garage :\linebreak + +\begin{columns} +\column{0.2 \linewidth} +\begin{center} +\includegraphics[width=2.5cm]{nextcloud-logo.png} +\end{center} +\column{0.2 \linewidth} +\begin{center} +\includegraphics[width=2.5cm]{peertube-logo.png} +\end{center} +\column{0.2 \linewidth} +\begin{center} +\includegraphics[width=2.5cm]{matrix-logo.png} +\end{center} +\column{0.2 \linewidth} +\begin{center} +\includegraphics[width=2.5cm]{mastodon-logo.png} +\end{center} +\end{columns} +~\linebreak + +Et comme souvent avec S3, on peut assimiler un bucket à un site, et utiliser le serveur pour héberger des sites web statiques. +\end{center} +\end{frame} + +\section{Intégration chez Deuxfleurs} +\subsection{Matériel} +\begin{frame} +\begin{center} +\includegraphics[width=13cm]{neptune.jpg}\linebreak + +En pratique, nos serveurs ne sont effectivement que des machines achetées d'occasion (très souvent des anciens ordinateurs destinés à la bureautique en entreprise). +\end{center} +\end{frame} + +\subsection{Environnement logiciel} +\begin{frame} +\begin{center} +Pour faciliter la reproduction d'un environnement connu, NixOS est installé sur nos machines.\linebreak + +\vspace{1cm} +Pour s’accommoder des réseaux qu'on trouve derrière des routeurs pour particuliers, on s'aide de notre logiciel Diplonat\footnote[frame]{\texttt{https://git.deuxfleurs.fr/Deuxfleurs/diplonat}}. +\end{center} +\end{frame} + +\section{Au-delà...} +\subsection{... de Deuxfleurs} +\begin{frame} +\begin{center} +\includegraphics[width=10cm]{tedomum.png} +\end{center} +\end{frame} + +\subsection{... de Garage} +\begin{frame} +\begin{center} +Nous avons récemment lancé le développement d'Aérogramme\footnote[frame]{\texttt{https://git.deuxfleurs.fr/Deuxfleurs/aerogramme}}.\linebreak + +\vspace{1cm} +Il s'agit d'un serveur de stockage de courriels chiffrés.\linebreak + +\vspace{1cm} +Il est conçu pour pouvoir travailler avec Garage. +\end{center} +\end{frame} + +\section{Fin} +\subsection{Contacts} +\begin{frame} +\begin{center} +\begin{tikzpicture} +\node (ronce) {\includegraphics[width=0.95\textwidth]{ronce.jpg}}; +\node[white] at (-5.1,3.6) {Intéressé(e) ?}; +\node[white, align=center] at (4.2,-2.6) {Contactez-nous !\\\texttt{coucou@deuxfleurs.fr}\\\texttt{\#forum:deuxfleurs.fr}}; +\end{tikzpicture} +\end{center} +\end{frame} +\end{document} + diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/ronce.jpg b/doc/talks/2022-11-19-Capitole-du-Libre/ronce.jpg Binary files differnew file mode 100644 index 00000000..6e0ed207 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/ronce.jpg diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/rpc-amplification.png b/doc/talks/2022-11-19-Capitole-du-Libre/rpc-amplification.png Binary files differnew file mode 100644 index 00000000..741539a7 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/rpc-amplification.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/rpc-complexity.png b/doc/talks/2022-11-19-Capitole-du-Libre/rpc-complexity.png Binary files differnew file mode 100644 index 00000000..a5cf6316 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/rpc-complexity.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/rust-logo.png b/doc/talks/2022-11-19-Capitole-du-Libre/rust-logo.png Binary files differnew file mode 100644 index 00000000..d18bfeee --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/rust-logo.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/schéma europe.tex b/doc/talks/2022-11-19-Capitole-du-Libre/schéma europe.tex new file mode 100644 index 00000000..02e1c622 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/schéma europe.tex @@ -0,0 +1,52 @@ +\begin{tikzpicture} +\node (carte) {\includegraphics[width=\textwidth]{carte-Europe.pdf}}; + +% \personnage{position X}{position Y}{facteur d'échelle} +\newcommand{\personnage}[4] +{ +\fill[#4] ({#1-(0.4 * #3)},{#2-(0.9 * #3)}) .. controls ({#1-(0.4 * #3)},#2) and ({#1+(0.4 * #3)},#2) .. ({#1+(0.4 * #3)},{#2-(0.9 * #3)}) -- ({#1-(0.4 * #3)},{#2-(0.9 * #3)}); +\fill[#4] (#1,#2) circle ({0.25 * #3}); +} + +\onslide<1-11>{\personnage{-2.25}{-0.75}{0.75}{green}} +\onslide<1-11>{\draw (-1.9,-1.6) rectangle ++(1,1.2);} +\onslide<2-11>{\draw[fill=green] (-1.8,-1.525) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 1};} +\onslide<4-5>{\draw[fill=red] (-1.8,-1.15) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 3};} +\onslide<7-11>{\draw[fill=yellow] (-1.8,-1.15) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 4};} +\onslide<9-11>{\draw[fill=red] (-1.8,-0.775) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 6};} +\onslide<3-11>{\draw[fill=blue] (-1.35,-1.525) rectangle ++(0.35,0.3) node[pos=0.5, white] {\tiny 2};} +\onslide<8-11>{\draw[fill=blue] (-1.35,-1.15) rectangle ++(0.35,0.3) node[pos=0.5, white] {\tiny 5};} +\onslide<11-11>{\draw[fill=yellow] (-1.35,-0.775) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 8};} + +\personnage{1.65}{1.5}{0.75}{blue} +\draw (0.3,0.7) rectangle ++(1,1.2); +\onslide<2->{\draw[fill=green] (0.4,0.775) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 1};} +\onslide<4->{\draw[fill=red] (0.4,1.15) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 3};} +\onslide<10->{\draw[fill=green] (0.4,1.525) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 7};} +\onslide<3->{\draw[fill=blue] (0.85,0.775) rectangle ++(0.35,0.3) node[pos=0.5, white] {\tiny 2};} +\onslide<9->{\draw[fill=red] (0.85,1.15) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 6};} +\onslide<11->{\draw[fill=yellow] (0.85,1.525) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 8};} + +\personnage{1.85}{-2.3}{0.75}{red} +\draw (0.5,-3.15) rectangle ++(1,1.2); +\onslide<2->{\draw[fill=green] (0.6,-3.075) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 1};} +\onslide<4-5>{\draw[fill=red] (0.6,-2.7) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 3};} +\onslide<7->{\draw[fill=yellow] (0.6,-2.7) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 4};} +\onslide<9->{\draw[fill=red] (0.6,-2.325) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 6};} +\onslide<3-5>{\draw[fill=blue] (1.05,-3.075) rectangle ++(0.35,0.3) node[pos=0.5, white] {\tiny 2};} +\onslide<6->{\draw[fill=red] (1.05,-3.075) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 3};} +\onslide<8->{\draw[fill=blue] (1.05,-2.7) rectangle ++(0.35,0.3) node[pos=0.5, white] {\tiny 5};} +\onslide<10->{\draw[fill=green] (1.05,-2.325) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 7};} + +\onslide<5->{\personnage{1.05}{-0.15}{0.75}{yellow}} +\onslide<5->{\draw (-0.35,-1) rectangle ++(1,1.2);} +\onslide<6->{\draw[fill=blue] (-0.25,-0.925) rectangle ++(0.35,0.3) node[pos=0.5, white] {\tiny 2};} +\onslide<7->{\draw[fill=yellow] (-0.25,-0.55) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 4};} +\onslide<10->{\draw[fill=green] (-0.25,-0.175) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 7};} +\onslide<6->{\draw[fill=red] (0.2,-0.925) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 3};} +\onslide<8->{\draw[fill=blue] (0.2,-0.55) rectangle ++(0.35,0.3) node[pos=0.5,white] {\tiny 5};} +\onslide<11->{\draw[fill=yellow] (0.2,-0.175) rectangle ++(0.35,0.3) node[pos=0.5] {\tiny 8};} + +\onslide<12->{\draw[line width=0.25cm] (-2.15,-0.5) -- ++(1,-1);} +\onslide<12->{\draw[line width=0.25cm] (-2.15,-1.5) -- ++(1,1);} +\end{tikzpicture}
\ No newline at end of file diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/tedomum.png b/doc/talks/2022-11-19-Capitole-du-Libre/tedomum.png Binary files differnew file mode 100644 index 00000000..606c6fac --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/tedomum.png diff --git a/doc/talks/2022-11-19-Capitole-du-Libre/zones.png b/doc/talks/2022-11-19-Capitole-du-Libre/zones.png Binary files differnew file mode 100644 index 00000000..d492dd29 --- /dev/null +++ b/doc/talks/2022-11-19-Capitole-du-Libre/zones.png |
