From 1bbe3f9c583dd9098fa917e2d1e4706644b1d81f Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Thu, 9 Jun 2022 18:00:14 +0200 Subject: IPFS article: preliminary+vanilla IPFS --- content/blog/2022-ipfs/dog.jpg | Bin 0 -> 429888 bytes content/blog/2022-ipfs/explorer.png | Bin 0 -> 96061 bytes content/blog/2022-ipfs/idle.png | Bin 0 -> 28505 bytes content/blog/2022-ipfs/index.md | 102 ++++++++++++++++++++++++++++++++++++ 4 files changed, 102 insertions(+) create mode 100644 content/blog/2022-ipfs/dog.jpg create mode 100644 content/blog/2022-ipfs/explorer.png create mode 100644 content/blog/2022-ipfs/idle.png create mode 100644 content/blog/2022-ipfs/index.md diff --git a/content/blog/2022-ipfs/dog.jpg b/content/blog/2022-ipfs/dog.jpg new file mode 100644 index 0000000..704837f Binary files /dev/null and b/content/blog/2022-ipfs/dog.jpg differ diff --git a/content/blog/2022-ipfs/explorer.png b/content/blog/2022-ipfs/explorer.png new file mode 100644 index 0000000..4639593 Binary files /dev/null and b/content/blog/2022-ipfs/explorer.png differ diff --git a/content/blog/2022-ipfs/idle.png b/content/blog/2022-ipfs/idle.png new file mode 100644 index 0000000..c8e745e Binary files /dev/null and b/content/blog/2022-ipfs/idle.png differ diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md new file mode 100644 index 0000000..71918dd --- /dev/null +++ b/content/blog/2022-ipfs/index.md @@ -0,0 +1,102 @@ ++++ +title="We tried IPFS over Garage" +date=2022-06-09 ++++ + + +Once you have spawned your Garage cluster, you might want to federate or share efficiently your content with the rest of the world. +In this blog post, we try to connect the InterPlanetary File System (IPFS) daemon to Garage. +We discuss the different bottleneck and limitations of the currently available software. + + + + + + +## Preliminary + +People often struggle to see the difference between IPFS and Garage, so let's start by making clear that these projects are complementary and not interchangeable. +IPFS is a content-addressable network built in a peer-to-peer fashion. +With simple words, it means that you query the content you want with its identifier without having to know *where* it is hosted on the network, and especially on which machine. +As a side effect, you can share content over the Internet without any configuration (no firewall, NAT, fixed IP, DNS, etc.). +It has some nice benefits: if some content becomes very popular, all people that already accessed it can help serving it, and even if the original content provider goes offline, the content remains +availale in the network as long as one machine still have it. + +However, IPFS does not enforce any property on the durability and availablity of your data: the collaboration mentioned earlier is +done only on a spontaneous approach. So at first, if you want to be sure that your content remains alive, you must keep it on your node. +And if nobody makes a copy of your content, you will loose it as soon as your node goes offline and/or crashes. +Furthermore, if you need multiple nodes to store your content, IPFS is not able to automatically place content on your nodes, +enforce a given replication amount, check the integrity of your content, and so on. + +*Note: the IPFS project has another project named [IPFS Cluster](https://cluster.ipfs.io/) that addresses these problems. +We have not reviewed it yet but its 2 main differences with Garage are that 1) it does not expose an S3 API and 2) has a different consisenty model. +In the following, we suppose that these differences prevent you from deploying it in your infrastructure.* + + +➡️ **IPFS is designed to deliver content.** + +Garage, on the contrary, is designed to spread automatically your content over all your available nodes to optimize your storage space. At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter!) on different geographical zones (if possible). To access this content, you must have an API key, and have a correctly configured machine available over the network (including DNS/IP address/etc.). If the amount of traffic you receive is way larger than what your cluster can handle, your cluster will become simply unresponsive. Sharing content across people that do not trust each other, ie. who operate independant clusters, is not a feature of Garage: you have to rely on external software. + +➡️ **Garage is designed to durably store content.** + +In this blog post, we will explore if we can combine both properties by connecting an IPFS node to a Garage cluster. + +## Try #1: Vanilla IPFS over Garage + + + +IPFS is available as a pre-compiled binary. But to connect it with Garage, we need a plugin named [ipfs/go-ds-s3](https://github.com/ipfs/go-ds-s3). +The Peergos project has a fork because it seems that the plugin is notorious for hitting Amazon's rate limits [#105](https://github.com/ipfs/go-ds-s3/issues/105), [#205](https://github.com/ipfs/go-ds-s3/pull/205). +This is the one we will try in the following. + + +The easiest solution to use this plugin in IPFS is to bundle it in the main IPFS daemon, and thus recompile IPFS from source. +Following the instructions on the README file allowed me to spawn an IPFS daemon configured with S3 as the block store. + +Just be careful when adding the plugin to the `plugin/loader/preload_list` file, the given command lacks a newline. +You must edit the file manually after running it, you will directly see the problem and be able to fix it. + +After that, I just ran the daemon and accessed the web interface to upload a photo of my dog: + +![A dog](./dog.jpg) + +The photo is assigned a content identifier (CID): + +``` +QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn +``` + +And it now accessible on the whole network. +You can inspect it [from the official gateway](https://explore.ipld.io/#/explore/QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn) for example: + +![A screenshot of the IPFS explorer](./explorer.png) + +At the same time, I was monitoring Garage (through [the OpenTelemetry stack we have implemented earlier this year](/blog/2022-v0-7-released/)). +Just after launching the daemon and before doing anything, we have this surprisingly active Grafana plot: + +![Grafana API request rate when IPFS is idle](./idle.png) + +It means that in average, we have around 250 requests per second, mainly to check that an IPFS block does not exist locally. +But when I start interacting with IPFS, values become so high that our default OpenTelemetry configuration can not cope with them. +We have the following error in Garage's logs: + +``` +OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full +``` + +It is useless to change our parameters to see what is exactly the number of requests done on the cluster: it is way too high, multiple orders of magnitude too high. +As a comparison, this whole webpage, with its pictures, triggers around 10 requests on Garage to load, and I expect seeing the same order of magnitude with IPFS. + +I think we can conclude that this first try was a failure. +The S3 datastore on IPFS does too many request and would need some important work to optimize it. +But we should not give up too fast, because Peergos folks are known to run their software based on IPFS, in production, with an S3 backend. + +## Try #2: Peergos over Garage + + + -- cgit v1.2.3 From 5895e434e42028c0125d6d98e140c52640c5bad9 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Fri, 10 Jun 2022 17:25:53 +0200 Subject: IPFS article (Quentin's part) --- content/blog/2022-ipfs/grafa.png | Bin 0 -> 42891 bytes content/blog/2022-ipfs/grafa2.png | Bin 0 -> 46573 bytes content/blog/2022-ipfs/grafa3.png | Bin 0 -> 51953 bytes content/blog/2022-ipfs/index.md | 145 +++++++++++++++++++++++++++++++++---- content/blog/2022-ipfs/peergos.jpg | Bin 0 -> 226663 bytes content/blog/2022-ipfs/upload.png | Bin 0 -> 112418 bytes 6 files changed, 130 insertions(+), 15 deletions(-) create mode 100644 content/blog/2022-ipfs/grafa.png create mode 100644 content/blog/2022-ipfs/grafa2.png create mode 100644 content/blog/2022-ipfs/grafa3.png create mode 100644 content/blog/2022-ipfs/peergos.jpg create mode 100644 content/blog/2022-ipfs/upload.png diff --git a/content/blog/2022-ipfs/grafa.png b/content/blog/2022-ipfs/grafa.png new file mode 100644 index 0000000..4934105 Binary files /dev/null and b/content/blog/2022-ipfs/grafa.png differ diff --git a/content/blog/2022-ipfs/grafa2.png b/content/blog/2022-ipfs/grafa2.png new file mode 100644 index 0000000..e056028 Binary files /dev/null and b/content/blog/2022-ipfs/grafa2.png differ diff --git a/content/blog/2022-ipfs/grafa3.png b/content/blog/2022-ipfs/grafa3.png new file mode 100644 index 0000000..1b29b60 Binary files /dev/null and b/content/blog/2022-ipfs/grafa3.png differ diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 71918dd..93193a4 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -17,29 +17,42 @@ Still, you might want to share and collaborate with the rest of the world, and i or by connecting it to an application that will act as a "proxy" between Garage and the rest of the world. We refer as proxy software that know how to speak federated protocols (eg. Activity Pub, Solid, RemoteStorage, etc.) or distributed/p2p protocols (eg. Bittorrent, IPFS, etc.).--> -## Preliminary +## Some context People often struggle to see the difference between IPFS and Garage, so let's start by making clear that these projects are complementary and not interchangeable. -IPFS is a content-addressable network built in a peer-to-peer fashion. + +Personnaly, I see IPFS as the intersection between Bittorrent and a filesystem. Bittorrent remains probably one of the most efficient way to deliver +a copy of a file or a folder to a very large number of people. But it lacks some interactivity: once a torrent file has been generated, you can't simply +add a file to it. By adopting a filesystem abstraction, IPFS handles such use case smoothly. + + + -However, IPFS does not enforce any property on the durability and availablity of your data: the collaboration mentioned earlier is + + +However, you would probably not rely on Bittorrent to durably store your encrypted holiday pictures you shared with your friends, and this is the same for IPFS. +If at some time, everyone has a copy of the picture on their hard disk, people might delete it after a while without you knowing it. +You also can't easily collaborate to share this common treasure, for example, there is no automatic way to say that Alice and Bob +are in charge of storing the first half of the archive while Charlie and Eve are in charge of the second half. ➡️ **IPFS is designed to deliver content.** -Garage, on the contrary, is designed to spread automatically your content over all your available nodes to optimize your storage space. At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter!) on different geographical zones (if possible). To access this content, you must have an API key, and have a correctly configured machine available over the network (including DNS/IP address/etc.). If the amount of traffic you receive is way larger than what your cluster can handle, your cluster will become simply unresponsive. Sharing content across people that do not trust each other, ie. who operate independant clusters, is not a feature of Garage: you have to rely on external software. +*Note: the IPFS project has another project named [IPFS Cluster](https://cluster.ipfs.io/) that allow servers to collaborate on hosting IPFS content. +The Bittorrent project created Bittorrent Sync, renamed later [Resilio](https://www.resilio.com/individuals/), that provide a filesystem abstraction based on the Bittorrent protocol. +Reviewing these solutions is out of the scope of this article, feel free to try them by yourself!* + + +Garage, on the contrary, is designed to spread automatically your content over all your available nodes to optimize your storage space. At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter!) on different geographical zones (if possible). + +But independant Garage nodes can not collaborate to deliver popular contents: every one is playing alone. +All ressources created (keys, files, buckets) are tightly coupled to a cluster, people from different clusters can't collaborate on the same data (without additional software). ➡️ **Garage is designed to durably store content.** @@ -80,17 +93,24 @@ At the same time, I was monitoring Garage (through [the OpenTelemetry stack we h Just after launching the daemon and before doing anything, we have this surprisingly active Grafana plot: ![Grafana API request rate when IPFS is idle](./idle.png) +*Legend: y axis = requests per 10 seconds, x axis = time* + It means that in average, we have around 250 requests per second, mainly to check that an IPFS block does not exist locally. -But when I start interacting with IPFS, values become so high that our default OpenTelemetry configuration can not cope with them. +These requests are triggered by the DHT service of IPFS: my node being reachable over the Internet, it acts as a public DHT server and start answering global +block requests over the whole network. Each time it sees a block request, it sends a request to our backend to see if it exists. + +*We will try to tweak the IPFS configuration later - we know that we can deactivate the DHT server. For now, we will continue with the default parameters.* + +When I start interacting with IPFS by sending a file or browsing the default proposed catalogs (ie. the full XKCD archive), values become so high that our default OpenTelemetry configuration can not cope with them. We have the following error in Garage's logs: ``` OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full ``` -It is useless to change our parameters to see what is exactly the number of requests done on the cluster: it is way too high, multiple orders of magnitude too high. -As a comparison, this whole webpage, with its pictures, triggers around 10 requests on Garage to load, and I expect seeing the same order of magnitude with IPFS. +It is useless to change Garage parameters to see what is exactly the number of requests done on the cluster: it is way too high considering my targeted usage: sharing a picture. +As a comparison, this whole webpage, with its pictures, triggers around 10 requests on Garage to load, not thousands. I think we can conclude that this first try was a failure. The S3 datastore on IPFS does too many request and would need some important work to optimize it. @@ -98,5 +118,100 @@ But we should not give up too fast, because Peergos folks are known to run their ## Try #2: Peergos over Garage +[Peergos](https://peergos.org/) is designed as an end-to-end encrypted and federated alternative to Nextcloud. +Internally, it is built upon IPFS and is known to have an [integration of the S3 API](https://peergos.org/posts/direct-s3). +One important point of this integration is that your browser is able to bypass both the Peergos daemon and the IPFS daemon +to write and read IPFS blocks directly on the backemd. + + + +*I don't know exactly if Peergos is still considered in alpha or switched to beta, but keep in mind that it might be more experimental that you would like!* + + + +Starting Peergos on top of Garage required some small patches on both sides, +but in the end, we were able to get it working. +I was able to upload my file, see it in the interface, create a link to share it, +rename it, move it in a folder, and so on: + +![A screenshot of the Peergos interface](./upload.png) + +At the same time, the fans of my computer started to become a bit loud. +A quick look at Grafana shows that Garage is still very busy. + +![Screenshot of a grafana plot showing requests per second over time](./grafa.png) +*Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time* + +Again, the workload is dominated by the `HeadObject` requests. +After getting a look at `~/.peergos/.ipfs/config`, it seems the IPFS configuration used by the Peergos project is pretty standard. +And thus, again, we are acting as a DHT server, answering each second to thousands of block requests. + +We also some traffic on the `GetObject` endpoints (peaks of ~45 req/sec) and the `OPTIONS` HTTP verb. +This is the traffic generated by Peergos. The `OPTIONS` HTTP verb is here because we use the direct access feature of Peergos: our browser is sending CORS requests to Garage. +Internally, IPFS splits files in blocks of less than 256 kB, my picture is thus split in 2 blocks, requiring 2 requests over Garage to fetch it. +But even by knowing that IPFS split files in small blocks, I can't explain why we have so much `GetObject` requests. + +## Try #3: Optimizing IPFS + + + +We have seen in our 2 precedent tries that the main source of load was the +federation, and more especially, the DHT server. In this section, we want +to artificially remove this problem of the equation by preventing our IPFS node from federating +and see what pressure is put by Peergos alone on our local cluster. + +To isolate IPFS, we have set its routing type to `none`, we have cleared its bootstrap node list, +and we configured the swarm socket to listen only on localhost. + +Finally, we restart Peergos and observe this more peaceful graph: + +![Screenshot of a grafana plot showing requests per second over time](./grafa3.png) +*Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time* + +Now, for a given endpoint, we have peaks of around 10 req/sec which is way more reasonable. +Furthermore, we are not hammering anymore our backend with requests on objects that are not here. + +The next step would be to gradually allowing back our node to connect to the IPFS network, +while ensuring that the traffic to the S3 cluster remains low. For example, configuring our IPFS +node as a `dhtclient` instead of `dhtserver` would exempt it from answering public DHT requests. +Keeping an in-memory index (as a hashmap and/or blum filter) of the blocks stored on the current node +could also drastically reduce the number of requests. +It could also be interesting to explore ways to run in one process a full IPFS node with a DHT +server on the regularer filesystem datastore, and reserve a second process configured with the S3 datastore +to handle only our Peergos data. + +However, with these optimizations, the best we can expect is the traffic we have on the previous plot. +From a theoretical perspective, it is still higher than the optimal number of requests. +On S3, storing a file, downloading a file, listing available files are all actions that can be done in a single request. +Even if all requests have not the same cost on the cluster, processing a request has a non neglictible fixed cost. + + +## S3 and IPFS are incompatible? + +*Text by Alex* + +## Conclusion + +Running IPFS over a S3 backend does not quite work out of the box in term of performances yet. +We have identified some possible measures for improvement (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 backend only for your data) +that might allow you to still run an IPFS node over Garage. + +From a design perspective, it seems however that the numerous small blocks created by IPFS +do not map trivially to efficient S3 requests, and thus could be a limiting factor to any optimization work. + +As part of our test journey, we read some posts about performance issues on IPFS (eg. [#6283 - Reduce the impact of the DHT](https://github.com/ipfs/go-ipfs/issues/6283)) that are not +linked with the S3 connector. We might be negatively influenced by our failure to connect IPFS with S3, +but we are tempted to think that in any case, IPFS will be ressource intensive for your hardware. +On our side, we will continue our investigations towards more *minimalist* software that tends to limit the +number of requests they send. +This choice makes sense for us as we want to reduce the ecological impact of our services +by deploying optimized software on a limited number of second-hand servers. +*Yes we are aware of the existence of Nextcloud, Owncloud, Owncloud Infinite Scale, Seafile, Filestash, Pydio, SOLID, Remote Storage, etc. +We might even try one of them in a future post, so stay tuned!* diff --git a/content/blog/2022-ipfs/peergos.jpg b/content/blog/2022-ipfs/peergos.jpg new file mode 100644 index 0000000..439542b Binary files /dev/null and b/content/blog/2022-ipfs/peergos.jpg differ diff --git a/content/blog/2022-ipfs/upload.png b/content/blog/2022-ipfs/upload.png new file mode 100644 index 0000000..1a9c40b Binary files /dev/null and b/content/blog/2022-ipfs/upload.png differ -- cgit v1.2.3 From cf672377f35374b47554854a5c85c4b617561d70 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Fri, 10 Jun 2022 17:31:10 +0200 Subject: Add an adjective --- content/blog/2022-ipfs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 93193a4..b56eeb1 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -119,7 +119,7 @@ But we should not give up too fast, because Peergos folks are known to run their ## Try #2: Peergos over Garage [Peergos](https://peergos.org/) is designed as an end-to-end encrypted and federated alternative to Nextcloud. -Internally, it is built upon IPFS and is known to have an [integration of the S3 API](https://peergos.org/posts/direct-s3). +Internally, it is built upon IPFS and is known to have a [deep integration of the S3 API](https://peergos.org/posts/direct-s3) One important point of this integration is that your browser is able to bypass both the Peergos daemon and the IPFS daemon to write and read IPFS blocks directly on the backemd. -- cgit v1.2.3 From 0e2689efdbffe3c00c09810a984f628179a33783 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Fri, 10 Jun 2022 17:49:02 +0200 Subject: Rework conclusion --- content/blog/2022-ipfs/index.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index b56eeb1..03ce073 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -198,20 +198,19 @@ Even if all requests have not the same cost on the cluster, processing a request ## Conclusion Running IPFS over a S3 backend does not quite work out of the box in term of performances yet. -We have identified some possible measures for improvement (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 backend only for your data) -that might allow you to still run an IPFS node over Garage. +We have identified that the main problem is linked with the DHT service, +and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 backend only for your data). From a design perspective, it seems however that the numerous small blocks created by IPFS do not map trivially to efficient S3 requests, and thus could be a limiting factor to any optimization work. -As part of our test journey, we read some posts about performance issues on IPFS (eg. [#6283 - Reduce the impact of the DHT](https://github.com/ipfs/go-ipfs/issues/6283)) that are not +As part of our test journey, we also read some posts about performance issues on IPFS (eg. [#6283](https://github.com/ipfs/go-ipfs/issues/6283)) that are not linked with the S3 connector. We might be negatively influenced by our failure to connect IPFS with S3, -but we are tempted to think that in any case, IPFS will be ressource intensive for your hardware. +but we are tempted to think that IPFS is intrinsically ressource intensive. -On our side, we will continue our investigations towards more *minimalist* software that tends to limit the -number of requests they send. +On our side, we will continue our investigations towards more *minimalist* software. This choice makes sense for us as we want to reduce the ecological impact of our services -by deploying optimized software on a limited number of second-hand servers. +by deploying less servers, that use less energy, and that are renewed less frequently. *Yes we are aware of the existence of Nextcloud, Owncloud, Owncloud Infinite Scale, Seafile, Filestash, Pydio, SOLID, Remote Storage, etc. We might even try one of them in a future post, so stay tuned!* -- cgit v1.2.3 From 7c951f4376d380fb679d0f8a60959f217e4ca5e4 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Fri, 10 Jun 2022 17:58:15 +0200 Subject: Add a note about Peergos without IPFS --- content/blog/2022-ipfs/index.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 03ce073..848778d 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -176,7 +176,12 @@ Finally, we restart Peergos and observe this more peaceful graph: Now, for a given endpoint, we have peaks of around 10 req/sec which is way more reasonable. Furthermore, we are not hammering anymore our backend with requests on objects that are not here. -The next step would be to gradually allowing back our node to connect to the IPFS network, +After discussing with the developpers, it is possible to go even further by running Peergos without IPFS: +this is what they do for some of their tests. At the same time, if you increase the size +of a block, you might have a non-federated but efficient end-to-end encrypted "cloud storage" that works over Garage, +with your clients directly hitting the S3 API! + +If federation is a hard requirement for your, the next step would be to gradually allowing back our node to connect to the IPFS network, while ensuring that the traffic to the S3 cluster remains low. For example, configuring our IPFS node as a `dhtclient` instead of `dhtserver` would exempt it from answering public DHT requests. Keeping an in-memory index (as a hashmap and/or blum filter) of the blocks stored on the current node @@ -201,7 +206,11 @@ Running IPFS over a S3 backend does not quite work out of the box in term of per We have identified that the main problem is linked with the DHT service, and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 backend only for your data). -From a design perspective, it seems however that the numerous small blocks created by IPFS +It is possible to modify Peergos to make it work without IPFS. With some optimization on the block size, +you might have a great proof of concept of an end-to-end encrypted "cloud storage" over Garage. +*Ping us if you make a prototype!* + +From an IPFS design perspective, it seems however that the numerous small blocks handled by the protocol do not map trivially to efficient S3 requests, and thus could be a limiting factor to any optimization work. As part of our test journey, we also read some posts about performance issues on IPFS (eg. [#6283](https://github.com/ipfs/go-ipfs/issues/6283)) that are not -- cgit v1.2.3 From b1ff241de84d4f46d9f522ce096df9c83b355ec3 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Fri, 10 Jun 2022 17:59:55 +0200 Subject: Add an adjective --- content/blog/2022-ipfs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 848778d..64695e6 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -178,7 +178,7 @@ Furthermore, we are not hammering anymore our backend with requests on objects t After discussing with the developpers, it is possible to go even further by running Peergos without IPFS: this is what they do for some of their tests. At the same time, if you increase the size -of a block, you might have a non-federated but efficient end-to-end encrypted "cloud storage" that works over Garage, +of a block, you might have a non-federated but efficient end-to-end encrypted "cloud storage" that works well over Garage, with your clients directly hitting the S3 API! If federation is a hard requirement for your, the next step would be to gradually allowing back our node to connect to the IPFS network, -- cgit v1.2.3 From 51bcdcd68e8825cfc987141b8258471d0cb621b8 Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Fri, 10 Jun 2022 18:04:59 +0200 Subject: Be more precise about Resilio and Syncthing --- content/blog/2022-ipfs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 64695e6..9c60150 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -45,7 +45,7 @@ are in charge of storing the first half of the archive while Charlie and Eve are ➡️ **IPFS is designed to deliver content.** *Note: the IPFS project has another project named [IPFS Cluster](https://cluster.ipfs.io/) that allow servers to collaborate on hosting IPFS content. -The Bittorrent project created Bittorrent Sync, renamed later [Resilio](https://www.resilio.com/individuals/), that provide a filesystem abstraction based on the Bittorrent protocol. +[Resilio](https://www.resilio.com/individuals/) and [Syncthing](https://syncthing.net/) both feature protocols inspired by Bittorrent to synchronize a tree of your filesystem between multiple computers. Reviewing these solutions is out of the scope of this article, feel free to try them by yourself!* -- cgit v1.2.3 From 1458ef0ecabb5a3a67f1352bc7e1ef1755734356 Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Wed, 15 Jun 2022 19:47:23 +0200 Subject: start fix ipfs article --- content/blog/2022-ipfs/index.md | 154 ++++++++++++++++++++++------------------ 1 file changed, 83 insertions(+), 71 deletions(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 9c60150..3ff2332 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -4,9 +4,10 @@ date=2022-06-09 +++ -Once you have spawned your Garage cluster, you might want to federate or share efficiently your content with the rest of the world. -In this blog post, we try to connect the InterPlanetary File System (IPFS) daemon to Garage. -We discuss the different bottleneck and limitations of the currently available software. +*Once you have spawned your Garage cluster, you might be interested in finding ways to share efficiently your content with the rest of the world, +such as by joining federated platforms. +In this blog post, we experiment with interconnecting the InterPlanetary File System (IPFS) daemon with Garage. +We discuss the different bottlenecks and limitations of the software stack as it is currently available.* @@ -15,117 +16,125 @@ We discuss the different bottleneck and limitations of the currently available s It is an intended design decision: trusting each other enables Garage to spread data over the machines instead of duplicating it. Still, you might want to share and collaborate with the rest of the world, and it can be done in 2 ways with Garage: through the integrated HTTP server that can serve your bucket as a static website, or by connecting it to an application that will act as a "proxy" between Garage and the rest of the world. -We refer as proxy software that know how to speak federated protocols (eg. Activity Pub, Solid, RemoteStorage, etc.) or distributed/p2p protocols (eg. Bittorrent, IPFS, etc.).--> +We refer as proxy software that know how to speak federated protocols (eg. Activity Pub, Solid, RemoteStorage, etc.) or distributed/p2p protocols (eg. BitTorrent, IPFS, etc.).--> ## Some context People often struggle to see the difference between IPFS and Garage, so let's start by making clear that these projects are complementary and not interchangeable. -Personnaly, I see IPFS as the intersection between Bittorrent and a filesystem. Bittorrent remains probably one of the most efficient way to deliver -a copy of a file or a folder to a very large number of people. But it lacks some interactivity: once a torrent file has been generated, you can't simply -add a file to it. By adopting a filesystem abstraction, IPFS handles such use case smoothly. +Personally, I see IPFS as the intersection between BitTorrent and a file system. BitTorrent remains to this day one of the most efficient ways to deliver +a copy of a file or a folder to a very large number of destinations. It however lacks some form of interactivity: once a torrent file has been generated, you can't simply +add or remove files from it. By presenting itself more like a file system, IPFS is able to handle this use case out-of-the-box. - - - -However, you would probably not rely on Bittorrent to durably store your encrypted holiday pictures you shared with your friends, and this is the same for IPFS. -If at some time, everyone has a copy of the picture on their hard disk, people might delete it after a while without you knowing it. -You also can't easily collaborate to share this common treasure, for example, there is no automatic way to say that Alice and Bob +However, you would probably not rely on BitTorrent to durably store your encrypted holiday pictures you shared with your friends, +as content on the BitTorrent tends to vanish when no one in the network has a copy of it anymore. The same applies to IPFS. +If at some time, everyone has a copy of the pictures on their hard disk, people might delete these copies after a while without you knowing it. +You also can't easily collaborate to share this common treasure. For example, there is no automatic way to say that Alice and Bob are in charge of storing the first half of the archive while Charlie and Eve are in charge of the second half. ➡️ **IPFS is designed to deliver content.** *Note: the IPFS project has another project named [IPFS Cluster](https://cluster.ipfs.io/) that allow servers to collaborate on hosting IPFS content. -[Resilio](https://www.resilio.com/individuals/) and [Syncthing](https://syncthing.net/) both feature protocols inspired by Bittorrent to synchronize a tree of your filesystem between multiple computers. +[Resilio](https://www.resilio.com/individuals/) and [Syncthing](https://syncthing.net/) both feature protocols inspired by BitTorrent to synchronize a tree of your file system between multiple computers. Reviewing these solutions is out of the scope of this article, feel free to try them by yourself!* - -Garage, on the contrary, is designed to spread automatically your content over all your available nodes to optimize your storage space. At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter!) on different geographical zones (if possible). +Garage, on the contrary, is designed to spread automatically your content over all your available nodes, in a manner that makes the best use possible of your storage space. +At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter), +on different geographical zones when possible. -But independant Garage nodes can not collaborate to deliver popular contents: every one is playing alone. -All ressources created (keys, files, buckets) are tightly coupled to a cluster, people from different clusters can't collaborate on the same data (without additional software). +However, this means that when content is requested from a Garage cluster, there are only 3 nodes that are capable of returning it to the user. +As a consequence, when content becomes popular, these nodes might become a bottleneck. +Moreover, all resources created (keys, files, buckets) are tightly coupled to the Garage cluster on which they exist; +servers from different clusters can't collaborate to serve together the same data (without additional software). ➡️ **Garage is designed to durably store content.** -In this blog post, we will explore if we can combine both properties by connecting an IPFS node to a Garage cluster. +In this blog post, we will explore whether we can combine both properties by connecting an IPFS node to a Garage cluster. ## Try #1: Vanilla IPFS over Garage -IPFS is available as a pre-compiled binary. But to connect it with Garage, we need a plugin named [ipfs/go-ds-s3](https://github.com/ipfs/go-ds-s3). -The Peergos project has a fork because it seems that the plugin is notorious for hitting Amazon's rate limits [#105](https://github.com/ipfs/go-ds-s3/issues/105), [#205](https://github.com/ipfs/go-ds-s3/pull/205). +IPFS is available as a pre-compiled binary, but to connect it with Garage, we need a plugin named [ipfs/go-ds-s3](https://github.com/ipfs/go-ds-s3). +The Peergos project has a fork because it seems that the plugin is known for hitting Amazon's rate limits +([#105](https://github.com/ipfs/go-ds-s3/issues/105), [#205](https://github.com/ipfs/go-ds-s3/pull/205)). This is the one we will try in the following. - The easiest solution to use this plugin in IPFS is to bundle it in the main IPFS daemon, and thus recompile IPFS from source. -Following the instructions on the README file allowed me to spawn an IPFS daemon configured with S3 as the block store. +Following the instructions on the README file allowed me to spawn an IPFS daemon configured with S3 as the block store. -Just be careful when adding the plugin to the `plugin/loader/preload_list` file, the given command lacks a newline. -You must edit the file manually after running it, you will directly see the problem and be able to fix it. +I had a small issue when adding the plugin to the `plugin/loader/preload_list` file: the given command lacks a newline. +I had to edit the file manually after running it, the issue was directly visible and easy to fix. After that, I just ran the daemon and accessed the web interface to upload a photo of my dog: +
![A dog](./dog.jpg) +
-The photo is assigned a content identifier (CID): +A content identifier (CID) was assigned to this picture: ``` QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn ``` -And it now accessible on the whole network. +The photo it now accessible on the whole network. You can inspect it [from the official gateway](https://explore.ipld.io/#/explore/QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn) for example: ![A screenshot of the IPFS explorer](./explorer.png) At the same time, I was monitoring Garage (through [the OpenTelemetry stack we have implemented earlier this year](/blog/2022-v0-7-released/)). -Just after launching the daemon and before doing anything, we have this surprisingly active Grafana plot: +Just after launching the daemon and before doing anything, we had this surprisingly active Grafana plot: +
![Grafana API request rate when IPFS is idle](./idle.png) -*Legend: y axis = requests per 10 seconds, x axis = time* - +

*Legend: y axis = requests per 10 seconds, x axis = time*

+
-It means that in average, we have around 250 requests per second, mainly to check that an IPFS block does not exist locally. -These requests are triggered by the DHT service of IPFS: my node being reachable over the Internet, it acts as a public DHT server and start answering global -block requests over the whole network. Each time it sees a block request, it sends a request to our backend to see if it exists. +It means that on average, we have around 250 requests per second. Most of these requests are checks that an IPFS block does not exist locally. +These requests are triggered by the DHT service of IPFS: since my node is reachable over the Internet, it acts as a public DHT server and start answering global +block requests over the whole network. Each time it receives a request for a block, it sends a request to its storage back-end to see if it exists +(in our case, to Garage). *We will try to tweak the IPFS configuration later - we know that we can deactivate the DHT server. For now, we will continue with the default parameters.* -When I start interacting with IPFS by sending a file or browsing the default proposed catalogs (ie. the full XKCD archive), values become so high that our default OpenTelemetry configuration can not cope with them. +When I start interacting with IPFS by sending a file or browsing the default proposed catalogs (ie. the full XKCD archive), +we hit limits with our monitoring stack which, in its default configuration, is not able to ingest the traces of +so many requests being processed by Garage. We have the following error in Garage's logs: ``` OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full ``` -It is useless to change Garage parameters to see what is exactly the number of requests done on the cluster: it is way too high considering my targeted usage: sharing a picture. -As a comparison, this whole webpage, with its pictures, triggers around 10 requests on Garage to load, not thousands. +At this point, I didn't feel that it would be very interesting to fix this issue to see what is exactly the number of requests done on the cluster. +In my opinion, such a simple task of sharing a picture should not require so many requests to the storage server. +As a comparison, this whole webpage, with its pictures, triggers around 10 requests on Garage when loaded, not thousands. I think we can conclude that this first try was a failure. -The S3 datastore on IPFS does too many request and would need some important work to optimize it. -But we should not give up too fast, because Peergos folks are known to run their software based on IPFS, in production, with an S3 backend. +The S3 storage plugin for IPFS does too many request and would need some important work to optimize it. +But we should not give up too fast, because the Peergos folks are known to run their software based on IPFS, in production, with an S3 backend. ## Try #2: Peergos over Garage [Peergos](https://peergos.org/) is designed as an end-to-end encrypted and federated alternative to Nextcloud. -Internally, it is built upon IPFS and is known to have a [deep integration of the S3 API](https://peergos.org/posts/direct-s3) +Internally, it is built upon IPFS and is known to have a [deep integration with the S3 API](https://peergos.org/posts/direct-s3). One important point of this integration is that your browser is able to bypass both the Peergos daemon and the IPFS daemon -to write and read IPFS blocks directly on the backemd. +to write and read IPFS blocks directly from the S3 API server. - - -*I don't know exactly if Peergos is still considered in alpha or switched to beta, but keep in mind that it might be more experimental that you would like!* +*I don't know exactly if Peergos is still considered as alpha quality, or if a beta version was released, +but keep in mind that it might be more experimental that you would like!* -We have seen in our 2 precedent tries that the main source of load was the -federation, and more especially, the DHT server. In this section, we want -to artificially remove this problem of the equation by preventing our IPFS node from federating +We have seen in our 2 previous tries that the main source of load was the federation, and more especially, the DHT server. +In this section, we want to artificially remove this problem from the equation by preventing our IPFS node from federating and see what pressure is put by Peergos alone on our local cluster. To isolate IPFS, we have set its routing type to `none`, we have cleared its bootstrap node list, @@ -174,27 +186,25 @@ Finally, we restart Peergos and observe this more peaceful graph: *Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time* Now, for a given endpoint, we have peaks of around 10 req/sec which is way more reasonable. -Furthermore, we are not hammering anymore our backend with requests on objects that are not here. +Furthermore, we are no longer hammering our back-end with requests on objects that are not there. -After discussing with the developpers, it is possible to go even further by running Peergos without IPFS: +After discussing with the developers, it is possible to go even further by running Peergos without IPFS: this is what they do for some of their tests. At the same time, if you increase the size of a block, you might have a non-federated but efficient end-to-end encrypted "cloud storage" that works well over Garage, with your clients directly hitting the S3 API! -If federation is a hard requirement for your, the next step would be to gradually allowing back our node to connect to the IPFS network, -while ensuring that the traffic to the S3 cluster remains low. For example, configuring our IPFS -node as a `dhtclient` instead of `dhtserver` would exempt it from answering public DHT requests. -Keeping an in-memory index (as a hashmap and/or blum filter) of the blocks stored on the current node +If federation is a hard requirement for your setup, the next step would be to gradually allow our node to connect to the IPFS network, +while ensuring that the traffic to the S3 cluster remains low. +For example, configuring our IPFS node as a `dhtclient` instead of `dhtserver` would exempt it from answering public DHT requests. +Keeping an in-memory index (as a hash map and/or Bloom filter) of the blocks stored on the current node could also drastically reduce the number of requests. It could also be interesting to explore ways to run in one process a full IPFS node with a DHT -server on the regularer filesystem datastore, and reserve a second process configured with the S3 datastore -to handle only our Peergos data. +server on the regular file system, and reserve a second process configured with the S3 back-end to handle only our Peergos data. However, with these optimizations, the best we can expect is the traffic we have on the previous plot. From a theoretical perspective, it is still higher than the optimal number of requests. On S3, storing a file, downloading a file, listing available files are all actions that can be done in a single request. -Even if all requests have not the same cost on the cluster, processing a request has a non neglictible fixed cost. - +Even if all requests don't have the same cost on the cluster, processing a request has a non-negligible fixed cost. ## S3 and IPFS are incompatible? @@ -202,24 +212,26 @@ Even if all requests have not the same cost on the cluster, processing a request ## Conclusion -Running IPFS over a S3 backend does not quite work out of the box in term of performances yet. +Running IPFS over an S3 storage back-end does not quite work out of the box in term of performances in the current state of affairs. We have identified that the main problem is linked with the DHT service, -and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 backend only for your data). +and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 back-end only for your data). It is possible to modify Peergos to make it work without IPFS. With some optimization on the block size, you might have a great proof of concept of an end-to-end encrypted "cloud storage" over Garage. -*Ping us if you make a prototype!* +*If you happen to be working on this, please inform us!* From an IPFS design perspective, it seems however that the numerous small blocks handled by the protocol do not map trivially to efficient S3 requests, and thus could be a limiting factor to any optimization work. As part of our test journey, we also read some posts about performance issues on IPFS (eg. [#6283](https://github.com/ipfs/go-ipfs/issues/6283)) that are not linked with the S3 connector. We might be negatively influenced by our failure to connect IPFS with S3, -but we are tempted to think that IPFS is intrinsically ressource intensive. +but we are tempted to think that IPFS is intrinsically resource-intensive. -On our side, we will continue our investigations towards more *minimalist* software. +On our side, we will continue our investigations towards more *minimalistic* software. This choice makes sense for us as we want to reduce the ecological impact of our services by deploying less servers, that use less energy, and that are renewed less frequently. -*Yes we are aware of the existence of Nextcloud, Owncloud, Owncloud Infinite Scale, Seafile, Filestash, Pydio, SOLID, Remote Storage, etc. -We might even try one of them in a future post, so stay tuned!* +*We are aware of the existence of many other software projects for file sharing +such as Nextcloud, Owncloud, Owncloud Infinite Scale, Seafile, Filestash, Pydio, SOLID, Remote Storage, etc. +Many of these could be connected to an S3 back-end such as Garage. +We might even try some of them in future blog posts, so stay tuned!* -- cgit v1.2.3 From c60063c1807148f6945b5fbd73621eb19488a301 Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Wed, 15 Jun 2022 20:19:58 +0200 Subject: More improvements to ipfs article --- content/blog/2022-ipfs/index.md | 109 +++++++++++++++++++--------------------- 1 file changed, 51 insertions(+), 58 deletions(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 3ff2332..4dade1c 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -48,7 +48,7 @@ are in charge of storing the first half of the archive while Charlie and Eve are [Resilio](https://www.resilio.com/individuals/) and [Syncthing](https://syncthing.net/) both feature protocols inspired by BitTorrent to synchronize a tree of your file system between multiple computers. Reviewing these solutions is out of the scope of this article, feel free to try them by yourself!* -Garage, on the contrary, is designed to spread automatically your content over all your available nodes, in a manner that makes the best use possible of your storage space. +Garage, on the contrary, is designed to spread automatically your content over all your available nodes, in a manner that makes the best possible use of your storage space. At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter), on different geographical zones when possible. @@ -79,9 +79,7 @@ I had to edit the file manually after running it, the issue was directly visible After that, I just ran the daemon and accessed the web interface to upload a photo of my dog: -
![A dog](./dog.jpg) -
A content identifier (CID) was assigned to this picture: @@ -90,27 +88,24 @@ QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn ``` The photo it now accessible on the whole network. -You can inspect it [from the official gateway](https://explore.ipld.io/#/explore/QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn) for example: +For example you can inspect it [from the official gateway](https://explore.ipld.io/#/explore/QmNt7NSzyGkJ5K9QzyceDXd18PbLKrMAE93XuSC2487EFn): ![A screenshot of the IPFS explorer](./explorer.png) At the same time, I was monitoring Garage (through [the OpenTelemetry stack we have implemented earlier this year](/blog/2022-v0-7-released/)). Just after launching the daemon and before doing anything, we had this surprisingly active Grafana plot: -
-![Grafana API request rate when IPFS is idle](./idle.png) -

*Legend: y axis = requests per 10 seconds, x axis = time*

-
+![Grafana API request rate when IPFS is idle](./idle.png) +
Legend: y axis = requests per 10 seconds, x axis = time

It means that on average, we have around 250 requests per second. Most of these requests are checks that an IPFS block does not exist locally. -These requests are triggered by the DHT service of IPFS: since my node is reachable over the Internet, it acts as a public DHT server and start answering global -block requests over the whole network. Each time it receives a request for a block, it sends a request to its storage back-end to see if it exists -(in our case, to Garage). +These requests are triggered by the DHT service of IPFS: since my node is reachable over the Internet, it acts as a public DHT server and has to answer global +block requests over the whole network. Each time it receives a request for a block, it sends a request to its storage back-end (in our case, to Garage) to see if it exists. *We will try to tweak the IPFS configuration later - we know that we can deactivate the DHT server. For now, we will continue with the default parameters.* -When I start interacting with IPFS by sending a file or browsing the default proposed catalogs (ie. the full XKCD archive), -we hit limits with our monitoring stack which, in its default configuration, is not able to ingest the traces of +When I start interacting with IPFS by sending a file or browsing the default proposed catalogs (i.e. the full XKCD archive), +I hit limits with our monitoring stack which, in its default configuration, is not able to ingest the traces of so many requests being processed by Garage. We have the following error in Garage's logs: @@ -118,53 +113,50 @@ We have the following error in Garage's logs: OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full ``` -At this point, I didn't feel that it would be very interesting to fix this issue to see what is exactly the number of requests done on the cluster. -In my opinion, such a simple task of sharing a picture should not require so many requests to the storage server. +At this point, I didn't feel that it would be very interesting to fix this issue to see what was exactly the number of requests done on the cluster. +In my opinion, such a simple task of sharing a picture should not require so many requests to the storage server anyway. As a comparison, this whole webpage, with its pictures, triggers around 10 requests on Garage when loaded, not thousands. I think we can conclude that this first try was a failure. -The S3 storage plugin for IPFS does too many request and would need some important work to optimize it. -But we should not give up too fast, because the Peergos folks are known to run their software based on IPFS, in production, with an S3 backend. +The S3 storage plugin for IPFS does too many request and would need some important work to be optimized. +However, we should not give up too fast, because the people behind Peergos are known to run their software based on IPFS in production with an S3 backend. ## Try #2: Peergos over Garage [Peergos](https://peergos.org/) is designed as an end-to-end encrypted and federated alternative to Nextcloud. -Internally, it is built upon IPFS and is known to have a [deep integration with the S3 API](https://peergos.org/posts/direct-s3). +Internally, it is built on IPFS and is known to have a [deep integration with the S3 API](https://peergos.org/posts/direct-s3). One important point of this integration is that your browser is able to bypass both the Peergos daemon and the IPFS daemon to write and read IPFS blocks directly from the S3 API server. *I don't know exactly if Peergos is still considered as alpha quality, or if a beta version was released, -but keep in mind that it might be more experimental that you would like!* +but keep in mind that it might be more experimental that you'd like!* -Starting Peergos on top of Garage required some small patches on both sides, -but in the end, we were able to get it working. -I was able to upload my file, see it in the interface, create a link to share it, -rename it, move it in a folder, and so on: +Starting Peergos on top of Garage required some small patches on both sides, but in the end, I was able to get it working. +I was able to upload my file, see it in the interface, create a link to share it, rename it, move it in a folder, and so on: ![A screenshot of the Peergos interface](./upload.png) -At the same time, the fans of my computer started to become a bit loud. +At the same time, the fans of my computer started to become a bit loud! A quick look at Grafana shows that Garage is still very busy: -
-![Screenshot of a grafana plot showing requests per second over time](./grafa.png) -

*Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time*

-
+![Screenshot of a grafana plot showing requests per second over time](./grafa.png) +
Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time

-Again, the workload is dominated by the `HeadObject` requests. -After getting a look at `~/.peergos/.ipfs/config`, it seems that the IPFS configuration used by the Peergos project is quite standard, -which means that, same as before, we are acting as a DHT server and having to answer to thousands of block requests every second. +Again, the workload is dominated by `HeadObject` requests. +After taking a look at `~/.peergos/.ipfs/config`, it seems that the IPFS configuration used by the Peergos project is quite standard, +which means that, as before, we are acting as a DHT server and having to answer to thousands of block requests every second. We also have some traffic on the `GetObject` and `OPTIONS` endpoints (with peaks up to ~45 req/sec). This traffic is all generated by Peergos. The `OPTIONS` HTTP verb is here because we use the direct access feature of Peergos, meaning that our browser is talking directly to Garage and has to use CORS to validate requests for security. + Internally, IPFS splits files in blocks of less than 256 kB. My picture is thus split in 2 blocks, requiring 2 requests over Garage to fetch it. -But even by knowing that IPFS split files in small blocks, I can't explain why we have so much `GetObject` requests. +But even by knowing that IPFS split files in small blocks, I can't explain why we have so many `GetObject` requests. ## Try #3: Optimizing IPFS @@ -174,36 +166,36 @@ Routing = dhtclient --> We have seen in our 2 previous tries that the main source of load was the federation, and more especially, the DHT server. -In this section, we want to artificially remove this problem from the equation by preventing our IPFS node from federating +In this section, we'd like to artificially remove this problem from the equation by preventing our IPFS node from federating and see what pressure is put by Peergos alone on our local cluster. -To isolate IPFS, we have set its routing type to `none`, we have cleared its bootstrap node list, -and we configured the swarm socket to listen only on localhost. - -Finally, we restart Peergos and observe this more peaceful graph: +To isolate IPFS, I have set its routing type to `none`, I have cleared its bootstrap node list, +and I configured the swarm socket to listen only on `localhost`. +Finally, I restarted Peergos and was able to observe this more peaceful graph: -![Screenshot of a grafana plot showing requests per second over time](./grafa3.png) -*Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time* +![Screenshot of a grafana plot showing requests per second over time](./grafa3.png) +
Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time

Now, for a given endpoint, we have peaks of around 10 req/sec which is way more reasonable. Furthermore, we are no longer hammering our back-end with requests on objects that are not there. After discussing with the developers, it is possible to go even further by running Peergos without IPFS: -this is what they do for some of their tests. At the same time, if you increase the size -of a block, you might have a non-federated but efficient end-to-end encrypted "cloud storage" that works well over Garage, -with your clients directly hitting the S3 API! - -If federation is a hard requirement for your setup, the next step would be to gradually allow our node to connect to the IPFS network, -while ensuring that the traffic to the S3 cluster remains low. -For example, configuring our IPFS node as a `dhtclient` instead of `dhtserver` would exempt it from answering public DHT requests. -Keeping an in-memory index (as a hash map and/or Bloom filter) of the blocks stored on the current node +this is what they do for some of their tests. If at the same time we increased the size of data blocks, +we might have a non-federated but quite efficient end-to-end encrypted "cloud storage" that works well over Garage, +with our clients directly hitting the S3 API! + +For setups where federation is a hard requirement, +the next step would be to gradually allow our node to connect to the IPFS network, +while ensuring that the traffic to the Garage cluster remains low. +For example, configuring our IPFS node as a `dhtclient` instead of a `dhtserver` would exempt it from answering public DHT requests. +Keeping an in-memory index (as a hash map and/or a Bloom filter) of the blocks stored on the current node could also drastically reduce the number of requests. It could also be interesting to explore ways to run in one process a full IPFS node with a DHT server on the regular file system, and reserve a second process configured with the S3 back-end to handle only our Peergos data. -However, with these optimizations, the best we can expect is the traffic we have on the previous plot. +However, even with these optimizations, the best we can expect is the traffic we have on the previous plot. From a theoretical perspective, it is still higher than the optimal number of requests. -On S3, storing a file, downloading a file, listing available files are all actions that can be done in a single request. +On S3, storing a file, downloading a file and listing available files are all actions that can be done in a single request. Even if all requests don't have the same cost on the cluster, processing a request has a non-negligible fixed cost. ## S3 and IPFS are incompatible? @@ -212,22 +204,23 @@ Even if all requests don't have the same cost on the cluster, processing a reque ## Conclusion -Running IPFS over an S3 storage back-end does not quite work out of the box in term of performances in the current state of affairs. +Running IPFS over an S3 storage back-end does not quite work out of the box in term of performances. We have identified that the main problem is linked with the DHT service, -and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 back-end only for your data). +and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 back-end only for user data). -It is possible to modify Peergos to make it work without IPFS. With some optimization on the block size, -you might have a great proof of concept of an end-to-end encrypted "cloud storage" over Garage. +It is possible to modify Peergos to make it work without IPFS. +With some optimizations on the block size, +we might have a great proof of concept of an end-to-end encrypted "cloud storage" over Garage. *If you happen to be working on this, please inform us!* From an IPFS design perspective, it seems however that the numerous small blocks handled by the protocol -do not map trivially to efficient S3 requests, and thus could be a limiting factor to any optimization work. +do not map trivially to efficient use of the S3 API, and thus could be a limiting factor to any optimization work. -As part of our test journey, we also read some posts about performance issues on IPFS (eg. [#6283](https://github.com/ipfs/go-ipfs/issues/6283)) that are not -linked with the S3 connector. We might be negatively influenced by our failure to connect IPFS with S3, -but we are tempted to think that IPFS is intrinsically resource-intensive. +As part of my testing journey, I also stumbled upon some posts about performance issues on IPFS (eg. [#6283](https://github.com/ipfs/go-ipfs/issues/6283)) +that are not linked with the S3 connector. I might be negatively influenced by my failure to connect IPFS with S3, +but at this point I'm tempted to think that IPFS is intrinsically resource-intensive. -On our side, we will continue our investigations towards more *minimalistic* software. +On our side at Deuxfleurs, we will continue our investigations towards more *minimalistic* software. This choice makes sense for us as we want to reduce the ecological impact of our services by deploying less servers, that use less energy, and that are renewed less frequently. -- cgit v1.2.3 From e6f2d498167d5ad66f08d77d4b46e71217571dbc Mon Sep 17 00:00:00 2001 From: Quentin Dufour Date: Thu, 16 Jun 2022 17:36:09 +0200 Subject: Rework conclusion --- content/blog/2022-ipfs/index.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 4dade1c..152e0a5 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -208,11 +208,6 @@ Running IPFS over an S3 storage back-end does not quite work out of the box in t We have identified that the main problem is linked with the DHT service, and proposed some improvements (disabling the DHT server, keeping an in-memory index of the blocks, using the S3 back-end only for user data). -It is possible to modify Peergos to make it work without IPFS. -With some optimizations on the block size, -we might have a great proof of concept of an end-to-end encrypted "cloud storage" over Garage. -*If you happen to be working on this, please inform us!* - From an IPFS design perspective, it seems however that the numerous small blocks handled by the protocol do not map trivially to efficient use of the S3 API, and thus could be a limiting factor to any optimization work. @@ -224,7 +219,13 @@ On our side at Deuxfleurs, we will continue our investigations towards more *min This choice makes sense for us as we want to reduce the ecological impact of our services by deploying less servers, that use less energy, and that are renewed less frequently. -*We are aware of the existence of many other software projects for file sharing +After discussing with Peergos maintainers, we identified that it is possible to run Peergos without IPFS. +With some optimizations on the block size, we envision great synergies between Garage and Peergos that could lead to +an efficient and lightweight end-to-end encrypted "cloud storage" platform. +*If you happen to be working on this, please inform us!* + + +*We are also aware of the existence of many other software projects for file sharing such as Nextcloud, Owncloud, Owncloud Infinite Scale, Seafile, Filestash, Pydio, SOLID, Remote Storage, etc. Many of these could be connected to an S3 back-end such as Garage. We might even try some of them in future blog posts, so stay tuned!* -- cgit v1.2.3 From 7193a1cce9f9f10de13ad4c5847d7751c7ceb07f Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Mon, 20 Jun 2022 13:51:05 +0200 Subject: Write on IPFS vs. S3 --- content/blog/2022-ipfs/index.md | 61 +++++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 12 deletions(-) diff --git a/content/blog/2022-ipfs/index.md b/content/blog/2022-ipfs/index.md index 152e0a5..ec1649a 100644 --- a/content/blog/2022-ipfs/index.md +++ b/content/blog/2022-ipfs/index.md @@ -7,10 +7,12 @@ date=2022-06-09 *Once you have spawned your Garage cluster, you might be interested in finding ways to share efficiently your content with the rest of the world, such as by joining federated platforms. In this blog post, we experiment with interconnecting the InterPlanetary File System (IPFS) daemon with Garage. -We discuss the different bottlenecks and limitations of the software stack as it is currently available.* +We discuss the different bottlenecks and limitations of the software stack in its current state.* +--- + -However, you would probably not rely on BitTorrent to durably store your encrypted holiday pictures you shared with your friends, +However, you would probably not rely on BitTorrent to durably store the encrypted holiday pictures you shared with your friends, as content on the BitTorrent tends to vanish when no one in the network has a copy of it anymore. The same applies to IPFS. -If at some time, everyone has a copy of the pictures on their hard disk, people might delete these copies after a while without you knowing it. -You also can't easily collaborate to share this common treasure. For example, there is no automatic way to say that Alice and Bob +Even if at some time everyone has a copy of the pictures on their hard disk, people might delete these copies after a while without you knowing it. +You also can't easily collaborate on storing this common treasure. For example, there is no automatic way to say that Alice and Bob are in charge of storing the first half of the archive while Charlie and Eve are in charge of the second half. ➡️ **IPFS is designed to deliver content.** -*Note: the IPFS project has another project named [IPFS Cluster](https://cluster.ipfs.io/) that allow servers to collaborate on hosting IPFS content. +*Note: the IPFS project has another project named [IPFS Cluster](https://cluster.ipfs.io/) that allows servers to collaborate on hosting IPFS content. [Resilio](https://www.resilio.com/individuals/) and [Syncthing](https://syncthing.net/) both feature protocols inspired by BitTorrent to synchronize a tree of your file system between multiple computers. Reviewing these solutions is out of the scope of this article, feel free to try them by yourself!* -Garage, on the contrary, is designed to spread automatically your content over all your available nodes, in a manner that makes the best possible use of your storage space. +Garage, on the contrary, is designed to automatically spread your content over all your available nodes, in a manner that makes the best possible use of your storage space. At the same time, it ensures that your content is always replicated exactly 3 times across the cluster (or less if you change a configuration parameter), on different geographical zones when possible. @@ -119,7 +121,8 @@ As a comparison, this whole webpage, with its pictures, triggers around 10 reque I think we can conclude that this first try was a failure. The S3 storage plugin for IPFS does too many request and would need some important work to be optimized. -However, we should not give up too fast, because the people behind Peergos are known to run their software based on IPFS in production with an S3 backend. +However, we are aware that the people behind Peergos are known to run their software based on IPFS in production with an S3 backend, +so we should not give up too fast. ## Try #2: Peergos over Garage @@ -141,7 +144,7 @@ I was able to upload my file, see it in the interface, create a link to share it ![A screenshot of the Peergos interface](./upload.png) At the same time, the fans of my computer started to become a bit loud! -A quick look at Grafana shows that Garage is still very busy: +A quick look at Grafana showed again a very active Garage: ![Screenshot of a grafana plot showing requests per second over time](./grafa.png)
Legend: y axis = requests per 10 seconds on log(10) scale, x axis = time

@@ -156,7 +159,7 @@ The `OPTIONS` HTTP verb is here because we use the direct access feature of Peer meaning that our browser is talking directly to Garage and has to use CORS to validate requests for security. Internally, IPFS splits files in blocks of less than 256 kB. My picture is thus split in 2 blocks, requiring 2 requests over Garage to fetch it. -But even by knowing that IPFS split files in small blocks, I can't explain why we have so many `GetObject` requests. +But even knowing that IPFS splits files in small blocks, I can't explain why we have so many `GetObject` requests. ## Try #3: Optimizing IPFS @@ -165,7 +168,7 @@ Routing = dhtclient ![](./grafa2.png) --> -We have seen in our 2 previous tries that the main source of load was the federation, and more especially, the DHT server. +We have seen in our 2 previous tries that the main source of load was the federation, and in particular the DHT server. In this section, we'd like to artificially remove this problem from the equation by preventing our IPFS node from federating and see what pressure is put by Peergos alone on our local cluster. @@ -198,9 +201,43 @@ From a theoretical perspective, it is still higher than the optimal number of re On S3, storing a file, downloading a file and listing available files are all actions that can be done in a single request. Even if all requests don't have the same cost on the cluster, processing a request has a non-negligible fixed cost. -## S3 and IPFS are incompatible? +## Are S3 and IPFS incompatible? + +Tweaking IPFS in order to try and make it work on an S3 backend is all and good, +but in some sense, the assumptions made by IPFS are funamentally incompatible with using S3 as a block storage. + +First, data on IPFS is split in relatively small chunks: all IPFS blocks must be less than 1 MB, with most being 256 KB or less. +This means that large files or complex directory hierarchies will need thousands of blocks to be stored, +each of which is mapped to a single object in the S3 storage back-end. +On the other side, S3 implementations such as Garage are made to handle very large objects efficiently, +and they also provide their own primitives for rapidly listing all the objects present in a bucket or a directory. +There is thus a huge loss in performance when data is stored in IPFS's block format, because this format does not +take advantage of the optimizations provided by S3 back-ends in their standard usage scenarios. Instead, it +requires storing and retrieving thousands of small S3 objects even for very simple operations such +as retrieving a file or listing a directory, incurring a fixed overhead each time. + +This problem is compounded by the design of the IPFS data exchange protocol, +in which nodes may request any data blocks to any other node in the network +in its quest to answer a user's request (like retrieving a file, etc.). +When a node is missing a file or a directory it wants to read, it has to do as many requests to other nodes +as there are IPFS blocks in the object to be read. +On the receiving end, this means that any fully-fledged IPFS node has to answer large numbers +of requests for blocks required by users everywhere on the network, which is what we observed in our experiment above. +We were however surprised to observe that many requests comming from the IPFS network were for blocks +which our node wasn't locally storing a copy of: this means that somewhere in the IPFS protocol, an overly optimistic +assumption is made on where data could be found in the network, and this ends up translating in many requests +between nodes that return negative results. +When IPFS blocks are stored on a local filesystem, answering these requests fast might be possible. +However when using an S3 server as a storage back-end, this becomes prohibitively costly. + +If one wanted to design a distributed storage system for IPFS data blocks, they would probably need to start at a lower level. +Garage itself makes use of a block storage mechanism that allows small-sized blocks to be stored on a cluster and accessed +rapidly by nodes that need to access them. +However passing through the entire abstraction that provides an S3 API is wastefull and redundant, as this API is +designed to provide advanced functionnality such as mutating objects, associating metadata with objects, listing objects, etc. +Plugging the IPFS daemon directly into a lower-level distributed block storage like +Garage's might yield way better results by bypassing all of this complexity. -*Text by Alex* ## Conclusion -- cgit v1.2.3