diff options
| -rw-r--r-- | README.md | 80 | ||||
| -rw-r--r-- | TODO | 91 |
2 files changed, 145 insertions, 26 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..925036c --- /dev/null +++ b/README.md @@ -0,0 +1,80 @@ +What is Shard? +=============== + +*WARNING:* This is a primer on what I want Shard to become. As of today I'm only +starting work on this project and there is almost nothing to see. I hope this +will evolve and approach at least a significant portion of the goals identified +below. + +Shard is a decentralized communication system adapted to public and private +discussion and collaboration. + +The system is based on distributed objects called "shards" of different types +that users share and collaborate on. + +Shards are made to be usable offline or on small networks disconnected from the +broader Internet, with updates becoming synchronized automatically whenever +peers are able to communicate. + + +Shard compared to other systems +=============================== + +Shard is born from the synthesis of ideas already found in other networks, but +aiming to fix some of their shortcomings. + +Shard vs. ZeroNet +----------------- + +Just like ZeroNet, Shard will enable users to run public applications with +distributed collaboration. We wish to improve on ZeroNet in the following +significant manners: + +- Full support for one-to-one and groupwise private communication and + collaboration. Examples of such in ZeroNet are quite limited, for example + ZeroMail enables user to exchange private emails but requires all peers + participating in the ZeroMail system to store all messages sent by + everyone. We want to make this work better by only requiring participants + to store the private information. +- Collaboration on very large databases. We wish to be able to store big + collaborative databases (e.g. wikipedia) in a distributed way, so that + it can be run in a decentralized fashion without requiring everyone to store + a full copy. +- Performance improvements. This is done by using a language that is by + design well adapted to network programing with many things going on at once + (Elixir/Erlang), as well as by optimizing our data stores to enable fast + querying. + +Shard vs. Secure Scuttlebutt +---------------------------- + +Just like SSB, Shard will enable users to work off grid and exchange data +when they happen to communicate. We wish to improve upon the following +aspects of SSB: + +- More granularity on what peers store or not, by splitting a user's + contribution in different "shards" that correspond to different topics. + Users can still chose to replicate the shards that their friends use so + that they have more options to synchronize and backup their data, but + this is not required as it is in SSB where you have to replicate all the + data of your friends and their friends. +- Decentralized discovery: we don't want to rely on centralized servers such + as SSB pubs. We want to use DHT technology as a primary means of discovering + new people and finding contents. +- We wish to be very general from the start, and not limited to social-media-like + conversations. +- We want excellent support for one-to-one and groupwise communications. + +Shard vs. Matrix +---------------- + +We aim to be much more general than a chat room, supporting collaborative +edition, file sharing, any kinds of databases, etc. + +We also aim to work in a completely distributed fashion and not require +federation servers. + +Shard vs. Mastodon / GNU Social +------------------------------- + +Same remarks than for Matrix. @@ -1,10 +1,35 @@ +KEEP IT SIMPLE, STUPID! +----------------------- + +Ideas are all over the place for this project. When adding something, let's +make sure it's reasonnable to do so and doesn't burden the user with some +overcomplicated concepts. We want the minimal and most elegant solution that +achieves our goals. + + +TASK LIST +========= + +- Highest priority: pagehash, sign +- Medium priority: dht, ep, dep + +Fix page hash semantics (pagehash, EASY) +----------------------- + +Store and transmit the term binary and not the term itself so that +we are sure serialization is unique and hash is the same everywhere. + +Terminology: stop using the word "block", use "page" everywhere. + + DHT to find peers for a given shard (dht, EASY) ----------------------------------- First option: use a library for MLDHT, makes everything simple but makes us use UDP which does not work with Tor (can fix this later). -Second option: custom DHT protocol. +Second option: custom DHT protocol (we probably won't be doing this +anytime soon, if ever at all) Epidemic broadcast (ep, EASY) @@ -23,31 +48,6 @@ Best option: those that we are connected to + some random to reach a quota (for example 10 or so) -Block store root & GC handling (gc, QUITE EASY) ------------------------------- - -We want the block store app to be aware of what blocks are needed -or not. The Page protocol already implements dependencies between -blocks. - -The block store keeps all pages that have been put for a given -delay. Once the delay is passed, the pages are purged if they are -not required by a root we want to keep. - - -Partial sync/background pull for big objects (bg, req: gc, QUITE EASY) --------------------------------------------- - -Implement the copy protocol as a lazy call that launches the copy -in background. - -Remove the callback possibility in MerkleSearchTree.merge so that -pulling all the data is not required for a merge. The callback can -be only called on the items that are new in the last n (ex. 100) -items of the resulting tree, this is not implemented in the MST but -in the app that uses it since it is application specific. - - Partial merges, background pulls, caching (cache, req: bg, HARD) ----------------------------------------- @@ -109,3 +109,42 @@ Automated algorithms that take account the trust values in access control decisions (obviously these can only run when an identity with admin privilege is running). + +Shard dependency management (dep, MED) +--------------------------- + +Some Shards may pull other shards in, under certain conditions. For example +a stored folder shard will just be a list of other shards that we all pull in. + +We want a way to have toplevel shards, shards that are dependencies of +toplevel shards, and shards that we keep for a number of days as a cache but +will expire automatically. + + +COMPLETED TASKS +=============== + +Block store root & GC handling (gc, QUITE EASY) +------------------------------ + +We want the block store app to be aware of what blocks are needed +or not. The Page protocol already implements dependencies between +blocks. + +The block store keeps all pages that have been put for a given +delay. Once the delay is passed, the pages are purged if they are +not required by a root we want to keep. + + +Partial sync/background pull for big objects (bg, req: gc, QUITE EASY) +-------------------------------------------- + +Implement the copy protocol as a lazy call that launches the copy +in background. + +Remove the callback possibility in MerkleSearchTree.merge so that +pulling all the data is not required for a merge. The callback can +be only called on the items that are new in the last n (ex. 100) +items of the resulting tree, this is not implemented in the MST but +in the app that uses it since it is application specific. + |