Add some ideas

2 files changed, 145 insertions, 26 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..925036c
--- /dev/null
+++ b/README.md
@@ -0,0 +1,80 @@
+What is Shard?
+===============
+
+*WARNING:* This is a primer on what I want Shard to become. As of today I'm only
+starting work on this project and there is almost nothing to see. I hope this
+will evolve and approach at least a significant portion of the goals identified
+below.
+
+Shard is a decentralized communication system adapted to public and private
+discussion and collaboration.
+
+The system is based on distributed objects called "shards" of different types
+that users share and collaborate on.
+
+Shards are made to be usable offline or on small networks disconnected from the
+broader Internet, with updates becoming synchronized automatically whenever
+peers are able to communicate.
+
+
+Shard compared to other systems
+===============================
+
+Shard is born from the synthesis of ideas already found in other networks, but
+aiming to fix some of their shortcomings.
+
+Shard vs. ZeroNet
+-----------------
+
+Just like ZeroNet, Shard will enable users to run public applications with
+distributed collaboration. We wish to improve on ZeroNet in the following
+significant manners:
+
+- Full support for one-to-one and groupwise private communication and
+  collaboration. Examples of such in ZeroNet are quite limited, for example
+  ZeroMail enables user to exchange private emails but requires all peers
+  participating in the ZeroMail system to store all messages sent by
+  everyone. We want to make this work better by only requiring participants
+  to store the private information.
+- Collaboration on very large databases. We wish to be able to store big
+  collaborative databases (e.g. wikipedia) in a distributed way, so that
+  it can be run in a decentralized fashion without requiring everyone to store
+  a full copy.
+- Performance improvements. This is done by using a language that is by
+  design well adapted to network programing with many things going on at once
+  (Elixir/Erlang), as well as by optimizing our data stores to enable fast
+  querying.
+
+Shard vs. Secure Scuttlebutt
+----------------------------
+
+Just like SSB, Shard will enable users to work off grid and exchange data
+when they happen to communicate. We wish to improve upon the following
+aspects of SSB:
+
+- More granularity on what peers store or not, by splitting a user's
+  contribution in different "shards" that correspond to different topics.
+  Users can still chose to replicate the shards that their friends use so
+  that they have more options to synchronize and backup their data, but
+  this is not required as it is in SSB where you have to replicate all the
+  data of your friends and their friends.
+- Decentralized discovery: we don't want to rely on centralized servers such
+  as SSB pubs. We want to use DHT technology as a primary means of discovering
+  new people and finding contents.
+- We wish to be very general from the start, and not limited to social-media-like
+  conversations.
+- We want excellent support for one-to-one and groupwise communications.
+
+Shard vs. Matrix
+----------------
+
+We aim to be much more general than a chat room, supporting collaborative
+edition, file sharing, any kinds of databases, etc.
+
+We also aim to work in a completely distributed fashion and not require
+federation servers.
+
+Shard vs. Mastodon / GNU Social
+-------------------------------
+
+Same remarks than for Matrix.
diff --git a/TODO b/TODO
index fafcdf7..2034785 100644
--- a/TODO
+++ b/TODO
@@ -1,10 +1,35 @@
+KEEP IT SIMPLE, STUPID!
+-----------------------
+
+Ideas are all over the place for this project. When adding something, let's
+make sure it's reasonnable to do so and doesn't burden the user with some
+overcomplicated concepts. We want the minimal and most elegant solution that
+achieves our goals.
+
+
+TASK LIST
+=========
+
+- Highest priority: pagehash, sign
+- Medium priority: dht, ep, dep
+
+Fix page hash semantics (pagehash, EASY)
+-----------------------
+
+Store and transmit the term binary and not the term itself so that
+we are sure serialization is unique and hash is the same everywhere.
+
+Terminology: stop using the word "block", use "page" everywhere.
+
+
 DHT to find peers for a given shard (dht, EASY)
 -----------------------------------
 
 First option: use a library for MLDHT, makes everything simple but
 makes us use UDP which does not work with Tor (can fix this later).
 
-Second option: custom DHT protocol.
+Second option: custom DHT protocol (we probably won't be doing this
+anytime soon, if ever at all)
 
 
 Epidemic broadcast (ep, EASY)
@@ -23,31 +48,6 @@ Best option: those that we are connected to + some random to
 reach a quota (for example 10 or so)
 
 
-Block store root & GC handling (gc, QUITE EASY)
-------------------------------
-
-We want the block store app to be aware of what blocks are needed
-or not. The Page protocol already implements dependencies between
-blocks. 
-
-The block store keeps all pages that have been put for a given
-delay. Once the delay is passed, the pages are purged if they are
-not required by a root we want to keep.
-
-
-Partial sync/background pull for big objects (bg, req: gc, QUITE EASY)
---------------------------------------------
-
-Implement the copy protocol as a lazy call that launches the copy
-in background.
-
-Remove the callback possibility in MerkleSearchTree.merge so that
-pulling all the data is not required for a merge. The callback can
-be only called on the items that are new in the last n (ex. 100)
-items of the resulting tree, this is not implemented in the MST but
-in the app that uses it since it is application specific.
-
-
 Partial merges, background pulls, caching (cache, req: bg, HARD)
 -----------------------------------------
 
@@ -109,3 +109,42 @@ Automated algorithms that take account the trust values in
 access control decisions (obviously these can only run when an
 identity with admin privilege is running).
 
+
+Shard dependency management (dep, MED)
+---------------------------
+
+Some Shards may pull other shards in, under certain conditions. For example
+a stored folder shard will just be a list of other shards that we all pull in.
+
+We want a way to have toplevel shards, shards that are dependencies of
+toplevel shards, and shards that we keep for a number of days as a cache but
+will expire automatically.
+
+
+COMPLETED TASKS
+===============
+
+Block store root & GC handling (gc, QUITE EASY)
+------------------------------
+
+We want the block store app to be aware of what blocks are needed
+or not. The Page protocol already implements dependencies between
+blocks. 
+
+The block store keeps all pages that have been put for a given
+delay. Once the delay is passed, the pages are purged if they are
+not required by a root we want to keep.
+
+
+Partial sync/background pull for big objects (bg, req: gc, QUITE EASY)
+--------------------------------------------
+
+Implement the copy protocol as a lazy call that launches the copy
+in background.
+
+Remove the callback possibility in MerkleSearchTree.merge so that
+pulling all the data is not required for a merge. The callback can
+be only called on the items that are new in the last n (ex. 100)
+items of the resulting tree, this is not implemented in the MST but
+in the app that uses it since it is application specific.
+