What is Shard?

WARNING: This is a primer on what I want Shard to become. As of today I'm only starting work on this project and there is almost nothing to see. I hope this will evolve and approach at least a significant portion of the goals identified below.

Shard is a decentralized communication system adapted to public and private discussion and collaboration.

The system is based on distributed objects called "shards" of different types that users share and collaborate on.

Shards are made to be usable offline or on small networks disconnected from the broader Internet, with updates becoming synchronized automatically whenever peers are able to communicate.

Shard compared to other systems

Shard is born from the synthesis of ideas already found in other networks, but aiming to fix some of their shortcomings.

Shard vs. ZeroNet

Just like ZeroNet, Shard will enable users to run public applications with distributed collaboration. We wish to improve on ZeroNet in the following significant manners:

• Full support for one-to-one and groupwise private communication and collaboration. Examples of such in ZeroNet are quite limited, for example ZeroMail enables user to exchange private emails but requires all peers participating in the ZeroMail system to store all messages sent by everyone. We want to make this work better by only requiring participants to store the private information.
• Collaboration on very large databases. We wish to be able to store big collaborative databases (e.g. wikipedia) in a distributed way, so that it can be run in a decentralized fashion without requiring everyone to store a full copy.
• Performance improvements. This is done by using a language that is by design well adapted to network programing with many things going on at once (Elixir/Erlang), as well as by optimizing our data stores to enable fast querying.

Shard vs. Secure Scuttlebutt

Just like SSB, Shard will enable users to work off grid and exchange data when a network connection happens to be available. We wish to improve upon the following aspects of SSB:

• More granularity on what peers store or not, by splitting a user's contribution in different "shards" that correspond to different topics. Users can still chose to replicate the shards that their friends use so that they have more options to synchronize and backup their data, but this is not required as it is in SSB where you have to replicate all the data of your friends and their friends.
• Decentralized discovery: we don't want to rely on centralized servers such as SSB pubs. We want to use DHT technology as a primary means of discovering new people and finding contents.
• We wish to be very general from the start, and not limited to social-media-like conversations.
• We want excellent support for private one-to-one and groupwise communications.

Shard vs. Matrix

We aim to be much more general than a chat room, supporting collaborative edition, file sharing, any kinds of databases, etc.

We also aim to work in a completely distributed fashion and not require federation servers.

Shard vs. Mastodon / GNU Social

Same remarks than for Matrix.

Current status

What is available

All of these are rudimentary prototypes at an early stage.

• Chat rooms (public and private) with full history and efficient data structure for retrieving missing messages after disconnection
• File upload (public only)
• Directories as collections of links to shards

What is missing

See TODO file for more details.

• Finding peers via DHT (very easy to add)
• Invite/notification system
• Good access control
• Good networking behind NAT/Firewall
• Automatic discovery on local networks
• More applications

How to use it?

Installation

Shard has been tested under Linux and should probably work also under Mac OS. No idea about MS Windows.

We don't have an automated installation procedure for now. Sorry.

First, install the git version control system, as well as the Erlang and Elixir programming languages. This should be done using your favourite package manager.

If you wish to use the web UI, you must also install nodejs and npm.

Then, clone the Shard repository using git:

git clone git://adnab.me/shard.git

Web UI version

Enter the shard directory and create a data2 folder.

cd shard
mkdir data2     # for web UI

Enter the shardweb subdirectory and install Elixir dependencies.

cd shardweb
mix deps.get
mix compile

Build the web assets:

cd assets
npm install
cd ..

The default configuration assumes you have cloned the shard repo in your $HOME. If this is not the case, ajust the data path parameter in shardweb/config/config.exs accordingly.

CLI version only

Enter the shard directory and create a data folder.

cd shard
mkdir data      # for CLI use only

Enter the shard subdirectory and install Elixir dependencies.

cd shard
mix deps.get
mix compile

The default configuration assumes you have cloned the shard repo in your $HOME. If this is not the case, ajust the data path parameter in shard/config/config.exs accordingly.

Web Ui usage

Once you have completed the installation steps above, Shard web can be started by running the following in the shardweb directory of the repo:

iex -S mix phx.server

The UI will become accessible at address http://127.0.0.1:4000.

CLI usage

Once you have completed the installation steps above, Shard can be started by running the following in the shard directory of the repo:

iex -S mix

Once Shard has started and you have an iex prompt, you can enter the minimalist CLI interface by typing:

SCLI.run

This CLI supports a few basic commands:

• /connect 1.2.3.4 4044: connect to a Shard peer at IP 1.2.3.4 on port 4044 (default port)
• /join test: join chatroom called test
• /nick my_nickname: set nickname
• /list: list channels we are currently connected to
• /hist: show recent messages on current channel
• /pm nickname1 [nickname2] [...]: enter private conversation with someone
• /send_file path: make file available on the network and send link to current chat room. WARNING: all files are publicly available for now, even if they are sent in a private chat room.
• /shards: return the list of all shards on the system
• /info [file_shard_uri]: shows progress of a file download. Starts download if not already started.
• /quit: return to iex prompt