diff options
| -rw-r--r-- | README.md | 102 | ||||
| -rw-r--r-- | TODO | 31 | ||||
| -rw-r--r-- | shard/lib/app/chat.ex | 5 | ||||
| -rw-r--r-- | shard/lib/app/directory.ex | 4 | ||||
| -rw-r--r-- | shard/lib/app/file.ex | 6 | ||||
| -rw-r--r-- | shard/lib/app/identity.ex | 4 | ||||
| -rw-r--r-- | shard/lib/app/pagestore.ex | 16 | ||||
| -rw-r--r-- | shard/lib/cli/cli.ex | 10 | ||||
| -rw-r--r-- | shard/lib/manager.ex | 31 | ||||
| -rw-r--r-- | shardweb/lib/controllers/directory_controller.ex | 6 | ||||
| -rw-r--r-- | shardweb/lib/controllers/page_controller.ex | 8 | ||||
| -rw-r--r-- | shardweb/lib/router.ex | 1 | ||||
| -rw-r--r-- | shardweb/lib/templates/page/shard_list.html.eex | 15 |
13 files changed, 175 insertions, 64 deletions
@@ -1,10 +1,10 @@ What is Shard? =============== -*WARNING:* This is a primer on what I want Shard to become. As of today I'm +**WARNING: This is a primer on what I want Shard to become. As of today I'm only starting work on this project and there is almost nothing to see. I hope this will evolve and approach at least a significant portion of the goals -identified below. +identified below.** Shard is a decentralized communication system adapted to public and private discussion and collaboration. @@ -48,9 +48,9 @@ significant manners: Shard vs. Secure Scuttlebutt ---------------------------- -Just like SSB, Shard will enable users to work off grid and exchange data -when they happen to communicate. We wish to improve upon the following -aspects of SSB: +Just like SSB, Shard will enable users to work off grid and exchange data when +a network connection happens to be available. We wish to improve upon the +following aspects of SSB: - More granularity on what peers store or not, by splitting a user's contribution in different "shards" that correspond to different topics. @@ -63,7 +63,7 @@ aspects of SSB: new people and finding contents. - We wish to be very general from the start, and not limited to social-media-like conversations. -- We want excellent support for one-to-one and groupwise communications. +- We want excellent support for private one-to-one and groupwise communications. Shard vs. Matrix ---------------- @@ -80,6 +80,27 @@ Shard vs. Mastodon / GNU Social Same remarks than for Matrix. +Current status +============== + +What is available +----------------- + +* Chat rooms (public and private) with full history and efficient data structure for + retrieving missing messages after disconnection +* File upload (public only) +* Directories as collections of links to shards + +What is missing +--------------- + +See `TODO` file for more details. + +* Finding peers via DHT (very easy to add) +* Invite/notification system +* Good access control +* More applications + How to use it? ============== @@ -103,61 +124,77 @@ Then, clone the Shard repository using git: git clone git://adnab.me/shard.git ``` -### CLI version only -Enter the shard directory and create a data folder. +### Web UI version + +Enter the shard directory and create a data2 folder. ``` cd shard -mkdir data # for CLI use only +mkdir data2 # for web UI ``` -Enter the shard subdirectory and install Elixir dependencies. +Enter the shardweb subdirectory and install Elixir dependencies. ``` -cd shard +cd shardweb mix deps.get mix compile ``` +Build the web assets: + +``` +cd assets +npm install +cd .. +``` + **The default configuration assumes you have cloned the shard repo in your -`$HOME`. If this is not the case, ajust `config/config.exs` accordingly.** +`$HOME`. If this is not the case, ajust the data path parameter in +`shardweb/config/config.exs` accordingly.** -### Web UI version +### CLI version only -Enter the shard directory and create a data2 folder. +Enter the shard directory and create a data folder. ``` cd shard -mkdir data2 # for web UI +mkdir data # for CLI use only ``` -Enter the shardweb subdirectory and install Elixir dependencies. +Enter the shard subdirectory and install Elixir dependencies. ``` -cd shardweb +cd shard mix deps.get mix compile ``` -Build the web assets: +**The default configuration assumes you have cloned the shard repo in your +`$HOME`. If this is not the case, ajust the data path parameter in +`shard/config/config.exs` accordingly.** + + +Web Ui usage +------------ + +Once you have completed the installation steps above, Shard web can be started +by running the following in the `shardweb` directory of the repo: ``` -cd assets -npm install -cd .. +iex -S mix phx.server ``` -**The default configuration assumes you have cloned the shard repo in your -`$HOME`. If this is not the case, ajust `config/config.exs` accordingly.** +The UI will become accessible at address `http://127.0.0.1:4000`. CLI usage --------- Once you have completed the installation steps above, Shard can be started by -running the following in the `shard/shard` directory: +running the following in the `shard` directory of the repo: ``` iex -S mix @@ -177,18 +214,7 @@ This CLI supports a few basic commands: - `/nick my_nickname`: set nickname - `/list`: list channels we are currently connected to - `/hist`: show recent messages on current channel -- `/pm nickname`: enter private conversation with someone +- `/pm nickname1 [nickname2] [...]`: enter private conversation with someone +- `/send_file path`: make file available on the network and send link to current chat room. + **WARNING: all files are publicly available for now, even if they are sent in a private chat room.** - `/quit`: return to iex prompt - - -Web Ui usage ------------- - -Once you have completed the installation steps above, Shard web can be started -by running the following in the `shard/shardweb` directory: - -``` -iex -S mix phx.server -``` - -The UI will become accessible at address `http://127.0.0.1:4000`. @@ -10,13 +10,16 @@ achieves our goals. TASK LIST ========= -- High priority: invite -- Medium priority: dht, dep +- High priority: invite, dht, groups +- Medium priority: net -Invitation system (invite, EASY) +Invitation system (invite, MED) -------------------------------- +Current issue: if a user PM's another user, that user does not know it! +He has to pull the PM shard voluntarily to get the messages. + A user may generate invitation tokens: an invite token is a signature of the invited user's pk by the inviter user's sk. The user that redeems the token writes some data that is saved temporarily in the user's identity shard, which @@ -39,7 +42,6 @@ Ex: keep 100 total open connections that are sampled by proximity on the set of plus 2 or 5 full random for all shards. - DHT to find peers for a given shard (dht, EASY) ----------------------------------- @@ -99,16 +101,6 @@ access control decisions (obviously these can only run when an identity with admin privilege is running). -Shard lifetime and dependency management (dep, MED) ----------------------------------------- - -Some Shards may pull other shards in, under certain conditions. For example -a stored folder shard will just be a list of other shards that we all pull in. - -We want a way to have toplevel shards, shards that are dependencies of -toplevel shards, and shards that we keep for a number of days as a cache but -will expire automatically. - COMPLETED TASKS =============== @@ -187,3 +179,14 @@ Best option: those that we are connected to + some random to reach a quota (for example 10 or so) Implementation: netgroups (lib/net/groups.ex) + + +Shard lifetime and dependency management (dep, MED) +---------------------------------------- + +Some Shards may pull other shards in, under certain conditions. For example +a stored folder shard will just be a list of other shards that we all pull in. + +We want a way to have toplevel shards, shards that are dependencies of +toplevel shards, and shards that we keep for a number of days as a cache but +will expire automatically. diff --git a/shard/lib/app/chat.ex b/shard/lib/app/chat.ex index 2795153..ff0c97d 100644 --- a/shard/lib/app/chat.ex +++ b/shard/lib/app/chat.ex @@ -131,6 +131,11 @@ defmodule SApp.Chat do {:reply, state.manifest, state} end + def handle_call(:delete_shard, _from, state) do + GenServer.call(state.store, :delete_store) + {:stop, :normal, :ok, state} + end + def handle_call({:read_history, top_bound, num}, _from, state) do ret = MST.last(state.mst, top_bound, num) {:reply, ret, state} diff --git a/shard/lib/app/directory.ex b/shard/lib/app/directory.ex index e9482f7..cbea8c3 100644 --- a/shard/lib/app/directory.ex +++ b/shard/lib/app/directory.ex @@ -64,6 +64,10 @@ defmodule SApp.Directory do {:reply, state.manifest, state} end + def handle_call(:delete_shard, _from, state) do + {:stop, :normal, :ok, state} + end + def handle_call(:get_items, _from, state) do {:reply, SData.SignRev.get(state.items), state} end diff --git a/shard/lib/app/file.ex b/shard/lib/app/file.ex index ce28beb..e2a9798 100644 --- a/shard/lib/app/file.ex +++ b/shard/lib/app/file.ex @@ -84,6 +84,12 @@ defmodule SApp.File do {:reply, state.manifest, state} end + def handle_call(:delete_shard, _from, state) do + GenServer.call(state.store, :delete_store) + File.rm(state.path) + {:stop, :normal, :ok, state} + end + def handle_call(:get_info, _from, state) do reply = cond do state.info != nil and GenServer.call(state.store, {:have_rec, state.info.merkle_root}) -> diff --git a/shard/lib/app/identity.ex b/shard/lib/app/identity.ex index 59a4b90..78abbe7 100644 --- a/shard/lib/app/identity.ex +++ b/shard/lib/app/identity.ex @@ -65,6 +65,10 @@ defmodule SApp.Identity do {:reply, state.manifest, state} end + def handle_call(:delete_shard, _from, state) do + {:stop, :normal, :ok, state} + end + def handle_call(:get_info, _from, state) do {:reply, SData.SignRev.get(state.state), state} end diff --git a/shard/lib/app/pagestore.ex b/shard/lib/app/pagestore.ex index 5165b84..3cda51d 100644 --- a/shard/lib/app/pagestore.ex +++ b/shard/lib/app/pagestore.ex @@ -25,7 +25,7 @@ defmodule SApp.PageStore do @max_failures 4 # Maximum of peers that reply not_found before we abandon defmodule State do - defstruct [:shard_id, :path, :netgroup, :store, :reqs, :retries] + defstruct [:shard_id, :path, :netgroup, :store, :reqs, :retries, :store_path] end @@ -36,15 +36,21 @@ defmodule SApp.PageStore do def init([shard_id, path, netgroup]) do Shard.Manager.dispatch_to(shard_id, path, self()) - store_path = [Application.get_env(:shard, :data_path), "#{shard_id|>Base.encode16}.#{path}"] |> Path.join |> String.to_atom - {:ok, store} = :dets.open_file store_path, [type: :set] + store_path = [Application.get_env(:shard, :data_path), "#{shard_id|>Base.encode16}.#{path}"] |> Path.join + {:ok, store} = :dets.open_file(String.to_atom(store_path), [type: :set]) Process.send_after(self(), :clean_cache, 1000) - {:ok, %State{shard_id: shard_id, path: path, netgroup: netgroup, store: store, reqs: %{}, retries: %{}}} + {:ok, %State{shard_id: shard_id, path: path, netgroup: netgroup, store: store, reqs: %{}, retries: %{}, store_path: store_path}} end + def handle_call(:delete_store, _from, state) do + :dets.close state.store + File.rm state.store_path + {:stop, :normal, :ok, state} + end + def handle_call({:get, key, prefer_ask}, from, state) do case :dets.lookup state.store, key do [{_, _, bin}] -> @@ -84,9 +90,11 @@ defmodule SApp.PageStore do case :dets.lookup state.store, hash do [] -> :dets.insert state.store, {hash, {:cached, System.os_time(:seconds) + @cache_ttl}, bin} + :dets.sync state.store nil [{_, why}] -> :dets.insert state.store, {hash, why, bin} + :dets.sync state.store why [{_, _, _}] -> nil diff --git a/shard/lib/cli/cli.ex b/shard/lib/cli/cli.ex index 380282d..54b882f 100644 --- a/shard/lib/cli/cli.ex +++ b/shard/lib/cli/cli.ex @@ -94,8 +94,16 @@ defmodule SCLI do end defp handle_command(state, ["list"]) do - IO.puts "List of known channels:" + IO.puts "Private conversations:" + for {_chid, %SApp.Chat.PrivChat.Manifest{pk_list: pk_list}, _} <- Shard.Manager.list_shards do + pk_list + |> Enum.filter(&(&1 != state.pk)) + |> Enum.map(fn pk -> "#{SApp.Identity.get_nick pk} #{Shard.Keys.pk_display pk}" end) + |> Enum.join(", ") + |> IO.puts + end + IO.puts "Public channels we are connected to:" for {_chid, %SApp.Chat.Manifest{channel: chan}, _} <- Shard.Manager.list_shards do IO.puts "##{chan}" end diff --git a/shard/lib/manager.ex b/shard/lib/manager.ex index c178e2f..c3897a3 100644 --- a/shard/lib/manager.ex +++ b/shard/lib/manager.ex @@ -82,6 +82,27 @@ defmodule Shard.Manager do {:reply, pid, state} end + def handle_call({:delete, shard_id}, _from, state) do + case :dets.lookup(@shard_db, shard_id) do + [] -> + {:reply, {:error, :not_found}, state} + [{^shard_id, manifest, {:cached, _}, _}] -> + pid = case :ets.lookup(:shard_procs, {shard_id, nil}) do + [] -> + {:ok, pid} = apply(Shard.Manifest.module(manifest), :start_link, [manifest]) + pid + [{{^shard_id, nil}, pid}] -> + :ets.delete(:shard_procs, {shard_id, nil}) + pid + end + GenServer.call(pid, :delete_shard) + :dets.delete(@shard_db, shard_id) + {:reply, :ok, state} + [{^shard_id, _, _, _}] -> + {:reply, {:error, :pinned}, state} + end + end + def handle_cast({:dispatch_to, shard_id, path, pid}, state) do :ets.insert(:shard_procs, { {shard_id, path}, pid }) state = Map.put(state, pid, {shard_id, path}) @@ -105,6 +126,7 @@ defmodule Shard.Manager do case :dets.lookup(@shard_db, shard_id) do [{^shard_id, manifest, why_have_it, _old_state}] -> :dets.insert(@shard_db, {shard_id, manifest, why_have_it, shst}) + :dets.sync(@shard_db) end {:noreply, state} end @@ -169,7 +191,7 @@ defmodule Shard.Manager do for [id] <- shards do case :ets.lookup(:shard_procs, {id, nil}) do [{{^id, nil}, pid}] -> - GenServer.cast(pid, :delete_shard) + GenServer.call(pid, :delete_shard) _ -> nil end :dets.delete(@shard_db, id) @@ -366,6 +388,13 @@ defmodule Shard.Manager do end @doc""" + Delete a shard + """ + def delete(shard_id) do + GenServer.call(__MODULE__, {:delete, shard_id}) + end + + @doc""" Return the list of all shards. Returns a list of tuples: {id, manifest, why_have_it} diff --git a/shardweb/lib/controllers/directory_controller.ex b/shardweb/lib/controllers/directory_controller.ex index d1b7136..ecb9c97 100644 --- a/shardweb/lib/controllers/directory_controller.ex +++ b/shardweb/lib/controllers/directory_controller.ex @@ -9,7 +9,7 @@ defmodule ShardWeb.DirectoryController do view(conn, false, args) end - def view(conn, public, %{"owner" => owner, "name" => name}=args) do + def view(conn, public, %{"owner" => owner, "name" => name}) do owner = Base.decode16! owner manifest = %SApp.Directory.Manifest{public: public, owner: owner, name: name} shard = SData.term_hash manifest @@ -28,7 +28,6 @@ defmodule ShardWeb.DirectoryController do dir_public = (dir_public == "true") manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name} - shard = SData.term_hash manifest pid = Shard.Manager.find_or_start manifest item_manifest = ShardURI.to_manifest(add_uri) @@ -41,7 +40,6 @@ defmodule ShardWeb.DirectoryController do dir_public = (dir_public == "true") manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name} - shard = SData.term_hash manifest pid = Shard.Manager.find_or_start manifest SApp.Directory.rm_item(pid, rm_name) @@ -54,7 +52,6 @@ defmodule ShardWeb.DirectoryController do item_stored = (item_stored == "true") manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name} - shard = SData.term_hash manifest pid = Shard.Manager.find_or_start manifest SApp.Directory.set_stored(pid, item_name, item_stored) @@ -68,7 +65,6 @@ defmodule ShardWeb.DirectoryController do dir_public = (dir_public == "true") manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name} - shard = SData.term_hash manifest pid = Shard.Manager.find_or_start manifest {:ok, file_manifest, _file_pid} = SApp.File.create(path, mime_type) diff --git a/shardweb/lib/controllers/page_controller.ex b/shardweb/lib/controllers/page_controller.ex index 5c90416..baf72b9 100644 --- a/shardweb/lib/controllers/page_controller.ex +++ b/shardweb/lib/controllers/page_controller.ex @@ -9,6 +9,14 @@ defmodule ShardWeb.PageController do render conn, "shard_list.html" end + def shard_action(conn, %{"shard_id" => shard_id, "action" => action}) do + shard_id = Base.decode16! shard_id + case action do + "delete" -> Shard.Manager.delete(shard_id) + end + redirect conn, to: page_path(conn, :shard_list) + end + def add_peer(conn, _params) do try do ip = conn.params["ip"] diff --git a/shardweb/lib/router.ex b/shardweb/lib/router.ex index a6a61f9..0b1f686 100644 --- a/shardweb/lib/router.ex +++ b/shardweb/lib/router.ex @@ -19,6 +19,7 @@ defmodule ShardWeb.Router do pipe_through :browser # Use the default browser stack get "/", PageController, :shard_list + post "/", PageController, :shard_action get "/peers", PageController, :peer_list post "/peer/add", PageController, :add_peer diff --git a/shardweb/lib/templates/page/shard_list.html.eex b/shardweb/lib/templates/page/shard_list.html.eex index 3fffef7..79fd794 100644 --- a/shardweb/lib/templates/page/shard_list.html.eex +++ b/shardweb/lib/templates/page/shard_list.html.eex @@ -23,9 +23,10 @@ <tr> <th></th> <th>Shard</th> + <th></th> <th>URI</th> </tr> - <%= for {_id, manifest, why_have_it} <- shard_list() do %> + <%= for {id, manifest, why_have_it} <- shard_list() do %> <tr> <td> <%= case why_have_it do %> @@ -37,6 +38,18 @@ <td> <%= render "shard_entry.html", conn: @conn, manifest: manifest, pk: @pk %> </td> + <td> + <%= case why_have_it do %> + <% {:cached, _} -> %> + <%= form_for @conn, page_path(@conn, :shard_action), [class: "form-inline", style: "display: inline", onsubmit: "return confirm('Delete this shard and all its data?');"], fn f -> %> + <%= hidden_input f, :shard_id, value: id |> Base.encode16 %> + <%= hidden_input f, :action, value: "delete" %> + <%= submit "Delete", [class: "btn btn-xs btn-danger"] %> + <% end %> + <% _ -> %> + <% end %> + + </td> <td><small class="shard_uri"><%= manifest |> ShardURI.from_manifest %></small></td> </tr> <% end %> |