Shard deletion ; update README and TODO

13 files changed, 175 insertions, 64 deletions

diff --git a/README.md b/README.md
index 107d3bd..c247e4e 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,10 @@
 What is Shard?
 ===============
 
-*WARNING:* This is a primer on what I want Shard to become. As of today I'm
+**WARNING: This is a primer on what I want Shard to become. As of today I'm
 only starting work on this project and there is almost nothing to see. I hope
 this will evolve and approach at least a significant portion of the goals
-identified below.
+identified below.**
 
 Shard is a decentralized communication system adapted to public and private
 discussion and collaboration.
@@ -48,9 +48,9 @@ significant manners:
 Shard vs. Secure Scuttlebutt
 ----------------------------
 
-Just like SSB, Shard will enable users to work off grid and exchange data
-when they happen to communicate. We wish to improve upon the following
-aspects of SSB:
+Just like SSB, Shard will enable users to work off grid and exchange data when
+a network connection happens to be available. We wish to improve upon the
+following aspects of SSB:
 
 - More granularity on what peers store or not, by splitting a user's
   contribution in different "shards" that correspond to different topics.
@@ -63,7 +63,7 @@ aspects of SSB:
   new people and finding contents.
 - We wish to be very general from the start, and not limited to social-media-like
   conversations.
-- We want excellent support for one-to-one and groupwise communications.
+- We want excellent support for private one-to-one and groupwise communications.
 
 Shard vs. Matrix
 ----------------
@@ -80,6 +80,27 @@ Shard vs. Mastodon / GNU Social
 Same remarks than for Matrix.
 
 
+Current status
+==============
+
+What is available
+-----------------
+
+* Chat rooms (public and private) with full history and efficient data structure for
+  retrieving missing messages after disconnection
+* File upload (public only)
+* Directories as collections of links to shards
+
+What is missing
+---------------
+
+See `TODO` file for more details.
+
+* Finding peers via DHT (very easy to add)
+* Invite/notification system
+* Good access control
+* More applications
+
 How to use it?
 ==============
 
@@ -103,61 +124,77 @@ Then, clone the Shard repository using git:
 git clone git://adnab.me/shard.git
 ```
 
-### CLI version only
 
-Enter the shard directory and create a data folder.
+### Web UI version
+
+Enter the shard directory and create a data2 folder.
 
 ```
 cd shard
-mkdir data		# for CLI use only
+mkdir data2		# for web UI
 ```
 
-Enter the shard subdirectory and install Elixir dependencies.
+Enter the shardweb subdirectory and install Elixir dependencies.
 
 ```
-cd shard
+cd shardweb
 mix deps.get
 mix compile
 ```
 
+Build the web assets:
+
+```
+cd assets
+npm install
+cd ..
+```
+
 **The default configuration assumes you have cloned the shard repo in your
-`$HOME`. If this is not the case, ajust `config/config.exs` accordingly.**
+`$HOME`. If this is not the case, ajust the data path parameter in
+`shardweb/config/config.exs` accordingly.**
 
 
-### Web UI version
+### CLI version only
 
-Enter the shard directory and create a data2 folder.
+Enter the shard directory and create a data folder.
 
 ```
 cd shard
-mkdir data2		# for web UI
+mkdir data		# for CLI use only
 ```
 
-Enter the shardweb subdirectory and install Elixir dependencies.
+Enter the shard subdirectory and install Elixir dependencies.
 
 ```
-cd shardweb
+cd shard
 mix deps.get
 mix compile
 ```
 
-Build the web assets:
+**The default configuration assumes you have cloned the shard repo in your
+`$HOME`. If this is not the case, ajust the data path parameter in
+`shard/config/config.exs` accordingly.**
+
+
+Web Ui usage
+------------
+
+Once you have completed the installation steps above, Shard web can be started
+by running the following in the `shardweb` directory of the repo:
 
 ```
-cd assets
-npm install
-cd ..
+iex -S mix phx.server
 ```
 
-**The default configuration assumes you have cloned the shard repo in your
-`$HOME`. If this is not the case, ajust `config/config.exs` accordingly.**
+The UI will become accessible at address `http://127.0.0.1:4000`.
 
 
 CLI usage
 ---------
 
 Once you have completed the installation steps above, Shard can be started by
-running the following in the `shard/shard` directory:
+running the following in the `shard` directory of the repo:
 
 ```
 iex -S mix
@@ -177,18 +214,7 @@ This CLI supports a few basic commands:
 - `/nick my_nickname`: set nickname
 - `/list`: list channels we are currently connected to
 - `/hist`: show recent messages on current channel
-- `/pm nickname`: enter private conversation with someone
+- `/pm nickname1 [nickname2] [...]`: enter private conversation with someone
+- `/send_file path`: make file available on the network and send link to current chat room.
+  **WARNING: all files are publicly available for now, even if they are sent in a private chat room.**
 - `/quit`: return to iex prompt
-
-
-Web Ui usage
-------------
-
-Once you have completed the installation steps above, Shard web can be started
-by running the following in the `shard/shardweb` directory:
-
-```
-iex -S mix phx.server
-```
-
-The UI will become accessible at address `http://127.0.0.1:4000`.
diff --git a/TODO b/TODO
index dd13e1d..baf34f7 100644
--- a/TODO
+++ b/TODO
@@ -10,13 +10,16 @@ achieves our goals.
 TASK LIST
 =========
 
-- High priority: invite
-- Medium priority: dht, dep
+- High priority: invite, dht, groups
+- Medium priority: net
 
 
-Invitation system (invite, EASY)
+Invitation system (invite, MED)
 --------------------------------
 
+Current issue: if a user PM's another user, that user does not know it!
+He has to pull the PM shard voluntarily to get the messages.
+
 A user may generate invitation tokens: an invite token is a signature of the
 invited user's pk by the inviter user's sk.  The user that redeems the token
 writes some data that is saved temporarily in the user's identity shard, which
@@ -39,7 +42,6 @@ Ex: keep 100 total open connections that are sampled by proximity on the set of
 plus 2 or 5 full random for all shards.
 
 
-
 DHT to find peers for a given shard (dht, EASY)
 -----------------------------------
 
@@ -99,16 +101,6 @@ access control decisions (obviously these can only run when an
 identity with admin privilege is running).
 
 
-Shard lifetime and dependency management (dep, MED)
-----------------------------------------
-
-Some Shards may pull other shards in, under certain conditions. For example
-a stored folder shard will just be a list of other shards that we all pull in.
-
-We want a way to have toplevel shards, shards that are dependencies of
-toplevel shards, and shards that we keep for a number of days as a cache but
-will expire automatically.
-
 
 COMPLETED TASKS
 ===============
@@ -187,3 +179,14 @@ Best option: those that we are connected to + some random to
 reach a quota (for example 10 or so)
 
 Implementation: netgroups (lib/net/groups.ex)
+
+
+Shard lifetime and dependency management (dep, MED)
+----------------------------------------
+
+Some Shards may pull other shards in, under certain conditions. For example
+a stored folder shard will just be a list of other shards that we all pull in.
+
+We want a way to have toplevel shards, shards that are dependencies of
+toplevel shards, and shards that we keep for a number of days as a cache but
+will expire automatically.
diff --git a/shard/lib/app/chat.ex b/shard/lib/app/chat.ex
index 2795153..ff0c97d 100644
--- a/shard/lib/app/chat.ex
+++ b/shard/lib/app/chat.ex
@@ -131,6 +131,11 @@ defmodule SApp.Chat do
     {:reply, state.manifest, state}
   end
 
+  def handle_call(:delete_shard, _from, state) do
+    GenServer.call(state.store, :delete_store)
+    {:stop, :normal, :ok, state}
+  end
+
   def handle_call({:read_history, top_bound, num}, _from, state) do
     ret = MST.last(state.mst, top_bound, num)
     {:reply, ret, state}
diff --git a/shard/lib/app/directory.ex b/shard/lib/app/directory.ex
index e9482f7..cbea8c3 100644
--- a/shard/lib/app/directory.ex
+++ b/shard/lib/app/directory.ex
@@ -64,6 +64,10 @@ defmodule SApp.Directory do
     {:reply, state.manifest, state}
   end
 
+  def handle_call(:delete_shard, _from, state) do
+    {:stop, :normal, :ok, state}
+  end
+
   def handle_call(:get_items, _from, state) do
     {:reply, SData.SignRev.get(state.items), state}
   end
diff --git a/shard/lib/app/file.ex b/shard/lib/app/file.ex
index ce28beb..e2a9798 100644
--- a/shard/lib/app/file.ex
+++ b/shard/lib/app/file.ex
@@ -84,6 +84,12 @@ defmodule SApp.File do
     {:reply, state.manifest, state}
   end
 
+  def handle_call(:delete_shard, _from, state) do
+    GenServer.call(state.store, :delete_store)
+    File.rm(state.path)
+    {:stop, :normal, :ok, state}
+  end
+
   def handle_call(:get_info, _from, state) do
     reply = cond do
       state.info != nil and GenServer.call(state.store, {:have_rec, state.info.merkle_root}) ->
diff --git a/shard/lib/app/identity.ex b/shard/lib/app/identity.ex
index 59a4b90..78abbe7 100644
--- a/shard/lib/app/identity.ex
+++ b/shard/lib/app/identity.ex
@@ -65,6 +65,10 @@ defmodule SApp.Identity do
     {:reply, state.manifest, state}
   end
 
+  def handle_call(:delete_shard, _from, state) do
+    {:stop, :normal, :ok, state}
+  end
+
   def handle_call(:get_info, _from, state) do
     {:reply, SData.SignRev.get(state.state), state}
   end
diff --git a/shard/lib/app/pagestore.ex b/shard/lib/app/pagestore.ex
index 5165b84..3cda51d 100644
--- a/shard/lib/app/pagestore.ex
+++ b/shard/lib/app/pagestore.ex
@@ -25,7 +25,7 @@ defmodule SApp.PageStore do
   @max_failures 4         # Maximum of peers that reply not_found before we abandon
 
   defmodule State do
-    defstruct [:shard_id, :path, :netgroup, :store, :reqs, :retries]
+    defstruct [:shard_id, :path, :netgroup, :store, :reqs, :retries, :store_path]
   end
 
 
@@ -36,15 +36,21 @@ defmodule SApp.PageStore do
   def init([shard_id, path, netgroup]) do
     Shard.Manager.dispatch_to(shard_id, path, self())
 
-    store_path = [Application.get_env(:shard, :data_path), "#{shard_id|>Base.encode16}.#{path}"] |> Path.join |> String.to_atom
-    {:ok, store} = :dets.open_file store_path, [type: :set]
+    store_path = [Application.get_env(:shard, :data_path), "#{shard_id|>Base.encode16}.#{path}"] |> Path.join
+    {:ok, store} = :dets.open_file(String.to_atom(store_path), [type: :set])
 
     Process.send_after(self(), :clean_cache, 1000)
 
-    {:ok, %State{shard_id: shard_id, path: path, netgroup: netgroup, store: store, reqs: %{}, retries: %{}}}
+    {:ok, %State{shard_id: shard_id, path: path, netgroup: netgroup, store: store, reqs: %{}, retries: %{}, store_path: store_path}}
   end
 
 
+  def handle_call(:delete_store, _from, state) do
+    :dets.close state.store
+    File.rm state.store_path
+    {:stop, :normal, :ok, state}
+  end
+
   def handle_call({:get, key, prefer_ask}, from, state) do
     case :dets.lookup state.store, key do
       [{_, _, bin}] ->
@@ -84,9 +90,11 @@ defmodule SApp.PageStore do
     case :dets.lookup state.store, hash do
       [] ->
         :dets.insert state.store, {hash, {:cached, System.os_time(:seconds) + @cache_ttl}, bin}
+        :dets.sync state.store
         nil
       [{_, why}] ->
         :dets.insert state.store, {hash, why, bin}
+        :dets.sync state.store
         why
       [{_, _, _}] ->
         nil
diff --git a/shard/lib/cli/cli.ex b/shard/lib/cli/cli.ex
index 380282d..54b882f 100644
--- a/shard/lib/cli/cli.ex
+++ b/shard/lib/cli/cli.ex
@@ -94,8 +94,16 @@ defmodule SCLI do
   end
 
   defp handle_command(state, ["list"]) do
-    IO.puts "List of known channels:"
+    IO.puts "Private conversations:"
+    for {_chid, %SApp.Chat.PrivChat.Manifest{pk_list: pk_list}, _} <- Shard.Manager.list_shards do
+      pk_list
+      |> Enum.filter(&(&1 != state.pk))
+      |> Enum.map(fn pk -> "#{SApp.Identity.get_nick pk} #{Shard.Keys.pk_display pk}" end)
+      |> Enum.join(", ")
+      |> IO.puts
+    end
 
+    IO.puts "Public channels we are connected to:"
     for {_chid, %SApp.Chat.Manifest{channel: chan}, _} <- Shard.Manager.list_shards do
       IO.puts "##{chan}"
     end
diff --git a/shard/lib/manager.ex b/shard/lib/manager.ex
index c178e2f..c3897a3 100644
--- a/shard/lib/manager.ex
+++ b/shard/lib/manager.ex
@@ -82,6 +82,27 @@ defmodule Shard.Manager do
     {:reply, pid, state}
   end
 
+  def handle_call({:delete, shard_id}, _from, state) do
+    case :dets.lookup(@shard_db, shard_id) do
+      [] ->
+        {:reply, {:error, :not_found}, state}
+      [{^shard_id, manifest, {:cached, _}, _}] ->
+        pid = case :ets.lookup(:shard_procs, {shard_id, nil}) do
+          [] ->
+            {:ok, pid} = apply(Shard.Manifest.module(manifest), :start_link, [manifest])
+            pid
+          [{{^shard_id, nil}, pid}] ->
+            :ets.delete(:shard_procs, {shard_id, nil})
+            pid
+        end
+        GenServer.call(pid, :delete_shard)
+        :dets.delete(@shard_db, shard_id)
+        {:reply, :ok, state}
+      [{^shard_id, _, _, _}] ->
+        {:reply, {:error, :pinned}, state}
+    end
+  end
+
   def handle_cast({:dispatch_to, shard_id, path, pid}, state) do
     :ets.insert(:shard_procs, { {shard_id, path}, pid })
     state = Map.put(state, pid, {shard_id, path})
@@ -105,6 +126,7 @@ defmodule Shard.Manager do
     case :dets.lookup(@shard_db, shard_id) do
       [{^shard_id, manifest, why_have_it, _old_state}] -> 
         :dets.insert(@shard_db, {shard_id, manifest, why_have_it, shst})
+        :dets.sync(@shard_db)
     end
     {:noreply, state}
   end
@@ -169,7 +191,7 @@ defmodule Shard.Manager do
     for [id] <- shards do
       case :ets.lookup(:shard_procs, {id, nil}) do
         [{{^id, nil}, pid}] ->
-          GenServer.cast(pid, :delete_shard)
+          GenServer.call(pid, :delete_shard)
         _ -> nil
       end
       :dets.delete(@shard_db, id)
@@ -366,6 +388,13 @@ defmodule Shard.Manager do
   end
 
   @doc"""
+  Delete a shard
+  """
+  def delete(shard_id) do
+    GenServer.call(__MODULE__, {:delete, shard_id})
+  end
+
+  @doc"""
   Return the list of all shards. Returns a list of tuples:
 
       {id, manifest, why_have_it}
diff --git a/shardweb/lib/controllers/directory_controller.ex b/shardweb/lib/controllers/directory_controller.ex
index d1b7136..ecb9c97 100644
--- a/shardweb/lib/controllers/directory_controller.ex
+++ b/shardweb/lib/controllers/directory_controller.ex
@@ -9,7 +9,7 @@ defmodule ShardWeb.DirectoryController do
     view(conn, false, args)
   end
 
-  def view(conn, public, %{"owner" => owner, "name" => name}=args) do
+  def view(conn, public, %{"owner" => owner, "name" => name}) do
     owner = Base.decode16! owner
     manifest = %SApp.Directory.Manifest{public: public, owner: owner, name: name}
     shard = SData.term_hash manifest
@@ -28,7 +28,6 @@ defmodule ShardWeb.DirectoryController do
     dir_public = (dir_public == "true")
 
     manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
-    shard = SData.term_hash manifest
     pid = Shard.Manager.find_or_start manifest
 
     item_manifest = ShardURI.to_manifest(add_uri)
@@ -41,7 +40,6 @@ defmodule ShardWeb.DirectoryController do
     dir_public = (dir_public == "true")
 
     manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
-    shard = SData.term_hash manifest
     pid = Shard.Manager.find_or_start manifest
 
     SApp.Directory.rm_item(pid, rm_name)
@@ -54,7 +52,6 @@ defmodule ShardWeb.DirectoryController do
     item_stored = (item_stored == "true")
 
     manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
-    shard = SData.term_hash manifest
     pid = Shard.Manager.find_or_start manifest
 
     SApp.Directory.set_stored(pid, item_name, item_stored)
@@ -68,7 +65,6 @@ defmodule ShardWeb.DirectoryController do
     dir_public = (dir_public == "true")
 
     manifest = %SApp.Directory.Manifest{public: dir_public, owner: conn.assigns.pk, name: dir_name}
-    shard = SData.term_hash manifest
     pid = Shard.Manager.find_or_start manifest
 
     {:ok, file_manifest, _file_pid} = SApp.File.create(path, mime_type)
diff --git a/shardweb/lib/controllers/page_controller.ex b/shardweb/lib/controllers/page_controller.ex
index 5c90416..baf72b9 100644
--- a/shardweb/lib/controllers/page_controller.ex
+++ b/shardweb/lib/controllers/page_controller.ex
@@ -9,6 +9,14 @@ defmodule ShardWeb.PageController do
     render conn, "shard_list.html"
   end
 
+  def shard_action(conn, %{"shard_id" => shard_id, "action" => action}) do
+    shard_id = Base.decode16! shard_id
+    case action do
+      "delete" -> Shard.Manager.delete(shard_id)
+    end
+    redirect conn, to: page_path(conn, :shard_list)
+  end
+
   def add_peer(conn, _params) do
     try do
       ip = conn.params["ip"]
diff --git a/shardweb/lib/router.ex b/shardweb/lib/router.ex
index a6a61f9..0b1f686 100644
--- a/shardweb/lib/router.ex
+++ b/shardweb/lib/router.ex
@@ -19,6 +19,7 @@ defmodule ShardWeb.Router do
     pipe_through :browser # Use the default browser stack
 
     get "/", PageController, :shard_list
+    post "/", PageController, :shard_action
     get "/peers", PageController, :peer_list
     post "/peer/add", PageController, :add_peer
 
diff --git a/shardweb/lib/templates/page/shard_list.html.eex b/shardweb/lib/templates/page/shard_list.html.eex
index 3fffef7..79fd794 100644
--- a/shardweb/lib/templates/page/shard_list.html.eex
+++ b/shardweb/lib/templates/page/shard_list.html.eex
@@ -23,9 +23,10 @@
   <tr>
     <th></th>
     <th>Shard</th>
+    <th></th>
     <th>URI</th>
   </tr>
-  <%= for {_id, manifest, why_have_it} <- shard_list() do %>
+  <%= for {id, manifest, why_have_it} <- shard_list() do %>
     <tr>
       <td>
         <%= case why_have_it do %>
@@ -37,6 +38,18 @@
       <td>
         <%= render "shard_entry.html", conn: @conn, manifest: manifest, pk: @pk %>
       </td>
+      <td>
+        <%= case why_have_it do %>
+          <% {:cached, _} -> %>
+            <%= form_for @conn, page_path(@conn, :shard_action), [class: "form-inline", style: "display: inline", onsubmit: "return confirm('Delete this shard and all its data?');"], fn f -> %>
+              <%= hidden_input f, :shard_id, value: id |> Base.encode16 %>
+              <%= hidden_input f, :action, value: "delete" %>
+              <%= submit "Delete", [class: "btn btn-xs btn-danger"] %>
+            <% end %>
+          <% _ -> %>&nbsp;
+        <% end %>
+
+      </td>
       <td><small class="shard_uri"><%= manifest |> ShardURI.from_manifest %></small></td>
     </tr>
   <% end %>