Merge pull request #38 from ikatson/refactoring-2023-12

Refactoring DHT

TODO.md

@@ -1,4 +1,4 @@
-- [ ] when we have the whole torrent, there's no point talking to peers that also have the whole torrent and keep reconnecting to them.
+- [x] when we have the whole torrent, there's no point talking to peers that also have the whole torrent and keep reconnecting to them.
 - [ ] per-file stats
 - [x (partial)] per-peer stats
 - [x] use some concurrent hashmap e.g. flurry or dashmap
@@ -8,29 +8,44 @@
 - [x] initializing/checking
   - [x] blocks the whole process. Need to break it up. On slower devices (rpi) just hangs for a good while
   - [x] checking torrents should be visible right away
-- [ ] server persistence
-  - [ ] it would be nice to restart the server and keep the state
+- [x] server persistence
+  - [x] it would be nice to restart the server and keep the state
 - [x] torrent actions
   - [x] pause/unpause
   - [x] remove including from disk
 - [ ] DHT
-  - [ ] for torrents with a few seeds might be cool to re-query DHT once in a while.
+  - [x] bootstrapping is lame
+  - [x] many nodes in "Unknown" status, do smth about it
+  - [x] for torrents with a few seeds might be cool to re-query DHT once in a while.
+  - [x] don't leak memory when deleting torrents (i.e. remove torrent information (seen peers etc) once the torrent is deleted)
+  - [x] Routing table - is it balanced properly?
+  - [ ]
+  - [x] Don't query Bad nodes
+  - [-] Buckets that have not been changed in 15 minutes should be "refreshed." (per RFC)
+    - [x] Did it, but it's flawed: starts repeating the same queries again as neighboring refreshes
+          don't know about the other ones, and DHT returns the same nodes again and again.
   - [x] it's sending many requests now way too fast, locks up Mac OS UI annoyingly
+  - [x] store peers sent to us with "announce_peer"
+  - [ ] announced peers should be persisted
+  - [ ] After the search is exhausted, the client then inserts the peer contact information for itself onto the responding nodes with IDs closest to the infohash of the torrent.
+
+    To do this, a
+  - [x] Ensure that if we query the "returned" nodes, they are even closer to our request than the responding node id was.
 
 someday:
 - [x] cancellation from the client-side for the lib (i.e. stop the torrent manager)
 
-- [ ] favicons for Web UI
+- [x] favicons for Web UI
 
 refactor:
+- [ ] session persistence: should add torrents even if we haven't resolved it yet
 - [x] where are peers stored
 - [x] http api pause/unpause etc
 - [x] when a live torrent fails writing to disk, it should transition to error state
 - [x] something is wrong when unpausing - can't finish. Recalculate needed/have from chunk tracker.
 - [x] silence this: WARN torrent{id=0}:external_peer_adder: librqbit::spawn_utils: finished with error: no longer live
 
 - [x] start from error state should be possible from UI
-- [ ] if the torrent was completed, not need to re-check it
 - [x] checking is very slow on raspberry
   checked. nothing much can be done here. Even if raspberry's own libssl.so is used it's still super slow (sha1)
-- [ ] .rqbit-session.json file has 0 bytes when disk full. I guess fs::rename does this when disk is full? at least on linux
+- [ ] .rqbit-session.json file has 0 bytes when disk full. I guess fs::rename does this when disk is full? at least on linux. Couldn't repro on MacOS

crates/dht/Cargo.toml

@@ -27,13 +27,16 @@ bencode = {path = "../bencode", default-features=false, package="librqbit-bencod
 anyhow = "1"
 parking_lot = "0.12"
 tracing = "0.1"
+backoff = "0.4.0"
 futures = "0.3"
 rand = "0.8"
 indexmap = "2"
 directories = "5"
+dashmap = {version = "5.5.3", features = ["serde"]}
 
 clone_to_owned = {path="../clone_to_owned", package="librqbit-clone-to-owned", version = "2.2.1"}
 librqbit-core = {path="../librqbit_core", version = "3.1.0"}
+chrono = {version = "0.4.31", features = ["serde"]}
 
 [dev-dependencies]
-tracing-subscriber = "0.3"
+tracing-subscriber = "0.3"

crates/dht/examples/dht.rs

@@ -2,7 +2,7 @@ use std::time::Duration;
 
 use anyhow::Context;
 use librqbit_core::magnet::Magnet;
-use librqbit_dht::Dht;
+use librqbit_dht::DhtBuilder;
 use tokio_stream::StreamExt;
 use tracing::info;
 
@@ -16,7 +16,7 @@ async fn main() -> anyhow::Result<()> {
 
     tracing_subscriber::fmt::init();
 
-    let dht = Dht::new().await.context("error initializing DHT")?;
+    let dht = DhtBuilder::new().await.context("error initializing DHT")?;
     let mut stream = dht.get_peers(info_hash)?;
 
     let stats_printer = async {
@@ -36,6 +36,7 @@ async fn main() -> anyhow::Result<()> {
                 let mut f = std::fs::OpenOptions::new()
                     .create(true)
                     .write(true)
+                    .truncate(true)
                     .open(filename)
                     .unwrap();
                 serde_json::to_writer_pretty(&mut f, r).unwrap();