improve route selection for uTP sockets #4850
Comments
What's "the fib"?
That's how it works, yes. Unless you configure specific interfaces to use.
What do you mean by "packets being ready"? uTP runs over a single UDP socket, which means you can stick that in epoll/kqueue etc.
That's intentional, for load balancing. Unless you select a specific (set of) interfaces to use.
The log may have some details about why it didn't use the tunnel IPs. It's not supposed to use down interfaces, that's a problem. |
|
Hi @arvidn,
Forwarding Information Base. It's basically the condensed and/or optimized form of the routingtable that used all sources (if there are routing protocols in the game) and did some basic optimizations. It's basically what the kernel works with; think of it as a compiled and "runtime" version of iptables/netfilter rules compared to the rules you type in.
so is there a callback whenever a packet is ready/buffered or do you have to actively select/poll or just busy-wait in a thread asking the socket if it has data? anywho... let's start with tcp as this should be easier to debug, then tackle the connectionless-stuff.
uff.. i think you can leave it out or better you should leave that out.
can you give me a bump in the right direction how i could do this in a reproducible way? i'm running deluge with "-L info", will bump this up to "debug" and see what it spits out, but maybe an example with with what should i test, current master? the data in the tcpdump you saw were from just forced-reannounce to a tracker, where it used the ip from a down interface. if you could hint me at how you normally gdb (or if that's somewhere in the docs) libtorrent, i could try to dig into this. my gdb is a little rusty. |
|
k.. some small infos, after battling with deluge vs. python3.7/logging on a python3.8 machine.... I still think it should'nt use all ips on the host, especially not localhost. all 10.8 and the routeable 301 address would have internet connectivity; route would point at tun0/10.8.0.4 i think it is putting all on the correct interface but never with the correct src-address.. do you filter out rfc ips from route selection? |
the former. the socket is waited on in epoll, when it's readable packets are drained until the socket doesn't have any more readable packets. then it's put back in epoll. This is just how UDP sockets are handled in general, nothing special.
I'll be more specific. The details are documented here: http://libtorrent.org/manual-ref.html#multi-homed-hosts If you set Tracker announces are different. Then we attempt to announce from every active listen interface. i.e. every IP that we accept incoming connection on, we attempt to announce to the tracker. In the normal case there's only one interface that can actually route to the tracker anyway, so that's the one that gets announced.
round-robin on one interface is the same as no round-robin.
I would think the most common case would be to not configure
I wouldn't expect this to happen with TCP connections. with
I would expect, if I bind an outgoing TCP socket to local address A, and try to
I'm not sure I understand what problem this is referring to. I understand there's an issue with UDP packets being sent from a socket bound to one interface but sent to an address that really should have used a different interface. That's the issue this ticket is referring to. Are you saying there's another issue? Does it affect TCP? |
|
does this have anything to do with it? or is this the embedded tracker or some sort of logic which "myip" should be inserted into the announce? |
It seems this issue affects 1.2.x, so
Ok. down interfaces should be added as something to fix. Was the interface up when the client started, and then brought down?
I'm not sure what you mean by this. When you build printing the verbose log from |
yup.. tun0 would have 10.8.something but the tracker announcement is sent from two wrong ips, even one from a down interface but this time respecting the kernels decision over which interface to route (which is different in the udp case). this is deluge with no "incoming address" or "outgoing interface" set. |
|
also, trackers and peer connections are different. for trackers libtorrent attempts to announce all IPs that can reach the tracker. There is no filtering for special address ranges. But a network, say, |
i think you should narrow this down to "we only announce it to the interface the routing to this tracker points at".
hmmk... that could explain why it's not using the "correct" address when bound to tun interfaces with rfc-ips: again, i tested with nothing specified in "listen address" or "interface". would that mean that if i bind to "tun0" which has a rfc ip, no annoncements would be made? what about nat then? |
Ok.. so discard all that, that's just the magic used to figure out the IP that should be in the L7 Packet to the tracker? |
The whole point of doing this is to support multi-homed machines, i.e. supporting IPv4/IPv6 dual stacks.
I don't really know the details of how Deluge and qBT map their settings to libtorrent's
If there is no route out of that local network, then only trackers inside that local network are considered reachable. If there's a gateway, then all public IPs are considered reachable.
Same thing.
Not really, it's used to determine whether a listen interface (local IP) should be used as the local address in a tracker announce. Not the tracker announce, there's one announce per IP that should be advertised. |
|
what do you mean by "RFC IPs". Do you mean reserved IP ranges? |
|
rfc
the rfc1918 ips, 10/8, 172.16/12, 192.168/16 |
|
but i might have already figured out my problem: docs say
ok... let's imagine we've a simple host set up with one interface, eth0 everything should be fine; it is a rfc ip BUT it has a defaultroute so it is fine to announce trough it. if we do which basically is the same as one /0 route, no announcements at all would be made? I really think we should consider asking the kernel about routing and only if this does return "no route to host" take that interface out of the list of connections to this tracker to be made. otherwise just use the interface and make the announcement over that host, let the kernel do all the routing and ip selection.. We don't have any control about which ip shows up on the tracker anyhow. the other quirks i'm experiencing with strange ips showing up as source-ips on the wrong interface might be a side-effect of not having a default-route... don't know if we want to fix that or fix it the right way by asking the routing-table? |
|
I was wondering if this could have something to do with the "information leak" i mentioned above and described in #4803. openvpn does this "two /1" routes thing to be "more specific" and attract all the traffic. But libtorrent wouldnt see this interface as it has no defaultroute set and the defaultroute is still the same as in the "unprotected" state? I'll try to make it a bit more clear what i mean libtorrents now sees eth0 and tun0. tracker to contact has 6.7.8.9. the packet is crafted with the src address of eth0 (or 0.0.0.0) and handed over to the kernel. is this correct so far? |
No, I don't think so. Any route that has a gateway, is assumed to be able to reach the internet. Any such routes will have an interface too, right? that they are sending their packets via. I would expect, in your scenario, that both |
finally managed to get it to compile (just my own stupidity or unfamiliraty with b2) tried to download ubuntu server with example_client: seems to be caused by "address/port already in use", could be more "speaking". might be caused by the "unable to bind". tried with yields seem to be really trying to bind to port "0". how do i correctly specify the port? Thought i saw somehere that ":33445" would do the trick. 0.0.0.0:33445 seems to work. seems to start announcing to the tracker(s): seems to try: note the down interface 1.2.3.4 there would be a route: could you try this on your machine? adding two /1 routes and then deleting the defaultroute? if i had to guess, the "no route to host" comes from this: |
|
regarding route selection and the unbound udp sockets... I think other udp services bind to * and just drop packets to dst-ips (local ips that is in this case for incoming packets) that are not configured to be listening (like ntp does). the can_route functions should then be doing something like this (should be a "global" function, not a function in an interface c++ instance) i found this library, which supposedly gives you access to the kernel routing table in a portable way there's a c non-library-snippet with rt_netlink just for linux: i've basically no idea how to do that on windows. |
|
this also gives access to the interfaces. and this gives access to the routing table. You seem to have figured out that you can't omit the IP or interface name in that doesn't look right. it looks like libtorrent is failing to list the routing table. Do you see anything in |
oh cool. you already have that rt_netlink route enumeration? enum_interfaces seems to work, breakpointed at the OT: is there a neater way to access a c++ vector in gdb?
it seems nl_dump_request returns no routes: does this help? |
|
strace says: OT: is this intentional that it does an accept() on all interfaces right after listen()? after the listen() setup, there's a lot of getsockname and the setsockopt for TOS, then it's doing another round of "getting routes" |
|
tiny babystep: it fails in
in |
|
patched around in the code a bit and ignored zero-length route informations (dunno where they originate). My error seems to arise from the different ways you can use to add routes. on the first case, the elements of the I'll keep looking at that. |
These do different things though, right? The first does not have a gateway, so it will be assumed to only be able to reach IPs on that local network. The second one, having a gateway is expected to be able to reach the internet via it.
As far as I can tell, in the first case, the only way to be able to reach the internet is if the route is a point-to-point connection, which isn't very well supported in libtorrent. I'm open to suggestion on how to discover that. |
|
@toasta I really appreciate your help in improving this. It sounds so far I should:
|
|
if |
|
actually, looking at your strace again, it really looks like the problem is that libtorrent uses a fixed size buffer to read the netlink response into, and your routing table is larger than that buffer (of 8kiB). I think the most important fix would be to support larger routing tables, so I'll make that buffer adapt. |
I have no incoming connections, only outgoing. |
Right. Looking at the routing table wouldn't really help. The question isn't "If I contact this IP, will my connection go over this interface?". The question is: "Is it possible to accept incoming connections from the internet (or the tracker's network) on this interface". The |
it proofs there is internet behind that interface and is less likely to leak any information. Just trying to help and think how i would do a "is there internet" check. |
Correct. https://libtorrent.org/reference-Settings.html#outgoing_interfaces |
What does that mean exactly? Does it mean the default listen interfaces (which are I think the bulk of this issue (or at least the first part of it that I'm trying to address) is concerning how the default settings (i.e. when the user doesn't specify listen IP(s) or interface(s)). When I say "expand" I mean turn The patch set fixes the following issues:
I would like to know your opinion about these changes.
Unfortunately I don't really know exactly what that means in Deluge. |
|
actually, I have one more question, apart from the issue of peer connections over uTP not honoring the routing table, can the other issues you experience be solved by the "escape hatch" of explicitly specifying the listen interface you want to use? I think it's important that there is a manual override that works. |
hmmk.. i read that but imho it's not clear, that tracker-announcements are completely separate.
there's no mentioning of "tracker" in the description of outgoing_interface, but listen_interfaces talks about outgoing (udp) trackers and that if you dont specify a port, outgoing tcp will still be possible. But you might consider changing that or adding another option so people having the "vpn dies and reveals my real ip" could simply put "tun0" in the outgoing_interface and never fear that any ip other than that of tun0 would be published. |
Yeah, I agree that it's not all that clear. The mutli-homed section of the documentation should also describe some of the rationale we've talked about here, and how trackers, incoming peer connections and outgoing peer connections are treated differently (and really, only the outgoing peer connections take the routing table into account, or are meant to do that at least).
If you put I've been considering retiring the |
tried that, i would expect it to not use any other interface for tracker-announcements, too, but this is what it does with both set to (note that But you mentioned tracker announcements are totally different and dont adhere to any restrictions one would expect from setting interfaces? I think - at least for http-announcements, they should follow the users wish to only use this one interface. supplying all 5 tunnels as outgoing and/or listen interfaces didnt make it better. Putting dht aside...
There might be a corner ultra special case where you want to use all available interfaces but always announce an ip from lo, but that's too special to think about right now and even if, announcing over one interface only should be enough. and it's a little weird... since no tracker-announcements gets trough, i have no clients to connect to, altough those would use the correct interface? |
What's the log output from libtorrent? I don't know what the log you posted means. libtorrent will print which interfaces and IPs it creates listen sockets for. I think that log would help understand what's going on. You set
No, I'm pretty sure I never said anything to suggest that. It's quite the opposite. tracker announces only care about the
What did it do? I'm getting the feeling that you may be getting error messages back from libtorrent, but you aren't seeing them because, maybe you're filtering alerts or not printing all of them.
neither do I
Well, it still need to enumerate the IPs assigned to the specified interface, since there may be more than one. Say, one IPv4 and one IPv6 address.
I would like to know more about what you mean here. There is no "destination" (at least not in deciding which IPs to "listen on", and announce to). The tracker itself is a destination, is that what you mean? do you mean that a tracker should always only be announced to once over whatever interface your routing table says it should use? (my understanding is that the routing table will always only have one best path to a destination IP). That wouldn't work. Consider having one IPv4 and one IPv6 address, should only one be exposed to the tracker? |
The tcpdump should just show that it's using almost all source ips the box has, but it always uses the same outgoing interface which will only work if one uses the correct source ip.
k. then i got
wrong.
yep, but it does this for all interfaces, not just the one that can actually reach the tracker (that's why i was always recommending the routing table)
yup.
nope. that's different address families. It's in two stacks and therefore like two different interfaces or two different running instances. I get that you need that, but you'll only do that if the address resolved has an AAAA record anyhow. Let's agree that if we say "interface" we both mean the tuple {interface,adressfamily}, so {eth0,v4} and {eth0,v6} are two interfaces. And all ips of that single interface that is used to reach the destination (tracker) should be announced (putting mixing rfc and routable ips aside, that's a bonus) but only the ips of this interface the route points at.
that's a tough one. I think this has to be left to the user. You already hinted at "there's only one best route". This is usually true, putting ecmp and med aside.
What libtorrent - according to strace - does, is looping over all source addresses the box has with bind() - setting the src ip -, then with connect() to the address. The problem is that setting src doesn't make the connection stick to or select the right interface (i think you also describe that in the docs I think the problem is that torrents get stuck in "Announcements sent" state with this configuration. Skimming over the code i see
assuming that force_reannounce from torrent_handle.cpp is the function the application (deluge, qBitorrent) calls which queues the async call, maybe it's only qbittorent that does not set this flaq on "force reannounce" but it can't get to reannounce once it's in this state. Waiting very very long (hours) does sometimes make this state go away, Pausing and unpausing, not really. This might also explain, why shutting down deluge/qBittorrent takes hours because it's waiting for all the "stopped" announcements to come back. I don't know if this maybe is Line 8994 in f4defa5 combined with Line 9012 in f4defa5 and maybe the counter What i also noticed in strace is that it's issuing the first n (5?) bind() calls followed by the connect() calls in sequence, then processes other stuff and then finally for the last (remaining?) src-ips (bind() and connect()) Hope that helps... |
|
Sorry for butting in, and just quickly wanted to add that I get same behaviour as before when testing a fresh build of has_default_routes libtorrent branch(#4881) i.e. still both VPN and own IP are reported when checking announce-source-IPs from ipmagnet website(http-tracker) on windows 10 with two different provider's openvpn configs in deluge 2.0.3 and not binding any interfaces/ips, so default setup(config). Unrelated, but I had never thought about if opening torrent client without any torrents in, and then starting VPN afterwards, then only the own IP is announced when adding ipmagnet link afterwards and not both, but that of-course also makes sence as having set listen/out sockets already. Qbittorrent's log states it sees the new IP and whatnot, but doesn't let libtorrent know seemingly to re-initialize. I know such behaviour is very exotic and doesn't make sence anyway lol, but just never thought about it, and saw it by chance when testing this and forgetting start VPN initially. This is both v1.2.7, v1.2.6 and has_default_routes i'm talking about here, but is torrent-client issue I understand. Sorry again for butting in, and just added this comment because old ticket referrenced this ticket to be used instead. Edit: Forgot to say that was thinking if the reason for the own IP isn't announced on Linux and only on windows(when not binding anything), when same profile used and same clients used both places from openvpn.net, maybe was because using tun on Linux and tap on windows, but changing to tap on Linux I couldn't get internet access, even though status from openvpn was OK, so couldn't test that scenario(I checked had nameservers in-place, but got no route to host on curl's/ping's etc.), and didn't know how use tun on windows, for additional testing neither. |
|
@toasta there are a few steps involved, and I realise some of the things I've said can easily be misinterpreted. I would like to clarify. Step 1, to establish which IPs to listen onThis is determined by the
I think option 3 here may be problematic in some cases. The set of patches in the pipeline to land attempts to improve this. For example by ignoring devices that are "down", and by not requiring a default route to an interface in order to consider it "connected to the internet" (to support two routes of Step 2, send announces to trackersIn this step, only IPs selected from step 1 are considered. This is where the If a
Step 3, making peer connections(let's assume For outgoing uTP peer connections, one of the UDP sockets associated with a
I don't think that's typically how the word "interface", "device" or "adapter" is used. I use those words to refer to a (physical or virtual) NIC. Each of which can be associated with any number of networks (i.e. address and prefix-length)
I'm not actually that familiar with how people would normally set up multiple networks over the same interface. But I've seen segmentations of local networks where some reach the internet, and some reach various internal services, that are fire-walled from the internet. Are you saying that it only ever makes sense to announce one network per interface? I can imagine having multiple networks for an interface that still reach different egress points to the internet, in which case both would make sense to be announced.
it is ultimately up to the user, because
What do you mean by this exactly? libtorrent will enumerate the interfaces and IPs somewhat regularly. Is that what you see in
It sounds like the heuristic of picking which interfaces to "listen on" is mismatching your routing table, and attempts to bind to device fail (which isn't too surprising, since you need root privileges for that on linux). Hence the mismatch of source IP and interface. Does that sound like it would explain the symptoms you see? I would like to improve this situation and I'm open to suggestions. However, you say that even if you set |
|
@mhertz thanks for the report. Announcing both the VPN and the direct connection is a somewhat separate issue, and I think it deserves its own ticket, feel free to create a ticket for this issue (I seem to recall there is one in qbt, that probably should be linked). I'm open to suggestions for how to solve that. Maybe the default should be to only expand unspecified addresses to IPs from the "main" interface. It would require some heuristic to figure out which the main one is. |
|
Sorry Arvid, I guess I misunderstood your closing post in my ticket about this, like as if you had one of the commits in #4881 made for fixing this and to direct also conversation of this issue here. I don't know enough about this to suggest if/what changes should possibly be made regarding that, and after reading your explenations previously I learned that it quite possibly was kinda unavoidable, given supporting multi-homed support, but anyway, I'll butt out again, and follow this interesting learningfull ticket from the outside :) Sorry again for the detraction. |
|
@toasta I believe the last issue I haven't addressed yet is the interface outgoing uTP peer connections are made over. I think the ideal solution would be to have a separate UDP socket, bound to the unspecified address to be used for peer connections. The tricky thing with that is that The simplest solution would probably be to take the routing table into account when picking the outgoing interface. |
|
Any update? |
|
It will be nice to have a good µTP (uTP) support for I2P: cc: @zzzi2p. |
|
I don't really see a reason to run uTP over i2p. |
libtorrent version (or branch): RC_1_2
This issue was posted here: qbittorrent/qBittorrent#13094
The logic that picks which UDP socket to use for a uTP connection is simple and does not pick the correct one in some circumstances. At least make it take into account:
The code is here:
For both UDP and TCP connections, it seems interfaces that are
DOWNare still considered.The text was updated successfully, but these errors were encountered: