qBittorrent not using the interface ip the routing decides to use.

[toasta]

Please provide the following information

qBittorrent version and Operating System

Version: 1:4.2.5.99~202004250117-7015-2c65b79~ubuntu20.04.1
20.04

If on linux, libtorrent-rasterbar and Qt version

1.2.6+git20200425.a9968916ca-1ppa1~20.04
Package: libqt5core5a
Version: 5.12.8+dfsg-0ubuntu1
(hope that's the one you ment, but shouldnt apply to this issue)

What is the problem

This is only for outgoing connections.
qBittorrent is using the wrong src-ip for outgoing requests, not honoring
the interfaces address the kernel-routing decides to use.

What is the expected behavior

Outgoing packets should use the ip of the interface the kernel decides to route over,
not one of the other interfaces qbittorrent is listening on.

Steps to reproduce

hope that works for you, still trying to find a mwe.

• configure an interface eth0 with address 1.2.3.4/24
  -- no additional routes set on this interface
• configure an interface eth1 with some other adress (eg. 3.4.5.6/24)(one should be able to do that with a dummy interface, or vpn)
• configure an interface eth2 with some other adress (same as for eth1)
• route 0.0.0.0/1 via eth1
• route 128.0.0.0/1 via eth2
• in qbittorrent config
  -- set "Network interface" to "Any interface"
  -- set "Optional Ip Address to bind to (requires restart)" to "All addresses"
• start tcpdump on eth1 (tcpdump -ni eth1 "port 12500 and not src 2.3.4.5")
• manually add peers which will be routed over eth1
• see wrong src address popping up in tcpdump

Extra info(if any)

if i do the "Steps to reproduce", qbittorrent sends request over interfaces with the wrong src-ip set.
this is especially annoying when using ecmp (granted, this is a corner case) or the normal behaviour of say openvpn which sets two /1 routes to be more specific than the /0 route and you want to route 128/1 over another tunnel for load-"balancing"

if i do the above tcpdump (note that eth0 has the 1.2.3.4)
and dump on eth1 (which in my case has 10.8.8.27)
i would expect that i see no other src-ip than 10.8.8.27 when i originate packets.

to faster demonstrate that, i manually add peers to a torrent.
(seq 0 255 | while read a; do echo "2.3.4.$a:12500"; done -- and paste that into the "manually add peer" txtbox)

• verify that eth1 would indeed be used:

root@spe:~# ip r get 2.3.4.5
2.3.4.5 dev eth1 src 10.8.8.27 uid 0 

• start dump (note that i have another interface, just ignore that)

root@XX:~# tcpdump -ni tun0 "port 12500 and not src 10.8.8.27"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
12:56:15.536764 IP 94.X.X.X.35247 > 2.3.4.0.12500: UDP, length 104
12:56:15.536800 IP 1.2.3.4.35323 > 2.3.4.0.12500: UDP, length 104
12:56:15.536988 IP 94.X.X.X.35247 > 2.3.4.1.12500: UDP, length 104
12:56:15.537012 IP 1.2.3.4.35323 > 2.3.4.1.12500: UDP, length 104
12:56:15.537124 IP 94.X.X.X.35247 > 2.3.4.2.12500: UDP, length 104
12:56:15.537149 IP 1.2.3.4.35323 > 2.3.4.2.12500: UDP, length 104
12:56:15.537259 IP 94.X.X.X.35247 > 2.3.4.3.12500: UDP, length 104
12:56:15.537282 IP 1.2.3.4.35323 > 2.3.4.3.12500: UDP, length 104
12:56:15.537390 IP 94.X.X.X.35247 > 2.3.4.4.12500: UDP, length 104
12:56:15.537414 IP 1.2.3.4.35323 > 2.3.4.4.12500: UDP, length 104
12:56:15.537521 IP 94.X.X.X.35247 > 2.3.4.5.12500: UDP, length 104
12:56:15.537544 IP 1.2.3.4.35323 > 2.3.4.5.12500: UDP, length 104
12:56:15.537655 IP 94.X.X.X.35247 > 2.3.4.6.12500: UDP, length 104
12:56:15.537678 IP 1.2.3.4.35323 > 2.3.4.6.12500: UDP, length 104
12:56:15.537794 IP 94.X.X.X.35247 > 2.3.4.7.12500: UDP, length 104
12:56:15.537817 IP 1.2.3.4.35323 > 2.3.4.7.12500: UDP, length 104
12:56:15.537920 IP 94.X.X.X.35247 > 2.3.4.8.12500: UDP, length 104

so for whatever reason, qtbittorent is asking the kernel for routing (i wouldnt know how to do that differently), but uses the ip it bound to for another interface instead of leaving it empty and letting the kernel figure out what src-ip to use.

it doesnt seem to happen if you've only a /0 route.

but finally, i wouldnt expect any packets showing up on eth1 with a src-address other than the ip of eth1 if originating traffic from the local machine (yes, if this would be a router than this is expected, but not for traffic originating from the machine itself with src-ip set null in connect() .

i'm kinda stuck with debugging here as i was expecting to see the src-ip set in the connect() syscall but strace says:

[pid 389042] write(8, "(N) 2020-07-02T13:07:45 - Peer \"3.4.5.224:12500\" is added to torrent \"XXXXXXXX\"\n", 92 <unfinished ...>
[pid 389046] sendmsg(22, {msg_name={sa_family=AF_INET, sin_port=htons(12500), sin_addr=inet_addr("3.4.5.224")}, msg_namelen=16, msg_iov=[{iov_base="XXXXXXX", iov_len=104}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL <unfinished ...>

normal telnet on the machine to those ips works fine (routing and selection of src-ip-wise)
deluge does not have this problem. it seems to be using another libtorrent trough it's libtorrent-python-bindings tough.

deluge:

lsof -np 231675 | grep -i libtorrent
deluged 231675 [...] /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.so.9.0.0

qbittorrent:

lsof -np 389042 | grep -i libtorrent
qbittorre 389042  [...] /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.so.10.0.0

[FranciscoPombal]

@toasta

Both deluge and qBittorrent use the same libtorrent: https://github.com/arvidn/libtorrent (also known as libtorrent-rasterbar). The thing is, on your system, qBittorrent is using libtorrent >1.2.6, and deluge is using 1.1.x (which I think is what libtorrent-rasterbar.so.9.0.0 amounts to, if I'm not mistaken).

In libtorrent 1.2.x, there were important changes made to multihoming support. If you are using "any interface"/"any address", libtorrent will try establish connections from all endpoints. So it is not sufficient to set one route with a higher priority to guarantee that libtorrent will only use a certain endpoint - you have to actively force it to do so. This can be done 2 ways:

• changing the network settings from "any interface"/"any address" to the specific one you want (you can only select one or "any" in qBittorrent unfortunately; libtorrent actually allows you to specify multiple).
• using some kind of firewall/"killswitch"/etc (like some "VPN apps" have) to force applications to use a certain endpoint for connections, not just "suggesting" with a higher priority.

Related discussion: arvidn/libtorrent#4803 (arvidn/libtorrent#4803 (comment), in particular).

[paxter]

Good issue and this points a problem I often saw on trackers recently. I think for usability reasons qbittorrent/libtorrent (not sure where the problem is located) should default choose the correct interface for outgoing connections. A lot of people doesn't know something about the advanced settings and won't notice that their real ip address is announced to trackers.

[FranciscoPombal]

should default choose the correct interface for outgoing connections.

It is impossible to "guess" what the user wants.

A lot of people doesn't know something about the advanced settings and won't notice that their real ip address is announced to trackers.

As mentioned in arvidn/libtorrent#4803 (comment), the onus is on software such as "VPN apps" (or ultimately, on the users) to not assume "one computer == one IP", and to correctly configure their network facing system settings/software. As a file-transfer piece of software, I think libtorrent does the correct thing by using all interfaces by default, since this maximizes connectivity by default.

At most, qBittorrent could show a notice (like the one about copyright law it currently shows on first startup), urging users to double-check their configurations.

[paxter]

Thanks for clarifying this @FranciscoPombal!
A notice on first startup is a really good idea.

[toasta]

Hi @FranciscoPombal

* changing the network settings from "any interface"/"any address" to the specific one you want (you can only select one or "any" in qBittorrent unfortunately; libtorrent actually allows you to specify multiple).

Yes, this currently is my workaround binding to - say eth2 with it's ip 2.3.4.5.
But that defeats the purpose.

Say you use bittorrent to distribute a new software-release to hundrerds of servers.
Some of them are in dev, staging, beta and prod "vlans".
It would be kinda strange for this to work to have to run 4 different configured instances for this to work.

It even does not work if you've your "normal" home setup with eth0 beeing some public/routeable address and your ultra-default 192.168.0/24 on eth1.
I could understand that it's a problem for receiving traffic and announcing stuff and whatnot, but not for the simple "send a packet" call every other program does -- let the kernel decide the routing (which it already does) and at the same time let the kernel put the correct source-address in.

* using some kind of firewall/"killswitch"/etc (like some "VPN apps" have) to force applications to use a certain endpoint for connections, not just "suggesting" with a higher priority.

it's not higher priority (like in metric) and neither "more specific"s, it's just routing and the normale "most specific" lookup the kernel does. /0 goes to eth0 but 192.168.0/24 goes to eth1 'caus 24 > 0.
I dont see a reason why this should be difference for two /1 routes.

and to make things worse, let's talk ecmp 'caus it's really easy with the effort cumulus put in the kernel.

Related discussion: arvidn/libtorrent#4803 (arvidn/libtorrent#4803 (comment), in particular).

I'll try to understand that.

But aren't we at least on the same boat that forcing packets with the wrong src address on an interface is wrong?

[toasta]

Hi @FranciscoPombal

It is impossible to "guess" what the user wants.

I want normal routing.
clients on the internet should be contacted from my routable address, clients on the internal-network should be connected from my 192.168 rfc ip.

As mentioned in arvidn/libtorrent#4803 (comment), the onus is on software such as "VPN apps" (or ultimately, on the users) to not assume "one computer == one IP", and to correctly configure their network facing system settings/software. As a file-transfer piece of software, I think libtorrent does the correct thing by using all interfaces by default, since this maximizes connectivity by default.

that would be fine, but again... i dont think this'll work if it's putting - in my case - its iscsi-only-vlan ip as the src-address when sending to some tracker... which the first well configured router will drop anyway.

I'm totally fine that you really have to somehow tell deluge any bittorrent client what to put in the l7 packet when it announces to some tracker, but all lower layers should be handeled by the kernel, not the userspace (despite setting its sourceport or something)

[toasta]

Hi @FranciscoPombal

Related discussion: arvidn/libtorrent#4803 (arvidn/libtorrent#4803 (comment), in particular).

I just skimmed over that.
Don't they just talk about information leakage and the problem with vpn-clients and that to ensure you don't accidently announce your real public ip this is their "workaround" to simply bind only to the tunnel-interface and ignore anything else, including normal routing instead of recommending to go without a defaultroute and let the openvpns of this world use their normal 2 /1 approach?
so when the tunnel goes down, the interface get's infty metric and/or all the routes pointing at the interface are deleted 'caus the interface got deleted.
Hindering packets to go out by using the "wrong" src-address is... well... a strange interpretation of security.

the comment explicitely talks about metrics, but "more specific" routing should always win (and it still does), it's just that unconditially providing the (wrong) src address is... some .. kind ... of protection but maybe done at the wrong level / layer.

I know it's pretty unusual for a bittorrent client to run on a fulltable (bgp) router with thousands of vlans and in a "default free zone" but at least connecting somewhere should use the correct address. Which address to put in an announcement is - to be left to the application or "layer8".

and maybe another comment on

"It is impossible to "guess" what the user wants."

I think the user wants for it to "just work", even if he's not the default user (possibly >99% of all users) with just one interface and just one route (which is rare or even not existing anymore, all have at least one /31 link, a /32 route to it's p2p-link (point2point, not peer2peer or, maybe again >99%: one default route and one /24 route on the lan)

[toasta]

digging trough libtorrent srccode i found this comment
session_impl.cpp:5078

       // verify that the given local address satisfies the requirements of
        // the outgoing interfaces. i.e. that one of the allowed outgoing
        // interfaces has this address. For uTP sockets, which are all backed
        // by an unconnected udp socket, we won't be able to tell what local
        // address is used for this peer's packets, in that case, just make
        // sure one of the allowed interfaces exists and maybe that it's the
        // default route. For systems that have SO_BINDTODEVICE, it should be
        // enough to just know that one of the devices exist
        bool session_impl::verify_bound_address(address const& addr, bool utp

One of my first thoughts was, that this applies only to udp only anywas (my scenario and my tcpdump)
but as i've set qbittorrent to "tcp-only" i ruled that out.

Is this normal behavior to try udp anyways first when contacting a peer for the first time, even if mode is set to tcp only?

[toasta]

Hmm...
session_impl.cpp:5017 looks promising:

                        for (auto& ls : m_listen_sockets)
                        {
                                if (is_v4(ls->local_endpoint) != remote_address.is_v4()) continue;
                                if (ls->ssl != ssl) continue;
                                if (!(ls->flags & listen_socket_t::local_network))
                                        with_gateways.push_back(ls);

                                if (match_addr_mask(ls->local_endpoint.address(), remote_address, ls->netmask))
                                {
                                        // is this better than the previous match?
                                        match = ls;
                                }
                        }
                        if (!match && !with_gateways.empty())
                                match = with_gateways[random(std::uint32_t(with_gateways.size() - 1))];

looks something like a most_specific "routing" without all the patricia-trie bells and whistels and would explain the random behaviour of rotating trough all available ips...

[FranciscoPombal]

@toasta

I think you know much more about this than I do, I think this discussion is becoming out of reach for me. But if I understand at least the basics correctly, your problem is not simply "I would like to be able to configure qBittorrent to use multiple, but not all of the machine's interfaces/IPs". Your issue is more with what libtorrent is doing behind the scenes, right?

If so, for convenience, and to isolate as many variables as possible, you may want to use libtorrent's client_test for testing: https://github.com/arvidn/libtorrent/blob/RC_1_2/examples/client_test.cpp (If you're building libtorrent with CMake, just pass -Dbuild_examples=ON to the CMake configuring command line).

@arvidn ping, you may want to take a look at this.

[arvidn]

@toasta great report, thanks!

I notice in your dump you only have UDP packets. and the reason you didn't see the source IP in strace was because that UDP socket (you saw the sendmsg() call on it) isn't "connected", it's used as the socket supporting many uTP connections (which runs over UDP). There should have been a call to bind() and possibly BINDTODEVICE right after it was created though.

I can see this being a problem with UDP if libtorrent fails to interpret the routing table (or if it changes after listen sockets and UDP sockets have been set up) where the wrong UDP socket can be chosen for a remote peer.

if you disable utp, and only use TCP connections, does everything work as expected?

[arvidn]

@toasta you're right about that piece of code being the one picking which UDP socket is used for a uTP connection, given a target address. Do you have any specific suggestions of how to improve that selection?

The source hint should be taken into account, and also the metric. Do you know what would make it work in your case?

[toasta]

Hi @arvidn,

btw... should we move this to libtorrent instead of qBittorrent?

I notice in your dump you only have UDP packets. and the reason you didn't see the source IP in strace was because that UDP socket (you saw the sendmsg() call on it) isn't "connected", it's used as the socket supporting many uTP connections (which runs over UDP). There should have been a call to bind() and possibly BINDTODEVICE right after it was created though.

k... have to go back about that a little... I had serious performance problems or "weirdness" happening on my 14.04 and i thought instead of debugging this i could just get away with updating to 20.04 and be fine.
Now i had a new deluge with a shiny new libtorrent and things started to seem fine..

Then some other weirdness started.
I havent ever hear of "utp", just that someone thought it was nice to reimplement all the tcp-features in an own udp overlaynet but this is not rtp/streaming so i thought why reinvent the wheel.. Now with most of the bittorrent client development coming to a halt (at least it seems so) and seeing other clientnames out there i thought i would give qbit and others a try.
There some report that deluge's "high performance seed" settings, which are just one of the presets of libtorrent, is disabling tcp and as the plugin to configure libtorrent in deluge is unable to edit anything but text-input-files... yada yada yada...
It's basically no fun atm :)

I finally found that deluge is willing to accept libtorrent settings from a file and disabled udp.

and the socketstuff should've shown up in strace when i was filtering for %net events.

I can see this being a problem with UDP if libtorrent fails to interpret the routing table (or if it changes after listen sockets and UDP sockets have been set up) where the wrong UDP socket can be chosen for a remote peer.

hmm.. I think it should still respect the fib.
Maybe - on startup and interface changes - it should create one socket foreach interface.
How do you get notified from the kernel about packets beeing ready? that is not select() as for tcp-sockets, or is it?

if you disable utp, and only use TCP connections, does everything work as expected?

I thought so... yes, but now deluge starts to do weird stuff even with tcp.
It's doing the same thing, cycling through all possible ips/interfaces for src-ip but it looks like it's at least respecting the routingtable.

Even after a restart (so no fib/routingtable changes after starting the client) it still tries the ip of

• a down interface
• the hosts public ip
• but none of the tunnel ips
  (yes, it's trying all src ips of the host, even for tcp....)

this time it seems to respect the routing-table (at least). i disabled ecmp for easier debugging...

root@X:~# tcpdump -ni tun0 "host XXXXX"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
17:46:25.000203 IP XXXX.66.34897 > YYYY.105.887766: Flags [S], seq 2225663861, win 64240, options [mss 1460,sackOK,TS val 3745807296 ecr 0,nop,wscale 7], length 0
17:46:25.000226 IP 1.2.3.4.59347 > YYYY.105.887766: Flags [S], seq 3741489093, win 64240, options [mss 1460,sackOK,TS val 3305707649 ecr 0,nop,wscale 7], length 0
17:46:26.030783 IP 1.2.3.4.59347 > YYYY.105.887766: Flags [S], seq 3741489093, win 64240, options [mss 1460,sackOK,TS val 3305708679 ecr 0,nop,wscale 7], length 0
17:46:26.030801 IP XXXX.66.34897 > YYYY.105.887766: Flags [S], seq 2225663861, win 64240, options [mss 1460,sackOK,TS val 3745808326 ecr 0,nop,wscale 7], length 0
17:46:28.046791 IP XXXX.66.34897 > YYYY.105.887766: Flags [S], seq 2225663861, win 64240, options [mss 1460,sackOK,TS val 3745810342 ecr 0,nop,wscale 7], length 0
17:46:28.046813 IP 1.2.3.4.59347 > YYYY.105.887766: Flags [S], seq 3741489093, win 64240, options [mss 1460,sackOK,TS val 3305710695 ecr 0,nop,wscale 7], length 0

root@X:~# ip r get YYYY.105
YYYY.105 dev tun0 src 10.1.2.3 uid 0
cache

telnet on the host to YYYY.105 works fine; kernel selects route and uses ip-address from tun0.

note that this is still

qBittorrent is using libtorrent >1.2.6, and deluge is using 1.1.x (which I think is what libtorrent-rasterbar.so.9.0.0 amounts to, if I'm not mistaken).

1.1.x as it's deluge, so this should be before changes to multihomed, ...
but lsof now says:

deluged 1020781   bf  mem       REG     8,2     3368464   669961 /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.so.10.0.0
deluged 1020781   bf  mem       REG     8,2     1889280   659314 /usr/lib/python3/dist-packages/libtorrent.cpython-38-x86_64-linux-gnu.so

root@bf-spielwiese:~# strings /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.so.10.0.0 | grep -i vers | less
# yeah that didnt work :)
root@bf-spielwiese:~# dpkg -S /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.so.10.0.0
libtorrent-rasterbar10: /usr/lib/x86_64-linux-gnu/libtorrent-rasterbar.so.10.0.0
root@bf-spielwiese:~# apt show libtorrent-rasterbar10
Package: libtorrent-rasterbar10
Version: 1.2.6+git20200425.a9968916ca-1ppa1~20.04

what i find most interesting is that it's using the ip of a "DOWN"-interface:

root@X:~# ip link show dev eth1
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether aa:f7:zz:zz:zz:a8 brd ff:ff:ff:ff:ff:ff
root@X:~# ip a show dev eth1
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether aa:f7:zz:zz:zz:a8 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/24 brd 1.2.3.255 scope global eth1
       valid_lft forever preferred_lft forever

i'll split this into multiple comments... seems already to long....

[toasta]

k... so for "asking for routing from a c++ app" i googled

https://trustica.cz/en/2016/03/22/linux-routing-api-no-abi-maybe-why/
(that's about sourcerouting, but should be applicable)
but they wrote their own api on top of the abi.. Couldnt find the sources so i looked at the RT_NETLINK stuff:

https://man7.org/linux/man-pages/man7/rtnetlink.7.html
  rtnetlink_socket = socket(AF_NETLINK, int socket_type,  NETLINK_ROUTE);
RTM_GETROUTE

              struct rtmsg {
                  unsigned char rtm_family;   /* Address family of route */
                  unsigned char rtm_dst_len;  /* Length of destination */
                  unsigned char rtm_src_len;  /* Length of source */
                  unsigned char rtm_tos;      /* TOS filter */
                  unsigned char rtm_table;    /* Routing table ID;
                                                 see RTA_TABLE below */
                  unsigned char rtm_protocol; /* Routing protocol; see below */
                  unsigned char rtm_scope;    /* See below */
                  unsigned char rtm_type;     /* See below */

                  unsigned int  rtm_flags;
              };

that should already be better (and faster) than using the src-ip of the interface for routing decisions (which here is binding?)
i saw a lot of "gimme interfaces and their ips" in strace, so you might already have something polling interfaces and addresses.
Is this already rt_netlink?

I can see this is way more work to make it work on non unices....

anywho... there seems to be something wrong on a lower layer, as it's affecting tcp with unchanged fib.

I'm happy to help debug/code/resolve this but i'll have a lot of stupid questions so this'll require lots of patience from you :)

[arvidn]

I think we should move here: arvidn/libtorrent#4850
I'll respond in that thread

[FranciscoPombal]

keeping this open to track it more easily until the libtorrent part is resolved.