qBittorrent crashes on start: check_invariant: current_stats_state() == ...

[WGH-]

libtorrent version (or branch): version: 1.2.6.0 ac4dd41
platform/architecture: Linux/amd64
compiler and compiler version: gcc 9.3.0

I'm not sure whether this is qBittorrent's or libtorrent's problem, but the error message asks to report to both.

I have a debug build of libtorrent with invariant checks enabled, and qBittorrent crashes a few seconds after start.

file: 'torrent.cpp'
line: 7909
function: check_invariant
expression: current_stats_state() == int(m_current_gauge_state + counters::num_checking_torrents) || m_current_gauge_state == no_gauge_state

stack:
1: libtorrent::assert_fail(char const*, int, char const*, char const*, char const*, int)
2: libtorrent::torrent::check_invariant() const
3: void libtorrent::check_invariant<libtorrent::torrent>(libtorrent::torrent const&)
4: libtorrent::torrent::status(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>)
5: void libtorrent::torrent_handle::sync_call<void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*&&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const
6: libtorrent::torrent_handle::status(libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const
7: NativeTorrentExtension::on_pause()
8: libtorrent::torrent::do_pause(libtorrent::flags::bitfield_flag<unsigned char, libtorrent::pause_flags_tag, void>)
9: libtorrent::torrent::on_piece_hashed(libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>, libtorrent::digest32<160l> const&, libtorrent::storage_error const&)
10: libtorrent::disk_io_job::call_callback()
11: libtorrent::disk_io_thread::call_job_handlers()
12: boost::asio::detail::completion_handler<std::_Bind<void (libtorrent::disk_io_thread::*(libtorrent::disk_io_thread*))()> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)
13: boost::asio::detail::scheduler::run(boost::system::error_code&)
14:
15:
16:
17: clone

See also qbittorrent/qBittorrent#12867

[arvidn]

I think that invariant check is probably wrong. It's clear that the torrent is just about to not be in checking mode anymore. I imagine the torrent state is updated slightly out-of-sync with the counters there, causing the invariant to be broken.

Does this happen when a file that's being checked encounters some kind of disk error? (that would explain why this case doesn't have good test coverage)

[WGH-]

Is missing files constitutes such an error? I do have several torrents permanently errored due to missing files. I can check which torrent it is and what seems wrong with it.

…

On May 22, 2020 12:24:31 AM GMT+03:00, Arvid Norberg ***@***.***> wrote: I think that invariant check is probably wrong. It's clear that the torrent is just about to not be in checking mode anymore. I imagine the torrent state is updated slightly out-of-sync with the counters there, causing the invariant to be broken. Does this happen when a file that's being checked encounters some kind of disk error? (that would explain why this case doesn't have good test coverage)

[arvidn]

you could try to build with invariant checks disabled, or comment out that specific invariant check that's failing to see. Also, the error should be visible in the debugger (whatever is causing the call to on_pause()).

Actually, come to think of it, this could be caused by the stop_when_ready flag, or queuing logic.

[WGH-]

$71 = {ec = {val_ = 2, failed_ = true, cat_ = 0x555555c77980 <boost::system::detail::cat_holder<void>::system_category_instance>}, file_idx = 15, operation = libtorrent::operation_t::file_open}

This is is strace fragment just before the crash. Yes, it's the same file being attempted to be opened multiple times.

[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/.${INFOHASH}.parts", O_RDONLY) = 102
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/.${INFOHASH}.parts", O_RDONLY) = 90
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)
[pid 2746739] openat(AT_FDCWD, "/home/wgh/Downloads/${FILE}", O_RDONLY|O_NOATIME) = -1 ENOENT (No such file or directory)

Yes, the torrent indeed has missing files, because I usually don't bother deleting torrents from the qBittorrent interface.

[arvidn]

I just noticed this stack frame:

NativeTorrentExtension::on_pause()

Is that part of qbt? That's probably why I haven't been able to reproduce this

[arvidn]

Could you (or someone else seeing this having this problem) give this a try? #4677

[WGH-]

@arvidn I applied the patch, and I've got another stack trace for you. It's the same torrent file with missing files.

file: 'torrent.cpp'
line: 7959
function: check_invariant
expression: want_tick() == m_links[aux::session_interface::torrent_want_tick].in_list()

stack:
1: libtorrent::assert_fail(char const*, int, char const*, char const*, char const*, int)
2: libtorrent::torrent::check_invariant() const
3: void libtorrent::check_invariant<libtorrent::torrent>(libtorrent::torrent const&)
4: libtorrent::invariant_checker_impl<libtorrent::torrent> libtorrent::make_invariant_checker<libtorrent::torrent>(libtorrent::torrent const&)
5: libtorrent::torrent::status(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>)
6: auto boost::asio::io_context::dispatch<libtorrent::torrent_handle::sync_call<void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*&&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}>(libtorrent::torrent_handle::sync_call<void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*&&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}&&)
7: void libtorrent::torrent_handle::sync_call<void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::torrent::*)(libtorrent::torrent_status*, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>), libtorrent::torrent_status*&&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const
8: libtorrent::torrent_handle::status(libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const
9: NativeTorrentExtension::on_pause()
10: libtorrent::torrent::do_pause(libtorrent::flags::bitfield_flag<unsigned char, libtorrent::pause_flags_tag, void>)
11: libtorrent::torrent::set_paused(bool, libtorrent::flags::bitfield_flag<unsigned char, libtorrent::pause_flags_tag, void>)
12: libtorrent::torrent::on_piece_hashed(libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>, libtorrent::digest32<160l> const&, libtorrent::storage_error const&)
13: std::_Function_handler<void (libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>, libtorrent::digest32<160l> const&, libtorrent::storage_error const&), std::_Bind<void (libtorrent::torrent::*(std::shared_ptr<libtorrent::torrent>, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>, libtorrent::digest32<160l> const&, libtorrent::storage_error const&)> >::_M_invoke(std::_Any_data const&, libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>&&, libtorrent::digest32<160l> const&, libtorrent::storage_error const&)
14: std::function<void (libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>, libtorrent::digest32<160l> const&, libtorrent::storage_error const&)>::operator()(libtorrent::aux::strong_typedef<int, libtorrent::aux::piece_index_tag, void>, libtorrent::digest32<160l> const&, libtorrent::storage_error const&) const
15: libtorrent::disk_io_job::call_callback()
16: libtorrent::disk_io_thread::call_job_handlers()
17: boost::asio::detail::completion_handler<std::_Bind<void (libtorrent::disk_io_thread::*(libtorrent::disk_io_thread*))()> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)
18: boost::asio::detail::scheduler::run(boost::system::error_code&)
19:
20:
21:
22: clone

[WGH-]

(gdb) p want_tick()
$14 = false
(gdb) p m_links._M_elems[aux::session_interface::torrent_want_tick.m_val]
$15 = {index = 0}

[WGH-]

I'm going to iteratively add update_xxx() calls until this's either fixed or I get something really confusing, and will report the results. Since update_want_tick() helped me get past the last assertion, I assume update_want_scrape() will help me get past this:

expression: (m_paused && m_auto_managed && !m_abort) == m_links[aux::session_interface::torrent_want_scrape].in_list()

EDIT: see also discussion in #4677 #4693

[zywo]

For me I have this:

file: '/home/zywo/libtorrent/src/bt_peer_connection.cpp'
line: 337
function: write_dht_port
expression: m_sent_bitfield

[arvidn]

The original issue of this ticket appears to be caused by a plugin altering the libtorrent state in one of its callbacks. This is reasonable and expected behavior from a plugin, but it's not something that the current invariant checks sprinkled throughout libtorrent has taken into account. I made an effort to address this, but it turns out to be non-trivial.

I'm tempted to simply suggest disabling invariant checks when using plugins. At least for now.

@zywo is your case also when using a plugin?

[luzpaz]

@zywo ping 👆