Discussion: Breaking changes in qBittorrent search-engine

[ngosang]

Update: Proposal => qBittorrent.pdf

---

Hello everybody,

TLDR: I don't have time to maintain the plugins, the search-engine code needs a huge refactor. I'm proposing to remove all search plugins and Python code and make a native integration with https://github.com/Jackett/Jackett

First of all, I think this feature is used by thousands (take a look at Reddit) and qBittorrent is the only bittorrent client with this functionality. So, I don't want to remove it. Furthermore, in the latest releases is supported in the WebUI making it convenient for seedboxes.

The current implementation (code) is a mess, both, the C++ and Python parts. In this issue #84 I list all required changes to make it clean and more maintainable. That are a lot of changes, breaks compatibility, I don't have the skills to make the changes in the C++ code, and more important, there aren't maintainers for the plugins.

Instead of doing the changes, I propose to remove all Python code, including official and unofficial plugins and make native integration with Jackett for the following reasons.

• https://github.com/Jackett/Jackett is a free and open source app that support more than 400 torrent sites. All official and unofficial plugins are supported. They are able to bypass captchas, cloudflare... We can't compete
• It's well maintained. I'm contributor and it provides releases every day. We don't have maintainers to update the plugins frequently and most unofficial plugins are not working. https://github.com/Jackett/Jackett/releases
• They provide a standard REST API based in Torznab. In simple words, it's a XML format well defined (never changes) and It's used by Radarr, Sonarr and many more software. https://github.com/Sonarr/Sonarr/wiki/Implementing-a-Torznab-indexer In case Jackett is discontinued, there are other projects that support Torznab and will be compatible also (see https://github.com/cardigann/cardigann). I'm talking about Jackett but we can make it generic to support all Torznab clients. Will be easy to request and parse the XML in C++ and keep the current UI/WebUI
• We already support Jackett and I know many people using it. They won't have to make big changes. https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin
• Runs in the same platforms as qBittorrent does (win, linux, mac)
• We won't have external dependencies, Python.
• We won't have to worry about updating plugins nor resolving issues.
• We won't have legal problems.
• We can focus in doing the best bittorrent client, and let other projects to take care about the search-engines.

I want to know what do you think about and I would like you to try Jackett https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin

ping @qbittorrent/demigods @qbittorrent/webdev @qbittorrent/frequent-contributors @qbittorrent/bug-handlers

UPDATE: Implementation draft

NOTE: In the first version I will try to make as few changes as possible.

1. Enable search engine

• Python is not required as external dependency. We can remove all code related to check Python version, download python, copy python code...

2. Install plugins / search configuration

• In a clean install, the list will be empty because we don't provide search engines anymore.
• Remove strikethrough parts in the screenshot
  
• When you click "add new indexer" or you "edit one" you will see a modal like this. We should add placeholders, tooltips or something to explain each field.
  
• All fields are mandatory and can be obtained from Jackett UI
  

3. Search UI

• By now we can let the UI as it is. Jackett provides additional information that we can add in the future.
  

4. Perform searches with Torznab

• GET <indexer_url>api?apikey=&q=<search_term>
• Eg GET http://127.0.0.1:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=ygm5k29tomyy11df265gio7iouhqxk4&q=ubuntu
• Eg response XML => https://pastebin.com/kK48jqAc
• Each item is a torrent. The field mapping with the current UI is:
  ** qBittorrent column => Torznab
  ** name => title
  ** size => size (in bytes)
  ** seeders => seeders
  ** leechers => peers - seeders
  ** search engine => torznab indexer name configured by the user (the real name of the indexer isin the response but it can confuse the user)
  ** description page => comments
  ** download link => link (can be a maget or a http link)
• Jackett also support categories. You must use this mapping (several categories are supported separated by comma) => https://github.com/qbittorrent/search-plugins/blob/master/nova3/engines/jackett.py#L64
• Eg GET http://127.0.0.1:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=ygm5k29tomyy11df265gio7iouhqxk4&q=ubuntu&cat=2000,5000

5. Future steps

• Add more columns in search results
  
• Move search engine to preferences ??
• Allow to select several categories in the same search (eg, tv and movies)
• Add all torznab categories. There are a lot.
• Group search results by infohash. Torznab provides a field with this information

[glassez]

We can focus in doing the best bittorrent client, and let other projects to take care about the search-engines.

👍

[Piccirello]

I like it, but I'm already a Jackett user.

[ngosang]

Updated the first message with the changes required.

@Chocobo1 Would you like to work in the development? I can test everything carefully. :)

[Chocobo1]

@Chocobo1 Would you like to work in the development? I can test everything carefully. :)

I probably won't have much time to do anything significant.

[ngosang]

If you don't mind I can help.

Of course you can. You have all the details in the first message. Just mention me when you have a PR or something. If at any point you are not longer interested in implementing this, please, notice me.

[ghost]

Imho the python integration has been there for a long time and it would not be ideal to drop it completely after all the work it required, just drop python2. Also, not all sites are supported and you can't use the "go to the description page" option.

[hannsen]

These would be the sites that we have but jackett does not:

Academic Torrents
ali213.net
anidex.info
CineCalidad
HorribleSubs
Linux Tracker
small-games.info
UnionDHT

But yeah, it sounds like a sensible idea, and there is no reason why these sites above could not be implemented in jackett.

and you can't use the "go to the description page" option.

Jackett does support that

[ngosang]

I opened some requests in Jackett to add them. anidex.info, HorribleSubs are in Jackett already.
@hannsen Maybe you can help porting some of them to Jackett, or just helping Jackett somehow. :)

[jarmo]

Has anything already been done regarding this? I did upgrade my qBittorrent to 4.2.3 after which I still see all the plugins I had before, but "categories" only list "All categories". Also, sometimes plugins do not find anything anymore.

I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers).

Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not).

[ngosang]

As far as I know no one is working in this. Most of the plugins included in qBittorrent are working (just make sure you have them updated). The categories are not tested well, just use "all categories".

I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers).

I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.

Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not).

I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections.
PRs are welcome but I don't see any.

[jarmo]

@ngosang thank you for your thoughts.

The categories are not tested well, just use "all categories".

My point was that they were working before and stopped working after upgrading. That's why I asked. It doesn't really matter to me because I always have been using "all categories" anyway. Just an observation.

I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.

Looking at the Jackett logs that was my initial suspicion too, that it works exactly like you just described in here. Good that you confirmed it, since that explains a lot. When I did enable all (public) indexers in the beginning then I had to start looking at the log and disable some, which were failing thus causing entire search to not return any results. Using your proposed API does make sense and sounds like a good idea.

I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections.

I totally understand this. One way of solving this problem would be to create better integration between Jackett and qBittorrent where Jackett is bundled with qBittorrent or can be installed somehow throught qBittorrent and it would start/stop Jackett server and configure API key automatically or whatever so that end-user would not need to know anything about Jackett or how to configure it etc. I'm quite sure that this is also not a small task and I totally understand that PRs are welcome since I'm developing some OSS stuff too.

Just my thoughts how things could be improved.

[glassez]

I mean as "subtabs" within the search tab itself. A picture is worth a thousand words (imagine both buttons with red text switch between "subtabs"; here, the results "subtab" would be active):

Well, it looks acceptable, except that it loses a bit of vertical space, which might worry someone.

[FranciscoPombal]

@glassez

Well, it looks acceptable, except that it loses a bit of vertical space, which might worry someone.

Good point, agree that horizontal space is cheaper. But the button to bring out the dialog would take up approximately the same amount of space. So I guess the ideal solution would be to position the tabs vertically on the right, the same we do it for the banned IPs tab.

[ngosang]

Well, I'm going to do this job in the meantime.

Nice to hear that. Please, quote me when there is something to test. I can help with that.

[glassez]

Torznab Spec => https://torznab.github.io/spec-1.3-draft/external/newznab/api.html

In the docs above is said:

The returned XML data stream is RSS 2.0 compliant and also contains additional information in the extra namespace.

But actually I see Jackett provide rss version="1.0" (although the content seems to look like RSS 2.0).
@ngosang, can you explain, what happens?

[ngosang]

Yes, you are right. I will fix it in Jackett today. How it affects you?

[glassez]

Yes, you are right. I will fix it in Jackett today.

👍

How it affects you?

I needed to know what format the data is provided in Torznab in order to parse it. When I saw rss version="1.0", I thought, "What the hell is this? What document type is it?". Then I found in Torznab specification that it should be RSS 2.0 (only). I looked closer in XML produced by Jackett, and realized that it was very similar to RSS 2.0, but this declaration still confused me.
If it was just a mistake, then it's okay. Just fix it.

I have a sad experience of working with RSS, when even very soft requirements of the standard are often not met by server side. So in this situation, I would prefer that the Jackett bugs, if found, be fixed in Jackett without any workarounds in qBittorrent.

[ngosang]

@glassez it's already fixed. New Jackett version will be released tomorrow. If you find more things just tell me.
Jackett/Jackett#11871

Released https://github.com/Jackett/Jackett/releases/tag/v0.18.255

[glassez]

Well, I've finally completed the bulk of the work on this (qbittorrent/qBittorrent#15126). It's still a draft, since some little things are missing, such as validating the input data, etc. But you can already try it in practice. At the same time, I would have heard preliminary feedback. Just please don't ask for anything more advanced than what is there. I would like to limit myself to the basic functionality at the first stage. Advanced features can be added later.

[glassez]

I repeat qbittorrent/qBittorrent#15126 (comment) here. Maybe I'll get an answer after all...

Recently, I began to worry about one question. What is the real benefit of integrating the search engine system (of course, I don't mean the old subsystem, but the new one). What is there that can't be done just by having qBittorrent (w/o any Search Engine Integration) and Jackett?
Maybe this question sounds stupid for those who actively use this feature. But I am not an active user of it, so I would like to get an explanation.

[FranciscoPombal]

@glassez qbittorrent/qBittorrent#15126 (comment)

[ghost]

Hello all,

What is the state of the proposed changes on macOS version of QBT 4.5.0 / LT2.0?

According to the WIKI page for Search Plugins this has already been implemented yet I am seeing a nova3 directory wasn't removed upon install of v4.5.0 and I still see Python plugins. No mention of adding a torznab indexer. Do we need to remove the nova3 directory for the new torznab feature to be implemented properly?

[ngosang]

It's not implemented yet and nobody is working on it. The development is here => qbittorrent/qBittorrent#15126
For now we are still using Official / Unofficial plugins written in Python. I recommend you to use the Jackett plugin for those indexers not supported => https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin

[catbrew]

@ngosang I had some issues with Jackett (since resolved) and that led me to discover Prowlarr. I saw that Torznab implementation has stalled, but since Jackett already works I thought that means qBt can talk Torznab well enough already and I could just feed it Prowlarr's info in jackett.json since it's just another Torznab implementation.
It failed.
I would like to know if it's because qBittorrent has been modified to work specifically with Jackett and nothing else. Or if it's a stupid error on my end.

[ngosang]

@amamelia Jacket is working fine. The qBittorrent plugin only works with Jackett but it could be modified to work with Prowlarr.

[Myth1-1]

QB最新版“动漫花园”导入不了

[travibe]

As per -> #121 (comment)

What is the status of all this? version 5 is out now and there have been major issues with the search plugins disappearing/not working. Are we only using Jackkett now? I think the community would really appreciate some very clear language on what the proper usage of Qbittorrent's search function looks like.

[Balls0fSteel]

@travibe The search plugin disappearing / not working has nothing to do with version 5. The affected users either had very old Python runtimes or unmaintained, outdated plugins installed. The plugins auto install, auto update and just work out of the box, just like they did before as of today.

If you require assistance with your installation, please visit our Reddit, Discord or Forums, whichever you prefer.

[travibe]

@travibe The search plugin disappearing / not working has nothing to do with version 5. The affected users either had very old Python runtimes or unmaintained, outdated plugins installed. The plugins auto install, auto update and just work out of the box, just like they did before as of today.

If you require assistance with your installation, please visit our Reddit, Discord or Forums, whichever you prefer.

Thank you, I greatly appreciate you following up on this, and for being so quick as well.

[xavier2k6]

@Balls0fSteel @travibe

qbittorrent/qBittorrent#21579 (comment)

[p060477]

my main question is: should be the new qbittorrent vers. also still PORTABLE...???...i mean that the change you make in the search engines feature will be compatible with the portability of Qbittorrent and Jackett,i've also it in portable way,
cause my main aim is to have as more portable sw on my pc as i can...
thxs so much in adv
cheers