[disablebot] Allow reading of disable issue for multiple tests

[clee2000]

The expected format of a disable issue for multiple tests is
Title: DISABLED MULTIPLE

Body:
additional info
disable the following tests:

test_name (test_suite): platforms, mac
test_name2 (test_suite):

additional info

For example

---

Tested by running old version and new version and diffing the files. There was no change

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
torchci	⬜️ Ignored (Inspect)	Visit Preview	Jan 9, 2025 6:52pm

[huydhn]

Do you think it would be simpler to not have an aggregated issues for tests disabled by the bot? #6044 (comment). Basically,

• Tests disabled manually via creating issue like we have today
• Tests disabled automatically by the bot go straight to https://github.com/pytorch/test-infra/blob/generated-stats/stats/disabled-tests-condensed.json

[ZainRizvi]

Do you think it would be simpler to not have an aggregated issues for tests disabled by the bot? #6044 (comment). Basically,

• Tests disabled manually via creating issue like we have today
• Tests disabled automatically by the bot go straight to https://github.com/pytorch/test-infra/blob/generated-stats/stats/disabled-tests-condensed.json

The challenge here is that we need a way to notify test owners when their tests get disabled. A json file is handy for tracking, but needs more support to make the new tests discoverable

[ZainRizvi]

Alternatively, what about if we supported prefixes in the disable issue name instead?

For example, the title could be "DISABLED PREFIX test_foo_bar" and that issue would only disable test starting with that prefix. (And it could list them out so that the disabling doesn't become overzealous)

Given that test storms are likely to affect related tests, that approach may do the trick while still letting us create separate issues for tests that are different enough.

[huydhn]

IMO, we could focus on cleaning up the spams issue first, for example, write a script to close all disabled tests issue what match a certain pattern, and follow up with a proper design discussion on this. I suspect that there might not be a quick solution for this after all

[huydhn]

Should we create a mock disabled multiple tests issue on PyTorch and try to run the script manually to confirm?

[huydhn]

Also, do you remember if we take the title of the issue into account? I wonder if we should enforce something like the disabled tests in the issue body need to belong to the test file / test class set by the title

[clee2000]

Should we create a mock disabled multiple tests issue on PyTorch and try to run the script manually to confirm?

Tested with pytorch/pytorch#144481, it did generate the 3 extra tests in the file

Also, do you remember if we take the title of the issue into account? I wonder if we should enforce something like the disabled tests in the issue body need to belong to the test file / test class set by the title

It does not. The only ways that I know of to validate that an issue only contains tests from a certain file would be to run the test file or to check a database. I think these two options are both a bit excessive. It is also possible that there is one underlying reason for multiple tests to break. However, the flaky test bot does have information about tests when it makes disable issues, so my plan is to have the bot separate by file when makes the issues, but not have anything to enforce this separation if a user decides they want to do something else