Write up a report / summary with feedback for mypyc

[ichard26]

Mypyc is still alpha software (which is funny given I'm trying to integrate it with "beta" software) so there's a lot of rough edges. Now they aren't that hard to workaround but it would be great if I didn't need to do 'em in the first place. So to be a helpful user of mypyc and help push it forward, let's write some sort document containing feedback, suggestions, other things we noticed.

What follows is a draft ...

---

First of all, here's the relevant links:

• Development branch: psf/black@main...mypyc-support-pt2 (contains crash workarounds, optimizations, and QOL improvements) - Implementing mypyc support pt. 2  psf/black#2431
• Wheel building workflow + other infra + project management homebase: https://github.com/ichard26/black-mypyc-wheels
• Relevant issue on the psf/black repository: Performance optimization discussion psf/black#366
• Performance report: https://gist.github.com/ichard26/b996ccf410422b44fcd80fb158e05b0d

The good

• The performance wins are excellent: ~2x average across multiple workloads!
• As long as the hot code wasn't too spread out, optimizing for mypyc wasn't that hard (tightening up types, adding lots of Final, etc.) - additional performance wins of 5-15% were achieved :D
• There weren't actually that many crashes or other incompatibilities (ignoring dataclasses) and they all had easy enough workaround
• Wheel building was pretty easy and straightforward (once some changes were made to the setup.py logic) - I did steal lots of ideas from mypyc/mypy_mypyc-wheels :P

The bad

• Dataclasses were responsible for the majority of crashes and other compatibility issues. The cases that led to issues include:
  • an dataclass that is also subclassing ast.NodeVisitor crashes with AttributeError: attribute 'dict' of 'type' objects is not writable (maybe not fixable?)
  • an dataclass inheriting from Generic (already reported in Dataclass + Generic fails during sleight of hand mypyc/mypyc#827)
  • abc.ABC + dataclass crashes too
• Unanalyzed or unreachable code continues to be problematic - a "fun" case involved this:

  >   if sys.platform == "win32":
  RuntimeError: Reached allegedly unreachable code!
  

  This was caused by mypy.ini hard-coding platform=linux. Other interesting case was one involving a module typed as Any causing an else branch to appears as it's never ever taken. The resulting unanalyzed code crashed when hit. More details in this commit: psf/black@acb77f7

And other various issues already reported on the mypyc issue tracker:

• GCC on Linux fails to compile tuple assignment with somewhat complex type annotation with "array subscript 1 is above array bounds" mypyc/mypyc#885
• Compiled nested functions don't retain their __name__ attribute mypyc/mypyc#884
• A class definition nested in *anything* crashes mypyc with KeyError mypyc/mypyc#864
• Class level scoping is miscompiled mypyc/mypyc#862
• Windows Python 3.5/3.6 x64 issues mypyc/mypyc#819 bit us too :(

Suggestions

• Add a note about platform=$OS being problematic
• Add a note about docstrings being stripped - the click CLI toolkit uses docstrings to get command descriptions ... which ended breaking without a clear reason why (until I stumbled on some issue on mypyc issue tracker that explained why :p)
• Prioritize Reduce mypy/c package size, using -g0 CFLAG mypyc/mypyc#849 - while I'm OK with wheel sizes going up adding mypyc, the ~25-35x (120KB -> ~3.3MB) I observed wasn't the easiest thing to swallow. Adding CFLAGS=-g0 brought the wheels to a much more reasonable ~1MB each.
• Prioritize [mypyc] Introduce subcommand to mypyc command line interface python/mypy#10395 - quick iteration and experimentation was slowed down by repeated cleaning up + python -c "import foo"
• Perhaps add a way to skip invoking the C extension build - this would make analyzing differences at the IR or C level faster ... although I would understand if this is too low-level and shouldn't be made easier

Questions

• Can anything be done about analysis + Python-to-C transcompilation being done twice during any modern wheel build? Unfortunately any evaluations of setup.py end up triggering the above and it turns out both pip wheel and python -m build call at-least two PEP 517 hooks that eventually end up evaluating setup.py. It's not a big deal, but it slows down wheel builds a little bit. Sounds hard to avoid tho (caching would be complicated).
• What else can we do to help mypyc mature and overall get better?

What's next?

Currently the stability and performance data for Linux were just collected and look good so a PR was opened against psf/black. There's a lot of TODOs listed in that PR but it is progress.

The main thing is that I need to do is getting the wheels significantly more battle tested so they can eventually find their way to PyPI. First of all I need to run a mypy-primer equivalent on Windows and MacOS. Assuming that doesn't turn up anything scary, the PR will probably get merged once the reviews are over with. Then at that point, it's off to community testing where we'll be asking members of the community to use the experimental wheels and report issues. So expect updates on this issue as more stability data comes in ... but I'm sure this is already enough insights for a post ^^

Finally

As much as this is riddled with negativity and bugs, I appreciate all of the work y'all put into mypyc. It's awesome to see so much work into making Python code faster lately! 🖤

[ichard26]

Whelp, I just posted the above: mypyc/mypyc#886. I plan to update it as more data comes in from testing, but it's a good start.

[ichard26]

This is done. Further updates will come in blog posts.