Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementing mypyc support pt. 2 #2431

Merged
merged 28 commits into from
Nov 16, 2021
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
9531e1b
Initial mypyc support changes
ichard26 Jun 6, 2021
6e9e0fb
Make the test suite usable for testing
ichard26 Jun 6, 2021
acb77f7
Fix mypyc KeyError on src/black/parsing.py
ichard26 Jun 12, 2021
a37fb77
Saves ~100 kB on my Linux machine :)
ichard26 Jun 30, 2021
2ccc774
Strings specific micro-optimization (1-10% perf boost)
ichard26 Jun 30, 2021
6f60e6e
Looks like I'll be marking more and more tests
ichard26 Jul 1, 2021
f508be4
Fix mypyc + Black on Windows
ichard26 Jul 10, 2021
94dcddb
Ask mypy to warn on unreachable code
ichard26 Jul 10, 2021
57a25ed
Clean up mypyc setup in setup.py
ichard26 Jul 10, 2021
8f42f28
Merge branch 'main' into mypyc-support-pt2
ichard26 Jul 17, 2021
f6a3e78
Initial mypyc optimizations - 5% faster parsing
ichard26 Jul 27, 2021
911d0d8
More parsing optimizations - 4% faster
ichard26 Jul 27, 2021
1f0df05
Just some cleanup
ichard26 Jul 31, 2021
58fbe9c
--version now indicates whether black is compiled
ichard26 Aug 1, 2021
c7de2ea
Round 3 of optimizations - 95% black + 5% blib2to3
ichard26 Aug 3, 2021
b956802
Merge branch 'main' into mypyc-support-pt2
ichard26 Aug 8, 2021
eaa4f6c
Fix crashes and errors since merge from main
ichard26 Aug 7, 2021
e9834e0
Mild hack so mypyc doesn't break diff-shades + cleanup
ichard26 Aug 11, 2021
f103dc0
Address feedback & cleanup comments
ichard26 Aug 21, 2021
5fc39fe
Merge branch 'main' into mypyc-support-pt2
ichard26 Oct 28, 2021
2f238ca
Skip a few more monkeypatching tests
ichard26 Oct 28, 2021
f561b0c
Bring back ignore for unused type ignore
ichard26 Oct 28, 2021
11b4f09
Merge branch 'main' into mypyc-support-pt2
ichard26 Oct 30, 2021
b0b3709
Merge branch 'main' into mypyc-support-pt2
ichard26 Oct 31, 2021
bb86dcf
Merge branch 'main' into mypyc-support-pt2
ichard26 Nov 14, 2021
5ef46f4
Optimizations + compatiblity fixes
ichard26 Nov 14, 2021
7c52cdb
Merge branch 'main' into mypyc-support-pt2
ichard26 Nov 14, 2021
f5f1099
Fix crash on PyPy by deoptimizing :(
ichard26 Nov 16, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix mypyc KeyError on src/black/parsing.py
Typing the ast3 / ast27 variables as Any breaks mypyc. More details in
the comment added in this commit.

Many thanks goes to Jelle for helping out, with their insight I was
able to find a workaround.
ichard26 committed Jun 30, 2021
commit acb77f7951229b2670edc3fad366ec256fa0c86f
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -32,3 +32,6 @@ optional-tests = [
"no_python2: run when `python2` extra NOT installed",
"no_blackd: run when `d` extra NOT installed",
]
markers = [
"incompatible_with_mypyc: run when testing mypyc compiled black"
]
19 changes: 17 additions & 2 deletions src/black/parsing.py
Original file line number Diff line number Diff line change
@@ -5,6 +5,11 @@
import sys
from typing import Any, Iterable, Iterator, List, Set, Tuple, Type, Union

if sys.version_info < (3, 8):
from typing_extensions import Final
else:
from typing import Final

# lib2to3 fork
from blib2to3.pytree import Node, Leaf
from blib2to3 import pygram, pytree
@@ -134,6 +139,10 @@ def parse_ast(src: str) -> Union[ast.AST, ast3.AST, ast27.AST]:
return ast27.parse(src)


ast3_AST: Final[Type] = ast3.AST
ast27_AST: Final[Type] = ast27.AST


def stringify_ast(
node: Union[ast.AST, ast3.AST, ast27.AST], depth: int = 0
) -> Iterator[str]:
@@ -173,7 +182,13 @@ def stringify_ast(
elif isinstance(item, (ast.AST, ast3.AST, ast27.AST)):
yield from stringify_ast(item, depth + 2)

elif isinstance(value, (ast.AST, ast3.AST, ast27.AST)):
# Note that we are referencing the typed-ast ASTs via global variables and not
# direct module attribute accesses because that breaks mypyc. It's probably
# something to do with the ast3 / ast27 variables being marked as Any leading
# mypy to think this branch is always taken, leaving the rest of the code
# unanalyzed. Tighting up the types for the typed-ast AST types avoids the
# mypyc crash.
elif isinstance(value, (ast.AST, ast3_AST, ast27_AST)):
yield from stringify_ast(value, depth + 2)

else:
@@ -192,7 +207,7 @@ def stringify_ast(
# To normalize, we strip any leading and trailing space from
# each line...
stripped = [line.strip() for line in value.splitlines()]
normalized = lineend.join(stripped)
normalized = lineend.join(stripped) # type: ignore[attr-defined]
# ...and remove any blank lines at the beginning and end of
# the whole string
normalized = normalized.strip()