is_type_comment() is wrong

[JelleZijlstra]

Currently, the implementation of is_type_comment() looks like this:

def is_type_comment(leaf: Leaf, suffix: str = "") -> bool:
    """Return True if the given leaf is a special comment.
    Only returns true for type comments for now."""
    t = leaf.type
    v = leaf.value
    return t in {token.COMMENT, STANDALONE_COMMENT} and v.startswith("# type:" + suffix)

But this is wrong, because typed-ast also recognizes a comment with multiple spaces after the # as a type comment. Concretely this could lead Black to skip some of the checks we have to avoid moving type comments to the wrong line.

Ideally, Black should also standardize the formatting of type comments by cleaning up excessive whitespace after the # and after the :.

[Pierre-Sassoulas]

I think there is also the pragmas in contains_pragma_comment at: comment.value.startswith(("# type:", "# noqa", "# pylint:")

[felix-hilden]

Should we also standardise spaces before the colon? Meaning we would accept (and fix) e.g. # type : Type or even # \t type : (some unexpected whitespace type) Type?

[itxasos23]

Hey everyone! Can I get some maintainer reivews on the PR?

[xantric]

@Pierre-Sassoulas Hello... I am new here and I would like to start contributing in the repository. So can you please assign this issue to me
Thank you

[Pierre-Sassoulas]

Hey @ishankamboj, sorry I'm not a black maintainer. I think you can start implementing right away and open a PR though :)

[debapriyakumar]

hello i am new here and would like to contribute to the project can you help me out

[Pedro-Muller29]

Hi everyone,

Referring to @ichard26's comment on PR#2698:

"we can't really normalize type comments until all of the major AST libraries that provide first-class type comment support are liberal in the whitespace they understand. It wouldn't be good to have a type comment that's ignored by typed-ast transformed into one that is noticed which then affects a type checker like mypy."

I have a few questions:

1. is_type_comment Function: Is the issue with the is_type_comment function's handling of extra spaces still present? Is addressing that without standardizing type comments something desired?

2. Standardizing Type Comments: Given the potential side effects mentioned, is standardizing type comments still desired?

3. Pragma Comments: Do the same concerns apply to pragma comments regarding whitespace sensitivity and standardization?

Looking forward to the maintainers' insights on these points. Thanks!

[JelleZijlstra]

Is the issue with the is_type_comment function's handling of extra spaces still present?

Yes, look at black.nodes.is_type_comment in the source.

Is addressing that without standardizing type comments something desired?

I think so. While the typing spec does not say anything about whitespace in these comments, I believe all type checkers use whitespace insensitivity.

Given the potential side effects mentioned, is standardizing type comments still desired?

There are multiple possible meanings of "standardizing". I would support a clarification to the typing spec about whitespace sensitivity of type comments. I would also support a change to Black to treat whitespace in type comments correctly, but we have to be careful that we do not break Black's stability policy.

Do the same concerns apply to pragma comments regarding whitespace sensitivity and standardization?

Other pragma comments are specific to individual tools. We'd have to look at those tools to figure out how they handle whitespace.

[Pierre-Sassoulas]

In pylint the regex used is here:

https://github.com/pylint-dev/pylint/blob/0972ba526d9ab010441f936fbcd034e18389cf27/pylint/utils/pragma_parser.py#L11-L26

As you can easily see (😄) there can be multiple white-spaces.

[Pedro-Muller29]

I have implemented type comment standardization, but the following tests I have written won't pass:

# test type comments
def fb(
    f,  #  type :  int  
    h,  # type: int
    i,  # type: int
):
    # type: (...) -> None
    pass


# output
# test type comments
def fb(
    f,  # type: int
    h,  # type: int
    i,  # type: int
):
    # type: (...) -> None
    pass

Even though the code will be formatted correctly, the output will produce a different AST because the python ast module does not recognize the following line as a type comment:

f,  #  type :  int  

But if it is formatted correctly, it will:

f,  # type: int

This is the diff given by the unit test:

annotation=
E                            None,  # NoneType
E                            arg=
E                            'f',  # str
E                            type_comment=
E           -                None,  # NoneType
E           +                'int',  # str
E                        )  # /arg
E                        arg(
E                            annotation=
E                            None,  # NoneType
E                            arg=

Question:

How should I go about implementing type comment standardization considering these issues?
I am new to the project, is there something here I am missing?

[JelleZijlstra]

I think we shouldn't normalize whitespace between "type" and the colon. Mypy also doesn't recognize # type : int as a type comment.

[shivamkumar71]

i am interested to contribute in this repository . please assign me