fix collections.abc.Callable and typing.Callable

[KotlinIsland]

typing.Callable is not a _SpecialForm, it's an _Alias, and collections.abc.Callable is a class

i is trying to address many issues with Callable that are either 100% special cased, or don't currently work

from collections.abc import Callable

def f(t: type[object]): ...

f(Callable)

class C(Callable): ...

a: Callable
a.__call__

status

undecided on how to proceed, two options present:

1. fix typing.Callable and introduce _collections_abc._Callable[**P, R]. this is less disruptive (i think)
2. fix typing.Callable and introduce _collections_abc.Callable[**P, R], this is potentially more disruptive, but more correct

[srittau]

I'm always in favor of bringing our stubs more in line with reality, so in general I'm in favor of the change. But unfortunately at least mypy seems heavily dependent on the current typing – as shown by the mypy primer output. We'd need to ensure that current type checkers will work with the updated stubs, before making this change.

Maybe it's also worth trying both changes (to typing and _collections_abc) in isolation to see if both break the world, or if you could at least do half of this PR.

[KotlinIsland]

We'd need to ensure that current type checkers will work with the updated stubs, before making this change.

this is a circular dependency. how can we ever change anything if the change needs to be compatible with our dependants

i will try them in isolation, i feel that the class with generics isn't going to be so well consumed, although i would be interested in pulling in the maintainers of other type checkers to see if they are welcomming to the change (fyi, i maintain pycharm)

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[srittau]

this is a circular dependency. how can we ever change anything if the change needs to be compatible with our dependants

In that case this needs coordination with the dependents, but we won't change this unilaterally.

Just changing typing.Callable looks more promising, indeed, but pyright would obviously needs to be changed as well (although I suspect the change is rather simple).

[AlexWaygood]

this is a circular dependency. how can we ever change anything if the change needs to be compatible with our dependants

That's why I said in astral-sh/ty#1215 (comment):

I think that would be likely to either break existing type checkers, or else have somewhat unexpected effects on their handling of Callable. I'm not sure it's worth the disruption to the ecosystem.

It's not impossible to make a change like this but, as @srittau says, it is nontrivial. You'll first need to get consensus among all the major type checkers that the change is worth making (or at the very least, make sure that they're aware the change is coming, and make sure that they're willing to adapt to the change). Ideally we'd have PRs "ready to go" in the type checkers that need changes before the typeshed change is merged.

[KotlinIsland]

so, any advice on resolving these failing tests? should we summon the maintainer of pyright? (or basedpyright?)

[srittau]

Yes, @erictraut might be able to help.

[AlexWaygood]

this does sort of make me wonder... an alias to what? For all other _Aliases in typeshed, ty is able to defer to an "actual class somewhere else" for that _Alias's semantics in type annotations etc.. But here there is no "actual class somewhere else" -- collections.abc.Callable is just a re-export of this symbol.

(I realise that you've done it this way because you tried the first way and it was much more disruptive...)

[KotlinIsland]

i see, i have addressed this in a comment in the "Conversion" tab

one option would be to introduce a _Callable[**P, R] definition

[erictraut]

This is a pretty disruptive change. Is there a specific problem that it solves? I'm not against making improvements if there's a real benefit to users or type checker authors, but it's not clear to me what those benefits are in this case.

As @AlexWaygood mentioned, if we want to model types.Callable as an _Alias, then I think there needs to be some other symbol that it aliases. It can't alias itself. Presumably, it aliases collections.abc.Callable.

But then how is collections.abc.Callable defined? If it's not defined as a _SpecialForm, then presumably it would be defined as a normal class definition? Defining it in that way will require some hard-coded logic in type checkers because Callable does not follow the normal rules for a normal class.

[KotlinIsland]

Is there a specific problem that it solves

primarily that Callable is actually a type, and has a __call__ attribute, and the flow-on consequences of that

from collections.abc import Callable

def f(t: type[object]): ...

f(Callable)

class C(Callable): ...

a: Callable
a.__call__

will require some hard-coded logic in type checkers because Callable does not follow the normal rules for a normal class.

what exactly are the different rules to a normal class? i'm not aware of any, i would expect:

class Callable[**Parameters, Return](Protocol): # not actually a protocol, but acts like one, same as everything else in collections.abc
    @abstractmethod
    def __call__(self, *args: Parameters.args, **kwargs: Parameters.kwargs) -> Return: ...

how much effort would be involved in un-special casing Callable and just using this new defintion? i'd imagine there would need to be some logic to connect the alias to the definition, but the rest would just be removing of special-casing?

so i see two options:

1. introduce a _Callable[**P, R] into collections.abc, this would be the definition for the alias
2. just fix the problem in full and make the definition as it should be, this would likely require more changes in type checkers than option 1

[erictraut]

I'm still trying to understand if the proposed change is solving an actual problem that users are hitting or if this falls more under the category of "it would be nice if this were more aligned with the runtime"? If it's the latter, then I think we need to consider the impact and cost of the change and weigh it against the benefits.

how much effort would be involved in un-special casing Callable and just using this new defintion?

I can't speak for other type checkers, but for pyright it's not as simple as "un-special casing". Callable does not act like normal classes in a number of respects, so special-casing is still required. Notably, when used as a type form, it accepts a list expression for its first type argument. It also accepts ... (which has a special meaning) and the Concatenate special form. I'd need to investigate further to enumerate all of the special casing.

[KotlinIsland]

I'm still trying to understand if the proposed change is solving an actual problem that users are hitting

i have run into these issues when trying to write code

additionally, it is very confusing to see that the definition of Callable is Callable: _SpecialForm

Notably, when used as a type form, it accepts a list expression for its first type argument. It also accepts ... (which has a special meaning) and the Concatenate special form.

i don't think there is anything special about these semantics at all:

Code sample in basedpyright playground

from collections.abc import Callable
from typing import Concatenate

class CustomCa[**P, R]: ...

_ = Callable[..., int]
_ = CustomCa[..., int]

def f[**P](
    a: Callable[[Concatenate[int, P]], None],
    b: CustomCa[[Concatenate[int, P]], None],
): ...

the only special casing that i am aware of is the assumption that Callable has a definition of __get__ that is the same as FunctionType.__get__