Lunch Time Python

Lunch 9: mypy

mypy is a static type checker for Python. By adding type annotations to your code mypy can find a variety of bugs. These type annotations also act as machine-checked documentation of your code, and your IDE can make use of them to improve its code completion. They doesn't affect how your program runs, as the Python interpreter ignores these type annotations at run-time

Press Spacebar to go to the next slide (or ? to see all navigation shortcuts)

Lunch Time Python, Scientific Software Center, Heidelberg University

Motivation

• Python is a dynamically typed language
  • It infers the type of objects automatically
  • "Duck typing" - if something looks like a duck and quacks like a duck, it's a duck
• This makes Python very flexible and convenient compared to a statically typed language like C
  • But this flexibility also leaves room for bugs, makes static analysis difficult
  • Harder to maintain and understand large complicated projects without type safety
• Solution: add optional type annotations (or hints) to your code
  • mypy can use these to check the code for bugs
  • they are ignored by the Python interpreter at run-time

How does this help?

Imagine you call a function with the wrong type of object

• This may be a runtime error, e.g. str + int
  • This is a good bug: the program stops at the place where the bug is!
  • Assuming your test suite executes this line, you find it by running the tests
  • But even better would be to have your IDE point this out as you write the code
• This may not be an error, just do something undesirable
  • Eventually this may cause a runtime error or incorrect output
  • Hopefully some test fails, but could be very far away from the original bug
  • This is a bad bug: can be hard to trace back to the root cause
  • If your IDE points out the root cause as you type it this is a big win

mypy installation

• Conda: conda install mypy
• Pip: python -m pip install mypy

mypy use

• mypy my_file_to_check.py

jupyter notebook use

These slides also use the nb-mypy extension:

• python -m pip install nb-mypy

This automatically runs mypy on every cell before it is executed.

This is just for convenience to show the mypy output for this talk.

In general I would recommend running mypy separately or as a pre-commit hook.

if "google.colab" in str(get_ipython()):
    !pip install nb-mypy -qqq
%load_ext nb_mypy

Version 1.0.5

Hello world

def greet(thing):
    return f"Hello {thing}"

print(greet("world"))

Hello world

print(greet(True))

Hello True

print(greet(greet))

Hello <function greet at 0x7fb2e5f1ca60>

def greet(thing: str) -> str:
    return f"Hello {thing}"

print(greet("world"))

Hello world

print(greet(True))

<cell>1: error: Argument 1 to "greet" has incompatible type "bool"; expected "str"  [arg-type]

Hello True

print(greet(greet))

<cell>1: error: Argument 1 to "greet" has incompatible type "Callable[[str], str]"; expected "str"  [arg-type]

Hello <function greet at 0x7fb2d47bbd90>

Basic types

• to annotate an object, add its type after a :
  • my_string: str
• to annotate the return type of a function, add the type after ->
  • def hello() -> str:
• intrinsic types like None, int, float, str, bool can be used directly
• other types like list, dict, tuple, set
  • import a capitalized version from typing: from typing import List
  • use list directly (only with Python >= 3.9)

def mul(a, b):
    return a * b


print(mul(2, 2))

4

print(mul(2, [0]))

[0, 0]

print(mul(2, "really?"))

really?really?

def mul(a: float, b: float) -> float:
    return a * b


print(mul(2, 2))

4

print(mul(2, [0]))

<cell>1: error: Argument 2 to "mul" has incompatible type "list[int]"; expected "float"  [arg-type]

[0, 0]

print(mul(2, "really?"))

<cell>1: error: Argument 2 to "mul" has incompatible type "str"; expected "float"  [arg-type]

really?really?

# does operation & returns result
def append1(l):
    return l + [1]


# does operation in place
def append2(l):
    l.append(1)


l0 = [0]
print(append1(l0))

[0, 1]

print(append2(l0))

None

from typing import List


def append1(l: List[int]) -> List[int]:
    return l + [1]


def append2(l: List[int]) -> None:
    l.append(1)


l0 = [0]
print(append1(l0))

[0, 1]

print(append2(l0))

<cell>1: error: "append2" does not return a value  [func-returns-value]

None

def count(l: List[str]):
    return len(l)


a = []
# a: List[str] = []
# a.append("ok")
# a = ["ok"]
print(count(a))

<cell>5: error: Need type annotation for "a" (hint: "a: List[<type>] = ...")  [var-annotated]

0

from typing import List, Tuple


def count(coords: List[Tuple[float, float]]) -> int:
    return len(coords)


print(count([(0, 0), (1, 1), (2, 2)]))

3

from typing import Dict


def invert(ages: Dict[str, int]) -> Dict[int, str]:
    return {key: value for value, key in ages.items()}


ages = {"bob": 2, "joe": 7}
names = invert(ages)
print(names)

{2: 'bob', 7: 'joe'}

Generic collections

• if you can do "for" to iterate over the object
  • my_obj: Iterable
• if it's like a list
  • my_obj: Sequence
• if it's like a read-only dict
  • my_obj: Mapping
• if it's a dict we can modify
  • my_obj: MutableMapping

from typing import Iterable


def count(items: Iterable):
    i = 0
    for item in items:
        i += 1
    return i


print(count(["a", "b", "c"]))

3

from typing import Sequence


def last(items: Sequence):
    return items[-1]


print(last([1, 2, 3]))

3

print(count(["a", "b", "c"]))
from typing import Dict, Mapping


def invert(ages: Mapping[str, int]) -> Dict[int, str]:
    # ages["simon"] = 12
    return {key: value for value, key in ages.items()}


ages = {"bob": 2, "joe": 7}
names = invert(ages)
print(names)

3
{2: 'bob', 7: 'joe'}

Flexible types

• if an object could have several types, use typing.Union
  • my_obj: Union[str, float]
• if an object can be either a dict or None
  • my_obj: Union[Dict, None]
  • my_obj: Optional[Dict]
• if an object can be anything
  • my_obj: Any

def hi(name=None):
    if name is None:
        name = "you"
    print(f"hello {name}")


hi()
hi("joe")

hello you
hello joe

from typing import Union


def hi(name: Union[str, None] = None) -> None:
    if name is None:
        name = "you"
    print(f"hello {name}")


hi()
hi("joe")

hello you
hello joe

from typing import Optional


def hi(name: Optional[str] = None) -> None:
    if name is None:
        name = "you"
    print(f"hello {name}")


hi()
hi("joe")

hello you
hello joe

Generics

• TypeVar specifies a generic type
  • Possible types can optionally be constrained
  • Within a scope it represents a single type
  • For any c++ programmers it's a bit like a template<typename T>

# concatenation of lists of a single type
def add(x, y):
    return x + y


# desired use
print(add([1, 2], [3]))
print(add(["A"], ["B"]))

[1, 2, 3]
['A', 'B']

# don't want to allow e.g.
print(add([1.0], ["B"]))  # this shouldn't be allowed
print(add([[0]], [{1, 2}]))  # this shouldn't be allowed

[1.0, 'B']
[[0], {1, 2}]

from typing import TypeVar, List

T = TypeVar("T")


def add(x: List[T], y: List[T]) -> List[T]:
    return x + y


print(add([1, 2], [3]))
print(add(["A"], ["B"]))

[1, 2, 3]
['A', 'B']

add([1.0], ["B"])  # this shouldn't be allowed
add([[0]], [{1, 2}])  # this shouldn't be allowed

<cell>1: error: Cannot infer type argument 1 of "add"  [misc]
<cell>2: error: Cannot infer type argument 1 of "add"  [misc]

[[0], {1, 2}]

from typing import TypeVar

IntOrStr = TypeVar("IntOrStr", str, int)


def add(x: List[IntOrStr], y: List[IntOrStr]) -> List[IntOrStr]:
    return x + y


print(add([1, 2], [3]))
print(add(["A"], ["B"]))

[1, 2, 3]
['A', 'B']

print(add([1.0, 2.0], [3.0]))

<cell>1: error: Value of type variable "IntOrStr" of "add" cannot be "float"  [type-var]

[1.0, 2.0, 3.0]

Other tricks

• to have mypy ignore a line, append
  • # type: ignore
• to see what type mypy infers for an object
  • reveal_type(my_obj)
• to override the inferred type
  • cast(str, my_obj)

def add1(x: int) -> int:
    return x + 1


f = 0.1

reveal_type(f)

<cell>7: note: Revealed type is "builtins.float"

# (mis)use of type ignore:
print(add1(f))  # type: ignore

1.1

from typing import cast

# (mis)use of cast:
print(add1(cast(int, f)))

1.1

3rd party libraries

• some libraries include type information and will just work with mypy
• type information for many other libraries is available from typeshed
  • e.g. for the requests library pip install types-requests
  • or run mypy with --install-types to automatically download them as needed
• a few libraries offer a separate stubs package to install instead

If no type information is available, you can add # type: ignore to the end of the line where you import the package to suppress mypy error messages related to this package.

Advanced features

• define your own protocols
  • an example of a built-in protocal is Iterable[T]
• define your own generic classes
  • and example of a similar built-in class is list[X]
• use mypyc to compile type-annoted Python to C extensions
  • similar to Cython but code remains valid python & get run-time type checks
  • note: still alpha and not competitive for numeric code

TLDR

Add type annotations and run mypy to catch bugs earlier and more easily

Strategy

• you don't need to annotate everything
• start with just a few functions in a single file
• mypy infers types where possible
  • e.g. a = [1, 2, 3] is automatically inferred to be a List[int]
• a few annotations can go a long way

More information

• mypy is well documented
• a good starting point: getting started
• basic summary: cheat sheet
• full documentation: mypy.readthedocs.io
• beyond that try github issues: github.com/python/mypy/issues