{
"cells": [
{
"cell_type": "markdown",
"id": "0",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Lunch Time Python\n",
"\n",
"## Lunch 9: mypy\n",
"\n",
"\n",
"[mypy](https://mypy.readthedocs.io/en/stable/) is a static type checker for Python.\n",
"By adding type annotations to your code mypy can find a variety of bugs.\n",
"These type annotations also act as machine-checked documentation of your code,\n",
"and your IDE can make use of them to improve its code completion.\n",
"They doesn't affect how your program runs, as the Python interpreter ignores\n",
"these type annotations at run-time\n",
"\n",
"*Press `Spacebar` to go to the next slide (or `?` to see all navigation shortcuts)*\n",
"\n",
"[Lunch Time Python](https://ssciwr.github.io/lunch-time-python/), [Scientific Software Center](https://ssc.iwr.uni-heidelberg.de), [Heidelberg University](https://www.uni-heidelberg.de/)"
]
},
{
"cell_type": "markdown",
"id": "1",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Motivation\n",
"\n",
"- Python is a **dynamically typed** language\n",
" - It infers the type of objects automatically\n",
" - \"Duck typing\" - if something looks like a duck and quacks like a duck, it's a duck\n",
"- This makes Python very flexible and convenient compared to a **statically typed** language like C\n",
" - But this flexibility also leaves room for bugs, makes static analysis difficult\n",
" - Harder to maintain and understand large complicated projects without type safety\n",
"- Solution: add optional **type annotations** (or hints) to your code\n",
" - mypy can use these to check the code for bugs\n",
" - they are ignored by the Python interpreter at run-time"
]
},
{
"cell_type": "markdown",
"id": "2",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"# How does this help?\n",
"\n",
"Imagine you call a function with the wrong type of object\n",
"\n",
"- This may be a runtime error, e.g. `str` + `int`\n",
" - This is a good bug: the program stops at the place where the bug is!\n",
" - Assuming your test suite executes this line, you find it by running the tests\n",
" - But even better would be to have your IDE point this out as you write the code\n",
"- This may not be an error, just do something undesirable\n",
" - Eventually this may cause a runtime error or incorrect output\n",
" - Hopefully some test fails, but could be very far away from the original bug\n",
" - This is a bad bug: can be hard to trace back to the root cause\n",
" - If your IDE points out the root cause as you type it this is a big win"
]
},
{
"cell_type": "markdown",
"id": "3",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"id": "4",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"id": "5",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"id": "6",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"id": "7",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# mypy installation\n",
"\n",
"- Conda: `conda install mypy`\n",
"- Pip: `python -m pip install mypy`\n",
"\n",
"# mypy use\n",
"\n",
"- `mypy my_file_to_check.py`"
]
},
{
"cell_type": "markdown",
"id": "8",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### jupyter notebook use\n",
"\n",
"These slides also use the [nb-mypy](https://pypi.org/project/nb-mypy/) extension:\n",
"\n",
"- `python -m pip install nb-mypy`\n",
"\n",
"This automatically runs mypy on every cell before it is executed.\n",
"\n",
"This is just for convenience to show the mypy output for this talk.\n",
"\n",
"In general I would recommend running mypy separately or as a pre-commit hook."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"if \"google.colab\" in str(get_ipython()):\n",
" !pip install nb-mypy -qqq\n",
"%load_ext nb_mypy"
]
},
{
"cell_type": "markdown",
"id": "10",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"# Hello world"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"def greet(thing):\n",
" return f\"Hello {thing}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "12",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(greet(\"world\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(greet(True))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(greet(greet))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def greet(thing: str) -> str:\n",
" return f\"Hello {thing}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "16",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(greet(\"world\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "17",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(greet(True))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "18",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(greet(greet))"
]
},
{
"cell_type": "markdown",
"id": "19",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Basic types\n",
"\n",
"- to annotate an object, add its type after a `:`\n",
" - `my_string: str`\n",
"- to annotate the return type of a function, add the type after `->`\n",
" - `def hello() -> str:`\n",
"- intrinsic types like `None`, `int`, `float`, `str`, `bool` can be used directly\n",
"- other types like `list`, `dict`, `tuple`, `set`\n",
" - import a capitalized version from typing: `from typing import List`\n",
" - use `list` directly (only with Python >= 3.9)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "20",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def mul(a, b):\n",
" return a * b\n",
"\n",
"\n",
"print(mul(2, 2))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(mul(2, [0]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "22",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(mul(2, \"really?\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "23",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def mul(a: float, b: float) -> float:\n",
" return a * b\n",
"\n",
"\n",
"print(mul(2, 2))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "24",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(mul(2, [0]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "25",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(mul(2, \"really?\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "26",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# does operation & returns result\n",
"def append1(l):\n",
" return l + [1]\n",
"\n",
"\n",
"# does operation in place\n",
"def append2(l):\n",
" l.append(1)\n",
"\n",
"\n",
"l0 = [0]\n",
"print(append1(l0))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "27",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(append2(l0))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import List\n",
"\n",
"\n",
"def append1(l: List[int]) -> List[int]:\n",
" return l + [1]\n",
"\n",
"\n",
"def append2(l: List[int]) -> None:\n",
" l.append(1)\n",
"\n",
"\n",
"l0 = [0]\n",
"print(append1(l0))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "29",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(append2(l0))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "30",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def count(l: List[str]):\n",
" return len(l)\n",
"\n",
"\n",
"a = []\n",
"# a: List[str] = []\n",
"# a.append(\"ok\")\n",
"# a = [\"ok\"]\n",
"print(count(a))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import List, Tuple\n",
"\n",
"\n",
"def count(coords: List[Tuple[float, float]]) -> int:\n",
" return len(coords)\n",
"\n",
"\n",
"print(count([(0, 0), (1, 1), (2, 2)]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import Dict\n",
"\n",
"\n",
"def invert(ages: Dict[str, int]) -> Dict[int, str]:\n",
" return {key: value for value, key in ages.items()}\n",
"\n",
"\n",
"ages = {\"bob\": 2, \"joe\": 7}\n",
"names = invert(ages)\n",
"print(names)"
]
},
{
"cell_type": "markdown",
"id": "33",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Generic collections\n",
"\n",
"- if you can do \"for\" to iterate over the object\n",
" - `my_obj: Iterable`\n",
"- if it's like a list\n",
" - `my_obj: Sequence`\n",
"- if it's like a read-only dict\n",
" - `my_obj: Mapping`\n",
"- if it's a dict we can modify\n",
" - `my_obj: MutableMapping`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "34",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import Iterable\n",
"\n",
"\n",
"def count(items: Iterable):\n",
" i = 0\n",
" for item in items:\n",
" i += 1\n",
" return i\n",
"\n",
"\n",
"print(count([\"a\", \"b\", \"c\"]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "35",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import Sequence\n",
"\n",
"\n",
"def last(items: Sequence):\n",
" return items[-1]\n",
"\n",
"\n",
"print(last([1, 2, 3]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"print(count([\"a\", \"b\", \"c\"]))\n",
"from typing import Dict, Mapping\n",
"\n",
"\n",
"def invert(ages: Mapping[str, int]) -> Dict[int, str]:\n",
" # ages[\"simon\"] = 12\n",
" return {key: value for value, key in ages.items()}\n",
"\n",
"\n",
"ages = {\"bob\": 2, \"joe\": 7}\n",
"names = invert(ages)\n",
"print(names)"
]
},
{
"cell_type": "markdown",
"id": "37",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"\n",
"# Flexible types\n",
"\n",
"- if an object could have several types, use typing.Union\n",
" - `my_obj: Union[str, float]`\n",
"- if an object can be either a dict or None\n",
" - `my_obj: Union[Dict, None]`\n",
" - `my_obj: Optional[Dict]`\n",
"- if an object can be anything\n",
" - `my_obj: Any`\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "38",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def hi(name=None):\n",
" if name is None:\n",
" name = \"you\"\n",
" print(f\"hello {name}\")\n",
"\n",
"\n",
"hi()\n",
"hi(\"joe\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import Union\n",
"\n",
"\n",
"def hi(name: Union[str, None] = None) -> None:\n",
" if name is None:\n",
" name = \"you\"\n",
" print(f\"hello {name}\")\n",
"\n",
"\n",
"hi()\n",
"hi(\"joe\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "40",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import Optional\n",
"\n",
"\n",
"def hi(name: Optional[str] = None) -> None:\n",
" if name is None:\n",
" name = \"you\"\n",
" print(f\"hello {name}\")\n",
"\n",
"\n",
"hi()\n",
"hi(\"joe\")"
]
},
{
"cell_type": "markdown",
"id": "41",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"\n",
"# Generics\n",
"\n",
"- TypeVar specifies a generic type\n",
" - Possible types can optionally be constrained\n",
" - Within a scope it represents a single type\n",
" - For any c++ programmers it's a bit like a `template`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "42",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# concatenation of lists of a single type\n",
"def add(x, y):\n",
" return x + y\n",
"\n",
"\n",
"# desired use\n",
"print(add([1, 2], [3]))\n",
"print(add([\"A\"], [\"B\"]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# don't want to allow e.g.\n",
"print(add([1.0], [\"B\"])) # this shouldn't be allowed\n",
"print(add([[0]], [{1, 2}])) # this shouldn't be allowed"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "44",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import TypeVar, List\n",
"\n",
"T = TypeVar(\"T\")\n",
"\n",
"\n",
"def add(x: List[T], y: List[T]) -> List[T]:\n",
" return x + y\n",
"\n",
"\n",
"print(add([1, 2], [3]))\n",
"print(add([\"A\"], [\"B\"]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "45",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"add([1.0], [\"B\"]) # this shouldn't be allowed\n",
"add([[0]], [{1, 2}]) # this shouldn't be allowed"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "46",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from typing import TypeVar\n",
"\n",
"IntOrStr = TypeVar(\"IntOrStr\", str, int)\n",
"\n",
"\n",
"def add(x: List[IntOrStr], y: List[IntOrStr]) -> List[IntOrStr]:\n",
" return x + y\n",
"\n",
"\n",
"print(add([1, 2], [3]))\n",
"print(add([\"A\"], [\"B\"]))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"print(add([1.0, 2.0], [3.0]))"
]
},
{
"cell_type": "markdown",
"id": "48",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Other tricks\n",
"\n",
"- to have mypy ignore a line, append\n",
" - `# type: ignore`\n",
"- to see what type mypy infers for an object\n",
" - `reveal_type(my_obj)`\n",
"- to override the inferred type\n",
" - `cast(str, my_obj)`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def add1(x: int) -> int:\n",
" return x + 1\n",
"\n",
"\n",
"f = 0.1\n",
"\n",
"reveal_type(f)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "50",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# (mis)use of type ignore:\n",
"print(add1(f)) # type: ignore"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "51",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"from typing import cast\n",
"\n",
"# (mis)use of cast:\n",
"print(add1(cast(int, f)))"
]
},
{
"cell_type": "markdown",
"id": "52",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 3rd party libraries\n",
"\n",
"- some libraries include type information and will just work with mypy\n",
"- type information for many other libraries is available from [typeshed](https://github.com/python/typeshed)\n",
" - e.g. for the requests library `pip install types-requests`\n",
" - or run mypy with `--install-types` to automatically download them as needed\n",
"- a few libraries offer a separate stubs package to install instead\n",
"\n",
"If no type information is available, you can add `# type: ignore` to the end of the line where you import the package to suppress mypy error messages related to this package."
]
},
{
"cell_type": "markdown",
"id": "53",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"# Advanced features\n",
"\n",
"- define your own [protocols](https://mypy.readthedocs.io/en/stable/protocols.html)\n",
" - an example of a built-in protocal is `Iterable[T]`\n",
"- define your own [generic classes](https://mypy.readthedocs.io/en/stable/generics.html)\n",
" - and example of a similar built-in class is `list[X]`\n",
"- use [mypyc](https://mypyc.readthedocs.io/) to compile type-annoted Python to C extensions\n",
" - similar to Cython but code remains valid python & get run-time type checks\n",
" - note: still alpha and not competitive for numeric code"
]
},
{
"cell_type": "markdown",
"id": "54",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# TLDR\n",
"\n",
"Add type annotations and run mypy to catch bugs earlier and more easily\n",
"\n",
"# Strategy\n",
"\n",
"- you don't need to annotate everything\n",
"- start with just a few functions in a single file\n",
"- mypy infers types where possible\n",
" - e.g. `a = [1, 2, 3]` is automatically inferred to be a `List[int]`\n",
"- a few annotations can go a long way"
]
},
{
"cell_type": "markdown",
"id": "55",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"# More information\n",
"\n",
"- mypy is well documented\n",
"- a good starting point: [getting started](https://mypy.readthedocs.io/en/stable/getting_started.html)\n",
"- basic summary: [cheat sheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html)\n",
"- full documentation: [mypy.readthedocs.io](https://mypy.readthedocs.io/)\n",
"- beyond that try github issues: [github.com/python/mypy/issues](https://github.com/python/mypy/issues)\n"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}