Lunch Time Python¶
28.10.2022: PyTorch¶
PyTorch is a free and open-source machine learning framework that was originally developed by engineers at Facebook, but is now part of the Linux foundation. The two main features of PyTorch are its tensor computations framework (similar to numpy) with great support for GPU acceleration and their support for neural networks via autograd.
Press Spacebar to go to the next slide (or ? to see all navigation shortcuts)
Lunch Time Python, Scientific Software Center, Heidelberg University
0 Why use PyTorch?¶
Source: Twitter
Source: Assembly AI
In [1]:
# first imports
import torch
from torch import nn # model
from torch import optim # optimizer
from torchvision import datasets, transforms # data and data transforms
from torch.utils.data import random_split, DataLoader # utilities
import numpy as np
import matplotlib.pyplot as plt
1 Tensors¶
In [2]:
# directly from data
data = [[1, 2], [3, 4]]
x_data = torch.tensor(data)
In [3]:
# from numpy array
np_array = np.array(data)
x_np = torch.from_numpy(np_array)
In [4]:
# from another tensor
x_ones = torch.ones_like(x_data) # retains the properties of x_data
print(f"Ones Tensor: \n {x_ones} \n")
x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")
Ones Tensor:
tensor([[1, 1],
[1, 1]])
Random Tensor:
tensor([[0.9294, 0.6675],
[0.7350, 0.6122]])
In [5]:
# use tuples to determine tensor dimensions
shape = (
2,
3,
)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)
print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")
Random Tensor:
tensor([[0.7464, 0.4589, 0.7214],
[0.5188, 0.9056, 0.8306]])
Ones Tensor:
tensor([[1., 1., 1.],
[1., 1., 1.]])
Zeros Tensor:
tensor([[0., 0., 0.],
[0., 0., 0.]])
In [6]:
# tensor attributes
tensor = torch.rand(3, 4)
print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")
Shape of tensor: torch.Size([3, 4]) Datatype of tensor: torch.float32 Device tensor is stored on: cpu
In [7]:
# by default, tensors are created on CPU
# We move our tensor to the GPU if available
if torch.cuda.is_available():
tensor = tensor.to("cuda")
In [8]:
# indexing like numpy
tensor = torch.ones(4, 4)
print(f"First row: {tensor[0]}")
print(f"First column: {tensor[:, 0]}")
print(f"Last column: {tensor[..., -1]}")
tensor[:, 1] = 0
print(tensor)
First row: tensor([1., 1., 1., 1.])
First column: tensor([1., 1., 1., 1.])
Last column: tensor([1., 1., 1., 1.])
tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])
In [9]:
# joining tensors
t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)
tensor([[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.]])
In [10]:
# This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value
y1 = tensor @ tensor.T
y2 = tensor.matmul(tensor.T)
y3 = torch.rand_like(y1)
torch.matmul(tensor, tensor.T, out=y3)
# This computes the element-wise product. z1, z2, z3 will have the same value
z1 = tensor * tensor
z2 = tensor.mul(tensor)
z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)
Out[10]:
tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])
In [11]:
# GPU via CUDA
# torch.randn(5).cuda()
# better (more flexible):
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
torch.randn(5).to(device)
Out[11]:
tensor([-1.1997, -1.9638, -0.4239, -0.2190, 1.5368])
2 Datasets and DataLoaders¶
- datasets: stores the samples and their corresponding labels
- DataLoader: wraps an iterable around the Dataset to enable easy access to the samples
In [12]:
# import and split data
train_data = datasets.MNIST(
"data", train=True, download=True, transform=transforms.ToTensor()
)
train, val = random_split(train_data, [55000, 5000])
train_loader = DataLoader(train, batch_size=32)
val_loader = DataLoader(val, batch_size=32)
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Failed to download (trying next): HTTP Error 403: Forbidden Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz
0%| | 0.00/9.91M [00:00<?, ?B/s]
2%|▏ | 164k/9.91M [00:00<00:09, 1.07MB/s]
7%|▋ | 688k/9.91M [00:00<00:03, 2.66MB/s]
28%|██▊ | 2.75M/9.91M [00:00<00:00, 8.35MB/s]
86%|████████▋ | 8.55M/9.91M [00:00<00:00, 22.0MB/s]
100%|██████████| 9.91M/9.91M [00:00<00:00, 17.7MB/s]
Extracting data/MNIST/raw/train-images-idx3-ubyte.gz to data/MNIST/raw
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Failed to download (trying next): HTTP Error 403: Forbidden Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to data/MNIST/raw/train-labels-idx1-ubyte.gz
0%| | 0.00/28.9k [00:00<?, ?B/s]
100%|██████████| 28.9k/28.9k [00:00<00:00, 410kB/s]
Extracting data/MNIST/raw/train-labels-idx1-ubyte.gz to data/MNIST/raw Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Failed to download (trying next): HTTP Error 403: Forbidden Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to data/MNIST/raw/t10k-images-idx3-ubyte.gz
0%| | 0.00/1.65M [00:00<?, ?B/s]
6%|▌ | 98.3k/1.65M [00:00<00:02, 697kB/s]
20%|█▉ | 328k/1.65M [00:00<00:01, 1.24MB/s]
78%|███████▊ | 1.28M/1.65M [00:00<00:00, 3.73MB/s]
100%|██████████| 1.65M/1.65M [00:00<00:00, 3.86MB/s]
Extracting data/MNIST/raw/t10k-images-idx3-ubyte.gz to data/MNIST/raw Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Failed to download (trying next): HTTP Error 403: Forbidden Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to data/MNIST/raw/t10k-labels-idx1-ubyte.gz
0%| | 0.00/4.54k [00:00<?, ?B/s]
100%|██████████| 4.54k/4.54k [00:00<00:00, 9.01MB/s]
Extracting data/MNIST/raw/t10k-labels-idx1-ubyte.gz to data/MNIST/raw
In [13]:
figure = plt.figure(figsize=(8, 8))
cols, rows = 3, 3
for i in range(1, cols * rows + 1):
sample_idx = torch.randint(len(train_data), size=(1,)).item()
img, label = train_data[sample_idx]
figure.add_subplot(rows, cols, i)
plt.axis("off")
plt.imshow(img.squeeze(), cmap="gray")
plt.show()
In [14]:
train_features, train_labels = next(iter(train_loader))
print(f"Feature batch shape: {train_features.size()}")
print(f"Labels batch shape: {train_labels.size()}")
img = train_features[0].squeeze()
label = train_labels[0]
plt.imshow(img, cmap="gray")
plt.show()
print(f"Label: {label}")
Feature batch shape: torch.Size([32, 1, 28, 28]) Labels batch shape: torch.Size([32])
Label: 0
3 Coding a neural network¶
In [15]:
# in theory easy via stateless approach
# import torch.nn.functional as F
# loss_func = F.cross_entropy
# def model(xb):
# return xb @ weights + bias
# print(loss_func(model(xb), yb), accuracy(model(xb), yb))
# gets messy quickly!
In [16]:
# define model via explicit nn.Module class
class MyModel(nn.Module):
def __init__(self):
super().__init__()
self.l1 = nn.Linear(28 * 28, 64)
self.l2 = nn.Linear(64, 64)
self.l3 = nn.Linear(64, 10)
self.do = nn.Dropout(0.1)
def forward(self, x):
h1 = nn.functional.relu(self.l1(x))
h2 = nn.functional.relu(self.l2(h1))
do = self.do(h2 + h1) # residual connection
logits = self.l3(do)
return logits
nn.Sequential is an ordered container of modules; good for easy and quick networks. No need to specify forward method!
In [17]:
# defining model via sequential
# shorthand, no need for forward method
model_seq = nn.Sequential(
nn.Linear(28 * 28, 64),
nn.ReLU(),
nn.Linear(64, 64),
nn.ReLU(),
nn.Dropout(0.1), # often helps with overfitting
nn.Linear(64, 10),
)
In [18]:
# move model to GPU/device memory
model = model_seq.to(device)
Many layers inside a neural network are parameterized, i.e. have associated weights and biases that are optimized during training. Subclassing nn.Module automatically tracks all fields defined inside your model object, and makes all parameters accessible using your model’s parameters() or named_parameters() methods.
In [19]:
print(f"Model structure: {model}\n\n")
for name, param in model.named_parameters():
print(f"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \n")
Model structure: Sequential(
(0): Linear(in_features=784, out_features=64, bias=True)
(1): ReLU()
(2): Linear(in_features=64, out_features=64, bias=True)
(3): ReLU()
(4): Dropout(p=0.1, inplace=False)
(5): Linear(in_features=64, out_features=10, bias=True)
)
Layer: 0.weight | Size: torch.Size([64, 784]) | Values : tensor([[-0.0346, 0.0330, -0.0311, ..., -0.0105, 0.0243, -0.0031],
[-0.0194, -0.0028, -0.0015, ..., 0.0344, -0.0172, 0.0127]],
grad_fn=<SliceBackward0>)
Layer: 0.bias | Size: torch.Size([64]) | Values : tensor([-0.0199, 0.0211], grad_fn=<SliceBackward0>)
Layer: 2.weight | Size: torch.Size([64, 64]) | Values : tensor([[-8.9403e-02, -1.2363e-01, -1.0378e-01, -6.4146e-02, -4.3081e-02,
2.1292e-02, 9.0755e-02, -4.9322e-02, -9.7042e-02, 6.8305e-02,
1.1680e-01, -1.0534e-01, 2.6677e-02, 6.6649e-02, 8.5539e-02,
-9.7714e-02, 1.3861e-02, -8.6489e-02, -3.5802e-02, -8.3183e-03,
-7.3912e-02, 1.0349e-01, -3.5458e-02, 1.0929e-01, 9.3228e-02,
-6.1126e-02, -7.3705e-02, -7.9222e-03, -1.2282e-02, 6.2558e-02,
-2.9499e-02, -1.0264e-01, 2.5058e-02, -2.4555e-02, 2.8137e-02,
3.1058e-02, 1.0932e-01, -5.7862e-02, -1.0911e-01, -5.9355e-02,
-1.8784e-02, -5.2841e-02, -1.9137e-02, -1.0920e-01, 9.9395e-02,
3.0874e-02, -9.8463e-02, 1.8025e-02, -8.8817e-02, 2.3255e-03,
3.4323e-03, -4.1539e-02, 4.5769e-02, -1.2543e-02, -1.0881e-01,
1.0117e-01, 7.3007e-02, -9.7989e-02, 8.0863e-02, -7.5114e-02,
2.9873e-02, -7.0424e-02, 1.6379e-02, 4.9317e-02],
[ 2.8254e-02, 1.1959e-01, 7.1552e-02, 2.8686e-02, 4.4923e-02,
-6.3941e-02, -8.7384e-02, 1.1375e-01, -6.8182e-03, 4.2997e-02,
7.3560e-02, -9.5101e-02, -7.9280e-03, -4.1439e-02, 6.7057e-02,
1.2924e-02, 9.9922e-03, 1.1180e-01, -1.0916e-01, -1.4380e-02,
2.3544e-05, 1.2128e-01, -1.1260e-01, -7.9696e-02, -1.2162e-01,
-7.9067e-02, 6.0333e-02, -1.2275e-01, -1.4212e-02, 5.1381e-02,
2.7067e-02, -1.9758e-02, -7.7300e-02, -2.3651e-02, 7.6287e-02,
-1.5099e-02, -5.7824e-02, 1.2142e-01, -2.8038e-02, -1.0963e-01,
-8.7499e-02, -1.0240e-01, 1.0987e-01, 6.8833e-02, 8.4547e-02,
-1.0387e-01, -1.2182e-01, -6.0656e-02, 2.4134e-02, -8.0250e-02,
-2.3269e-02, -3.1376e-02, -5.5353e-02, 8.5476e-02, -5.5301e-02,
-7.1915e-02, -1.0772e-01, 6.8427e-02, 1.0836e-01, 1.0171e-01,
-1.1017e-02, -8.8439e-02, -2.6192e-02, 4.9620e-02]],
grad_fn=<SliceBackward0>)
Layer: 2.bias | Size: torch.Size([64]) | Values : tensor([-0.0363, -0.0111], grad_fn=<SliceBackward0>)
Layer: 5.weight | Size: torch.Size([10, 64]) | Values : tensor([[ 0.0178, -0.0887, -0.0183, -0.0322, -0.0325, 0.0151, -0.1226, -0.0686,
-0.0157, 0.1140, 0.0180, 0.0164, -0.1135, -0.0921, 0.0097, -0.0379,
0.0010, -0.0749, -0.0476, -0.0768, -0.0122, -0.0812, -0.1151, -0.0476,
0.0311, 0.1100, 0.1021, -0.0354, -0.0041, 0.0650, -0.0531, 0.0807,
-0.0588, -0.0741, 0.0898, -0.1195, -0.0614, 0.0211, -0.1133, 0.0517,
-0.1094, -0.0272, 0.0399, 0.0264, -0.0665, -0.0848, 0.0157, -0.0060,
-0.0101, 0.0229, 0.0047, -0.0330, -0.0260, -0.0848, 0.0611, -0.0549,
-0.0028, -0.0231, 0.0211, 0.0603, -0.0803, -0.0477, 0.0880, 0.0154],
[ 0.1080, 0.1049, 0.0104, 0.0734, -0.1207, -0.0934, -0.0202, 0.1071,
-0.0151, -0.1248, -0.0303, 0.0364, -0.0543, 0.0772, 0.1072, -0.0778,
-0.0826, -0.0901, -0.0988, 0.0186, -0.1230, 0.0990, 0.1123, 0.0410,
-0.0086, 0.0315, -0.0751, -0.0638, 0.0475, 0.0670, -0.0943, 0.1142,
0.0130, -0.0907, -0.0178, -0.0954, 0.0497, -0.0401, 0.0116, -0.1096,
-0.1047, -0.1128, -0.0911, -0.0534, -0.0463, 0.0181, 0.0444, 0.0449,
-0.0685, 0.0756, -0.1140, 0.1059, -0.0439, -0.0156, 0.0833, 0.0647,
-0.0640, -0.0626, 0.0603, 0.0953, -0.0812, 0.0682, 0.0903, 0.1234]],
grad_fn=<SliceBackward0>)
Layer: 5.bias | Size: torch.Size([10]) | Values : tensor([ 0.1246, -0.1217], grad_fn=<SliceBackward0>)
In [20]:
# define loss function
loss = nn.CrossEntropyLoss() # softmax + neg. log
4 Backpropagation via Autograd¶
In a forward pass, autograd does two things simultaneously:
run the requested operation to compute a resulting tensor
maintain the operation’s gradient function in the DAG.
The backward pass kicks off when .backward() is called on the DAG root. autograd then:
computes the gradients from each .grad_fn,
accumulates them in the respective tensor’s .grad attribute
using the chain rule, propagates all the way to the leaf tensors.
5 Optimization of model parameters (training)¶
We define the following hyperparameters for training:
- Number of Epochs - the number times to iterate over the dataset
- Batch Size - the number of data samples propagated through the network before the parameters are updated (defined in train_Loader)
- Learning Rate - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable behavior during training.
In [21]:
lr = 1e-2
epochs = 5
In [22]:
# defining optimizer
params = model.parameters()
optimiser = optim.SGD(params, lr=1e-2)
Inside the training loop, optimization happens in three steps:
- Call optimizer.zero_grad() to reset the gradients of model parameters. Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.
- Backpropagate the prediction loss with a call to loss.backward(). PyTorch deposits the gradients of the loss w.r.t. each parameter.
- Once we have our gradients, we call optimizer.step() to adjust the parameters by the gradients collected in the backward pass.
In [23]:
# define training and validation loop
# training loop
for epoch in range(epochs):
losses = list()
accuracies = list()
model.train() # enables dropout/batchnorm
for batch in train_loader:
x, y = batch
batch_size = x.size(0)
# x: b x 1 x 28 x 28
x = x.view(batch_size, -1).to(device)
# 5 steps to train network
# 1 forward
l = model(x) # l: logits
# 2 compute objective function
J = loss(l, y.to(device))
# 3 cleaning the gradients (could also call this on optimiser)
model.zero_grad()
# optimizer.zero_grad() is equivalent
# manually: params.grad._zero()
# 4 accumulate the partial derivatives of J wrt params
J.backward()
# manually: params.grad.add_(dJ/dparams)
# 5 step in the opposite direction of the gradient
optimiser.step()
# could have done manual gradient update:
# with torch.no_grad():
# params = params - lr * params.grad
losses.append(J.item())
accuracies.append(y.eq(l.detach().argmax(dim=1).cpu()).float().mean())
print(f"epoch {epoch + 1}", end=", ")
print(f"training loss: {torch.tensor(losses).mean():.2f}", end=", ")
print(
f"training accuracy: {torch.tensor(accuracies).mean():.2f}"
) # print two decimals
# validation loop
losses = list()
accuracies = list()
model.eval() # disables dropout/batchnorm
for batch in val_loader:
x, y = batch
batch_size = x.size(0)
# x: b x 1 x 28 x 28
x = x.view(batch_size, -1).to(device)
# 5 steps to train network
# 1 forward
with torch.no_grad(): # more efficient, just tensor no graph connected
l = model(x) # l: logits
# 2 compute objective function
J = loss(l, y.to(device))
losses.append(J.item())
accuracies.append(y.eq(l.detach().argmax(dim=1).cpu()).float().mean())
print(f"epoch {epoch + 1}", end=", ")
print(f"validation loss: {torch.tensor(losses).mean():.2f}", end=", ")
print(
f"validation accuracy: {torch.tensor(accuracies).mean():.2f}"
) # print two decimals
epoch 1, training loss: 1.27, training accuracy: 0.65
epoch 1, validation loss: 0.48, validation accuracy: 0.86
epoch 2, training loss: 0.44, training accuracy: 0.87
epoch 2, validation loss: 0.35, validation accuracy: 0.90
epoch 3, training loss: 0.35, training accuracy: 0.90
epoch 3, validation loss: 0.30, validation accuracy: 0.91
epoch 4, training loss: 0.31, training accuracy: 0.91
epoch 4, validation loss: 0.27, validation accuracy: 0.92
epoch 5, training loss: 0.27, training accuracy: 0.92
epoch 5, validation loss: 0.24, validation accuracy: 0.93
6 Store models¶
In [24]:
# just save model weights without structure
torch.save(model.state_dict(), "model_weights.pth")
model.load_state_dict(torch.load("model_weights.pth"))
model.eval()
/tmp/ipykernel_1997/2099193612.py:3: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model.load_state_dict(torch.load("model_weights.pth"))
Out[24]:
Sequential( (0): Linear(in_features=784, out_features=64, bias=True) (1): ReLU() (2): Linear(in_features=64, out_features=64, bias=True) (3): ReLU() (4): Dropout(p=0.1, inplace=False) (5): Linear(in_features=64, out_features=10, bias=True) )
In [25]:
# save whole model
torch.save(model, "model.pth")
new_model = torch.load("model.pth")
/tmp/ipykernel_1997/2474383404.py:3: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
new_model = torch.load("model.pth")