Questionnaire

1. Write the Python code to implement a single neuron.
2. Write the Python code to implement ReLU.
3. Write the Python code for a dense layer in terms of matrix multiplication.
4. Write the Python code for a dense layer in plain Python (that is, with list comprehensions and functionality built into Python).
5. What is the “hidden size” of a layer?
6. What does the t method do in PyTorch?
7. Why is matrix multiplication written in plain Python very slow?
8. In matmul, why is ac==br?
9. In Jupyter Notebook, how do you measure the time taken for a single cell to execute?
10. What is “elementwise arithmetic”?
11. Write the PyTorch code to test whether every element of a is greater than the corresponding element of b.
12. What is a rank-0 tensor? How do you convert it to a plain Python data type?
13. What does this return, and why? tensor([1,2]) + tensor([1])
14. What does this return, and why? tensor([1,2]) + tensor([1,2,3])
15. How does elementwise arithmetic help us speed up matmul?
16. What are the broadcasting rules?
17. What is expand_as? Show an example of how it can be used to match the results of broadcasting.
18. How does unsqueeze help us to solve certain broadcasting problems?
19. How can we use indexing to do the same operation as unsqueeze?
20. How do we show the actual contents of the memory used for a tensor?
21. When adding a vector of size 3 to a matrix of size 3×3, are the elements of the vector added to each row or each column of the matrix? (Be sure to check your answer by running this code in a notebook.)
22. Do broadcasting and expand_as result in increased memory use? Why or why not?
23. Implement matmul using Einstein summation.
24. What does a repeated index letter represent on the left-hand side of einsum?
25. What are the three rules of Einstein summation notation? Why?
26. What are the forward pass and backward pass of a neural network?
27. Why do we need to store some of the activations calculated for intermediate layers in the forward pass?
28. What is the downside of having activations with a standard deviation too far away from 1?
29. How can weight initialization help avoid this problem?
30. What is the formula to initialize weights such that we get a standard deviation of 1 for a plain linear layer, and for a linear layer followed by ReLU?
31. Why do we sometimes have to use the squeeze method in loss functions?
32. What does the argument to the squeeze method do? Why might it be important to include this argument, even though PyTorch does not require it?
33. What is the “chain rule”? Show the equation in either of the two forms presented in this chapter.
34. Show how to calculate the gradients of mse(lin(l2, w2, b2), y) using the chain rule.
35. What is the gradient of ReLU? Show it in math or code. (You shouldn’t need to commit this to memory—try to figure it using your knowledge of the shape of the function.)
36. In what order do we need to call the *_grad functions in the backward pass? Why?
37. What is __call__?
38. What methods must we implement when writing a torch.autograd.Function?
39. Write nn.Linear from scratch, and test it works.
40. What is the difference between nn.Module and fastai’s Module?

Further Research

1. Implement ReLU as a torch.autograd.Function and train a model with it.
2. If you are mathematically inclined, find out what the gradients of a linear layer are in mathematical notation. Map that to the implementation we saw in this chapter.
3. Learn about the unfold method in PyTorch, and use it along with matrix multiplication to implement your own 2D convolution function. Then train a CNN that uses it.
4. Implement everything in this chapter using NumPy instead of PyTorch.

In [ ]: