1 Optional Lab: Python, NumPy, and Vectorization
A brief introduction to some of the scientific computing used in this course. In particular the NumPy scientific computing package and its use with python.
2 goals
In this lab the features of NumPy and Python used in the course will be reviewed.
Python is the programming language used in this course. The NumPy library extends python's basic functionality, adding richer data sets, including more numeric types, vectors, matrices, and many matrix functions. NumPy and python work together fairly seamlessly. Python arithmetic operators work on NumPy data types, and many NumPy functions will accept Python data types.
NumPy's basic data structure is an indexable n-dimensional array containing elements of the same type (dtype). Here dimension refers to the index number of the array. A one-dimensional array has an index. In Lesson 1, we represented vectors as NumPy one-dimensional arrays. One-dimensional array, shape(n,) : n elements indexed from [0] to [n-1].
Data creation routines in NumPy usually have a first argument, which is the shape of the object. This can be a single value for a 1D result, or a tuple (n, m, ...) specifying the shape of the result. Below are examples of vector creation using these routines.
import numpy as np # it is an unofficial standard to use np for numpy
import time
# NumPy routines which allocate memory and fill arrays with value
a = np.zeros(4); print(f"np.zeros(4) : a = {a}, a shape = {a.shape}, a data type = {a.dtype}")
a = np.zeros((4,)); print(f"np.zeros(4,) : a = {a}, a shape = {a.shape}, a data type = {a.dtype}")
a = np.random.random_sample(4); print(f"np.random.random_sample(4): a = {a}, a shape = {a.shape}, a data type = {a.dtype}")The output is:
np.zeros(4) : a = [0. 0. 0. 0.], a shape = (4,), a data type = float64 np.zeros(4,) : a = [0. 0. 0. 0.], a shape = (4,), a data type = float64 np.random.random_sample(4): a = [0.40589302 0.63171453 0.69259702 0.54159911], a shape = (4,), a data type = float64
Some data creation routines do not take tuples:
# NumPy routines which allocate memory and fill arrays with value but do not accept shape as input argument
a = np.arange(4.); print(f"np.arange(4.): a = {a}, a shape = {a.shape}, a data type = {a.dtype}")
a = np.random.rand(4); print(f"np.random.rand(4): a = {a}, a shape = {a.shape}, a data type = {a.dtype}")The output is:
np.arange(4.): a = [0. 1. 2. 3.], a shape = (4,), a data type = float64 np.random.rand(4): a = [0.54170759 0.00065357 0.46959253 0.09870197], a shape = (4,), a data type = float64
Values can also be specified manually:
# NumPy routines which allocate memory and fill with user specified values
a = np.array([5,4,3,2]); print(f"np.array([5,4,3,2]): a = {a}, a shape = {a.shape}, a data type = {a.dtype}")
a = np.array([5.,4,3,2]); print(f"np.array([5.,4,3,2]): a = {a}, a shape = {a.shape}, a data type = {a.dtype}")The output is:
np.array([5,4,3,2]): a = [5 4 3 2], a shape = (4,), a data type = int32 np.array([5.,4,3,2]): a = [5. 4. 3. 2.], a shape = (4,), a data type = float64
These all create a one-dimensional vector a with four elements. a.shape returns the dimension. Here, we see that a.shape = (4,) represents a one-dimensional array with 4 elements.
3 vector operations
3.1 Index
Elements of a vector can be accessed by indexing and slicing. NumPy provides a very complete set of indexing and slicing functions. We will only explore the basics required for the course here. See Slices and Indexes for more details.
Indexing means referring to an element of an array by its position in the array.
Slicing means getting a subset of elements from an array based on an index.
NumPy indexes from zero, so the 3rd element of vector a is a[2].
#vector indexing operations on 1-D vectors
a = np.arange(10)
print(a)
#access an element
print(f"a[2].shape: {a[2].shape} a[2] = {a[2]}, Accessing an element returns a scalar")
# access the last element, negative indexes count from the end
print(f"a[-1] = {a[-1]}")
#indexs must be within the range of the vector or they will produce and error
try:
c = a[10]
except Exception as e:
print("The error message you'll see is:")
print(e)output:
[0 1 2 3 4 5 6 7 8 9] a[2].shape: () a[2] = 2, Accessing an element returns a scalar a[-1] = 9 The error message you'll see is: index 10 is out of bounds for axis 0 with size 10
3.2 Slicing
Slicing creates indexed arrays using a set of three values (start: stop: step). A subset of values is also valid. Its usage can be explained with an example:
#vector slicing operations
a = np.arange(10)
print(f"a = {a}")
#access 5 consecutive elements (start:stop:step)
c = a[2:7:1]; print("a[2:7:1] = ", c)
# access 3 elements separated by two
c = a[2:7:2]; print("a[2:7:2] = ", c)
# access all elements index 3 and above
c = a[3:]; print("a[3:] = ", c)
# access all elements below index 3
c = a[:3]; print("a[:3] = ", c)
# access all elements
c = a[:]; print("a[:] = ", c)output:
a = [0 1 2 3 4 5 6 7 8 9] a[2:7:1] = [2 3 4 5 6] a[2:7:2] = [2 4 6] a[3:] = [3 4 5 6 7 8 9] a[:3] = [0 1 2] a[:] = [0 1 2 3 4 5 6 7 8 9]
3.3 Single vector operation
There are many useful operations involving operations on individual vectors.
a = np.array([1,2,3,4])
print(f"a : {a}")
# negate elements of a
b = -a
print(f"b = -a : {b}")
# sum all elements of a, returns a scalar
b = np.sum(a)
print(f"b = np.sum(a) : {b}")
b = np.mean(a)
print(f"b = np.mean(a): {b}")
b = a**2
print(f"b = a**2 : {b}")output:
a : [1 2 3 4] b = -a : [-1 -2 -3 -4] b = np.sum(a) : 10 b = np.mean(a): 2.5 b = a**2 : [ 1 4 9 16]
3.4 Vector and vector element operations
Most NumPy arithmetic, logical, and comparison operations also work on vectors. These operators work element-by-element on an element-by-element basis.
a = np.array([ 1, 2, 3, 4])
b = np.array([-1,-2, 3, 4])
print(f"Binary operators work element wise: {a + b}")output:
Binary operators work element wise: [0 0 6 8]
Of course, for this to work correctly, the vectors must have the same size:
#try a mismatched vector operation
c = np.array([1, 2])
try:
d = a + c
except Exception as e:
print("The error message you'll see is:")
print(e)output:
The error message you'll see is: operands could not be broadcast together with shapes (4,) (2,)
3.5 Scalar-Vector Operations
Vectors can be "scaled" by a scalar value. A scalar value is just a number. Scalar times all elements of the vector.
a = np.array([1, 2, 3, 4])
# multiply a by a scalar
b = 5 * a
print(f"b = 5 * a : {b}")output:
b = 5 * a : [ 5 10 15 20]
3.6 Vector-vector dot product
Dot products are the backbone of linear algebra and NumPy. This is an operation that is used extensively in this course. Dot product multiplies the values in two vectors element-wise and then sums the results. Vector dot product requires two vectors to be the same size. Note that the dot product should return a scalar value.
def my_dot(a, b):
"""
Compute the dot product of two vectors
Args:
a (ndarray (n,)): input vector
b (ndarray (n,)): input vector with same dimension as a
Returns:
x (scalar):
"""
x=0
for i in range(a.shape[0]):
x = x + a[i] * b[i]
return x
# test 1-D
a = np.array([1, 2, 3, 4])
b = np.array([-1, 4, 3, 2])
print(f"my_dot(a, b) = {my_dot(a, b)}")The output is:
my_dot(a, b) = 24
# test 1-D
a = np.array([1, 2, 3, 4])
b = np.array([-1, 4, 3, 2])
c = np.dot(a, b)
print(f"NumPy 1-D np.dot(a, b) = {c}, np.dot(a, b).shape = {c.shape} ")
c = np.dot(b, a)
print(f"NumPy 1-D np.dot(b, a) = {c}, np.dot(a, b).shape = {c.shape} ")The output is:
NumPy 1-D np.dot(a, b) = 24, np.dot(a, b).shape = () NumPy 1-D np.dot(b, a) = 24, np.dot(a, b).shape = ()
3.7 The Need for Speed: Vectors and Loops
The NumPy library is used because of its increased speed and memory efficiency.
np.random.seed(1)
a = np.random.rand(10000000) # very large arrays
b = np.random.rand(10000000)
tic = time.time() # capture start time
c = np.dot(a, b)
toc = time.time() # capture end time
print(f"np.dot(a, b) = {c:.4f}")
print(f"Vectorized version duration: {1000*(toc-tic):.4f} ms ")
tic = time.time() # capture start time
c = my_dot(a,b)
toc = time.time() # capture end time
print(f"my_dot(a, b) = {c:.4f}")
print(f"loop version duration: {1000*(toc-tic):.4f} ms ")
del(a);del(b) #remove these big arrays from memoryoutput:
np.dot(a, b) = 2501072.5817 Vectorized version duration: 46.8779 ms my_dot(a, b) = 2501072.5817 loop version duration: 4033.1399 ms
So vectorization provides a big speed boost in this case. This is because NumPy makes better use of the data parallelism available in the underlying hardware. GPUs and modern CPUs implement Single Instruction Multiple Data (SIMD) pipelines that allow multiple operations to be issued in parallel. This is critical in machine learning, where datasets are often very large.
4 matrix
matrix, which is a two-dimensional array. The elements of the matrix are all of the same type. In notation, matrices are represented by capital letters, boldface letters such as X. In this lab and others, m is usually the number of rows and columns. Elements of a matrix can be referenced with a two-dimensional index. In mathematical settings, the numbers in the index are usually from 1 to n. In computer science and in these labs, the index will run from 0 to n-1. In general matrix notation, the first index is the row and the second is the column.
NumPy's basic data structure is an indexable n-dimensional array containing elements of the same type (dtype). These are described previously. A matrix has a two-dimensional (2-D) index [m,n]. Below you will review:
- data creation
- Slicing and Indexing
4.1 Matrix creation
The functions for creating two-dimensional vectors are the same as those for creating one-dimensional vectors. Note how NumPy uses square brackets to denote each dimension. Taking it a step further, when printing, it will print one line per line.
a = np.zeros((1, 5))
print(f"a shape = {a.shape}, a = {a}")
a = np.zeros((2, 1))
print(f"a shape = {a.shape}, a = {a}")
a = np.random.random_sample((1, 1))
print(f"a shape = {a.shape}, a = {a}") output:
a shape = (1, 5), a = [[0. 0. 0. 0. 0.]] a shape = (2, 1), a = [[0.] [0.]] a shape = (1, 1), a = [[0.44236513]]
Data can also be specified manually. Dimensions are specified with extra parentheses, matching the format printed above.
# NumPy routines which allocate memory and fill with user specified values
a = np.array([[5], [4], [3]]); print(f" a shape = {a.shape}, np.array: a = {a}")
a = np.array([[5], # One can also
[4], # separate values
[3]]); #into separate rows
print(f" a shape = {a.shape}, np.array: a = {a}")output:
a shape = (3, 1), np.array: a = [[5] [4] [3]] a shape = (3, 1), np.array: a = [[5] [4] [3]]
4.2 Matrix operations
4.2.1 Index
Matrix index description [row, column]. Can return element or row/column. see below:
#vector indexing operations on matrices
a = np.arange(6).reshape(-1, 2) #reshape is a convenient way to create matrices
print(f"a.shape: {a.shape}, \na= {a}")
#access an element
print(f"\na[2,0].shape: {a[2, 0].shape}, a[2,0] = {a[2, 0]}, type(a[2,0]) = {type(a[2, 0])} Accessing an element returns a scalar\n")
#access a row
print(f"a[2].shape: {a[2].shape}, a[2] = {a[2]}, type(a[2]) = {type(a[2])}")output:
a.shape: (3, 2), a= [[0 1] [2 3] [4 5]] a[2,0].shape: (), a[2,0] = 4, type(a[2,0]) = <class 'numpy.int32'> Accessing an element returns a scalar a[2].shape: (2,), a[2] = [4 5], type(a[2]) = <class 'numpy.ndarray'>
One last example is worth noting. Accessing a matrix by specifying rows returns a 1D vector.
Reshape: Use reshape to set the shape of an array.
A = np.arange(6).reshape(- 1,2)
This line of code first creates a 1-D Vector with 6 elements. It then uses the reshape command to reshape this vector into a 2D array. It can be written like this:
A = np.arange(6).reshape(3,2)
to the same 3 row, 2 column array. The -1 argument tells the routine to count the number of rows for a given array size and number of columns.
4.2.2 Slicing
Slicing creates an indexed array using a set of three values (start:stop:step).
#vector 2-D slicing operations
a = np.arange(20).reshape(-1, 10)
print(f"a = \n{a}")
#access 5 consecutive elements (start:stop:step)
print("a[0, 2:7:1] = ", a[0, 2:7:1], ", a[0, 2:7:1].shape =", a[0, 2:7:1].shape, "a 1-D array")
#access 5 consecutive elements (start:stop:step) in two rows
print("a[:, 2:7:1] = \n", a[:, 2:7:1], ", a[:, 2:7:1].shape =", a[:, 2:7:1].shape, "a 2-D array")
# access all elements
print("a[:,:] = \n", a[:,:], ", a[:,:].shape =", a[:,:].shape)
# access all elements in one row (very common usage)
print("a[1,:] = ", a[1,:], ", a[1,:].shape =", a[1,:].shape, "a 1-D array")
# same as
print("a[1] = ", a[1], ", a[1].shape =", a[1].shape, "a 1-D array")
output:
a = [[ 0 1 2 3 4 5 6 7 8 9] [10 11 12 13 14 15 16 17 18 19]] a[0, 2:7:1] = [2 3 4 5 6] , a[0, 2:7:1].shape = (5,) a 1-D array a[:, 2:7:1] = [[ 2 3 4 5 6] [12 13 14 15 16]] , a[:, 2:7:1].shape = (2, 5) a 2-D array a[:,:] = [[ 0 1 2 3 4 5 6 7 8 9] [10 11 12 13 14 15 16 17 18 19]] , a[:,:].shape = (2, 10) a[1,:] = [10 11 12 13 14 15 16 17 18 19] , a[1,:].shape = (10,) a 1-D array a[1] = [10 11 12 13 14 15 16 17 18 19] , a[1].shape = (10,) a 1-D array
In this lab, we master the features of Python and NumPy that are required for the course.