Introduction to using NumPy module

Introduction to using NumPy

1. Introduction to NumPy scientific computing library and environment preparation

​ NumPy (Numerical Python) is an open source numerical computing extension for Python. Provides multi-dimensional array objects and various derived objects (such as masked arrays and matrices). This tool can be used to store and process large matrices more efficiently than Python's own nested list structure. (This structure can also be used to represent a matrix), supports a large number of dimensional array and matrix operations, and also provides a large number of mathematical function libraries for array operations, including mathematics, logic, shape operations, sorting, selection, Input and output, discrete Fourier transform, basic linear algebra, basic statistical operations and stochastic simulation, etc.

• Almost all data analysts working in Python take advantage of the power of NumPy

  • Powerful N-dimensional array
  • Mature broadcast function
  • Toolkit for integrating C/C++ and Fortran code
  • NumPy provides comprehensive mathematical functions, random number generators and linear algebra functions

• Install Python libraries

  • pip install jupyter -i https://pypi.tuna.tsinghua.edu.cn/simple
    

  • pip install numpy -i https://pypi.tuna.tsinghua.edu.cn/simple
    

  • Jupyter Notebook is a development tool and web-based application for interactive computing. It can be applied to the entire computing process: development, document writing, running code and displaying results, which can often greatly increase the efficiency of debugging code.

• Start command line terminal

  • Windows----> Shortcut key: win + R ----->Input: cmd return ------>Command line comes out
  • Mac ---->Start Terminal

• Start jupyter in the command line terminal

  • Enter the terminal and enter the command:jupyter notebook

  • [Note]: Where to start jupyter, the directory on the browser, corresponding to

There are a few things to note after starting:

1. An html connection file will be generated, which can be opened directly to enter the jupyter operation interface (C:\Users\Administrator\AppData\Roaming\jupyter\runtime\nbserver-14956-open.html)

2. Or copy the provided connection to access, for example: http://localhost:8888/?token=3566fc51fefd4fb9528c5a131d7f0beb7cc8c171639fff14

3. Token is the key information for user verification

4. If only IP plus port 8888 is opened, enter token verification in Password or token: and click login to log in.

5. After logging in, you can create a directory, create a development code file, and develop a code interactive interface in the New option.

First experience with jupyter:

The most commonly used shortcut keys in jupyter

Shift + tab function usage introduction
tab key, code completion
alt + enter, run the current cell, and insert a cell below
ctrl + enter, run the current cell, the cursor is still in the current cell esc,
then b key, insert below, a Press the esc key to insert the cell above
, then press the d key twice to delete the current cell.

There is a complete list of shortcut keys in the help

Compare pycharm:

Functional code can eventually realize the function. Jupyter can be deployed relatively quickly, and OutPut output can be obtained by referencing variables. In some places, it is more convenient to use pycharm for code display, so it will be used in the future.

2.Basic operations of NumPy

2-1. Create an array

The simplest way to create an array is to use the array function to convert the list under Python into an ndarray

For example, the code above

import numpy as np

blizzard_game = ["魔兽世界", "守望先锋", "炉石传说", "魔兽争霸", "暗黑破坏神", "星际争霸", "风暴英雄"]
arr = np.array(blizzard_game)
arr
blizzard_game

numpy.array() converts a list into a NumPy array

Judging from the output content, the data is the same

However, with subsequent use, you will gradually understand the same content. The NumPy array method is more powerful and can achieve more powerful functions than list.

[Note]: np and numpy later refer to the numpy library module, but there is no difference in the alias.

We can use some built-in functions in np to create arrays. For example, we can create an array of all 0s, an array of all 1s, an array of other numbers, an arithmetic sequence array, a normal distribution array, random number.

• Create an array of n 0s or n 1s

import numpy as np

arr1 = np.zeros(5)
arr2 = np.ones(5)
print(arr1)
print(arr2)

""" 输出
[0. 0. 0. 0. 0.]
[1. 1. 1. 1. 1.]
"""

• Create an array with a certain value x in row y and column y

import numpy as np

arr = np.full(shape=[2, 4], fill_value="暴雪凉凉了")
print(arr)

""" 输出
[['暴雪凉凉了' '暴雪凉凉了' '暴雪凉凉了' '暴雪凉凉了']
 ['暴雪凉凉了' '暴雪凉凉了' '暴雪凉凉了' '暴雪凉凉了']]
"""

shape=[2, 4] row, column

fill_value creates content

• Create an arithmetic sequence array

import numpy as np

arr = np.arange(start=0, stop=30, step=5)
print(arr)

""" 输出
[ 0  5 10 15 20 25]
"""

• Create an array of int random integers

import numpy as np

arr = np.random.randint(1, 7, size=6)
print(arr)

""" 输出
[5 5 5 1 3 3]
"""

The three parameters passed in correspond to

low minimum interval, including low value

high maximum interval, excluding high value

size takes several values

• Create normal distribution and array of random numbers

import numpy as np

arr1 = np.random.randn(6)
arr2 = np.random.random(size=6)
print(arr1)
print(arr2)

""" 输出
[-1.31009666  1.26289427 -0.90216964 -0.52891451 -0.02613398  2.99314766]
[0.5897261  0.53644782 0.54970934 0.60330606 0.02733836 0.80834182]
"""

Using the random.random function will return a random value between [0.0,1.0), zise defines the output number;

The random.randn function returns a standard normal distribution value, which is theoretically (negative infinity, positive infinity). In fact, it is hovering near the value 0

2-2. View operations

​NumPy’s array class is called ndarray, also known as the alias array. Note that numpy.array is different from the standard Python library class array.array, which only handles one-dimensional arrays and has fewer features

• jupyter extension

pip install jupyter_contrib_nbextensions -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install jupyter_nbextensions_configurator -i https://pypi.tuna.tsinghua.edu.cn/simple
jupyter contrib nbextension install --user
jupyter nbextensions_configurator enable --user

• jupyter_contrib_nbextensions
  • This package can extend jupyter notebook and enrich its functions, such as folding function codes. For longer notebooks, it can also be used to add a left directory bar for easy positioning.
• JupyterNbExtensions Configurator
  • It is an extension tool for Jupyter Notebook. You only need to check the corresponding plug-in to automatically load it. Can further improve your work efficiency on Jupyter Notebook

In this way, the newly started jupyter book will have a new menu paging Nbextensions

It can be understood that some other tools such as pycharm and other better functions are implemented through extension plug-ins. The extension tools do not need to be installed.

• View the number of axes and dimensions of the array

import numpy as np

arr = np.random.randint(0, 100, size=(3, 4, 5))
print(arr.ndim)

""" 输出
3
"""

• View array size shape

import numpy as np

arr = np.random.randint(0, 100, size=(3, 4, 5))
print(arr.shape)

""" 输出
(3, 4, 5)
"""

3 groups, 4 rows and 5 columns

• Check the total number of array elements

import numpy as np

arr1 = np.random.randint(0, 100, size=(3, 4, 5))
arr2 = np.random.randint(0, 100, size=(1, 2, 6))
print(arr1.size)
print(arr2.size)

""" 输出
60
12
"""

The total number of array elements is equal to the product of the dimensions

60 = 3 * 4 * 5

12 = 1 *2 * 6

• View the data type of an array

import numpy as np

arr = np.random.randint(0, 100, size=(3, 4, 5))
print(arr.dtype)

""" 输出
int32
"""

• Check the size of each element in an array

import numpy as np

arr = np.random.randint(0, 100, size=(3, 4, 5))
print(arr.itemsize)

""" 输出
4
"""

The data type is int32, one byte is 8 bits, and division results in 4

2-3 File IO operations

• save array

The save method saves ndarray to an npy file, or you can use savez to save multiple arrays into a .npz file.

import numpy as np

x = np.random.randn(3)
y = np.arange(0, 5, 1)
np.save("./wangt/xxx_arr", x)
np.savez("./wangt/some_array.npz", xarr=x, yarr=y)

Created via jupyter:

• Read array

​ Use the load method to read the stored array. If it is a .npz file, it is equivalent to forming a key-value type variable after reading. The corresponding array can be obtained through the key defined when saving.

import numpy as np

np.load('./wangting/xxx_arr.npy')
print(np.load('./wangting/some_array.npz')['yarr'])

""" 输出
[0 1 2 3 4]
"""

• Read and write csv, txt files

import numpy as np

arr = np.random.randint(0, 10, size=10)
np.savetxt("./wangt/arr_20221203.csv", arr, delimiter=',')
np.savetxt("./wangt/arr_20221203.txt", arr, delimiter=',')

np.loadtxt("./wangt/arr_20221203.csv", delimiter=',', dtype=np.int32)
np.loadtxt("./wangt/arr_20221203.txt", delimiter=',', dtype=np.int32)
print(np.loadtxt("./wangt/arr_20221203.csv", delimiter=',', dtype=np.int32))
print("-----")
print(np.loadtxt("./wangt/arr_20221203.txt", delimiter=',', dtype=np.int32))

""" 输出
[3 5 0 2 4 5 2 7 0 6]
-----
[3 5 0 2 4 5 2 7 0 6]
"""

3. Introduction to NumPy data types

Data type of ndarray:

• int

  • you8
  • uint8
  • int16
  • int32
  • int64

• float

  • float16
  • float32
  • float64

• str

Can be specified at creation time:

import numpy as np
np.array([1,2,3],dtype = 'float32')

Can be specified when converting asarray:

import numpy as np
arr = [1,2,3,4]
np.asarray(arr,dtype = 'float32')

Can be converted during data type conversion astype:

import numpy as np
arr = np.random.randint(0,10,size = 5,dtype = 'int16')
arr.astype('float32') 

4.NumPy array operations

• logic operation

import numpy as np

arr1 = np.array([1, 2, 3, 4, 5])
arr2 = np.array([1, 0, 2, 3, 5])
print(arr1 < 5)
print(arr1 >= 5)
print(arr1 == 5)
print(arr1 == arr2)
print(arr1 > arr2)

""" 输出
[ True  True  True  True False]
[False False False False  True]
[False False False False  True]
[ True False False False  True]
[False  True  True  True False]
"""

Logical comparison of elements in array arr one by one

• Array and scalar calculations

import numpy as np

arr = np.arange(1, 4)
print(1 / arr)
print(arr + 5)
print(arr * 5)

""" 输出
[1.         0.5        0.33333333]
[6 7 8]
[ 5 10 15]
"""

• *=, +=, -= operations

import numpy as np

arr = np.arange(5)
print(arr)
arr += 2
print(arr)
arr -= 2
print(arr)
arr *= 2
print(arr)

""" 输出
[0 1 2 3 4]
[2 3 4 5 6]
[0 1 2 3 4]
[0 2 4 6 8]
"""

5.NumPy copy and view

When manipulating an array, sometimes its data is copied into a new array, sometimes not

• No copy

In jupyter, use display() to output multiple contents at the same time.

• View, shallow copy

Different array objects can share the same data. The view method creates a new array object that views the same data

• deep copy

6.NumPy indexing, slicing and iteration

• Basic indexing and slicing

import numpy as np

a = np.random.randint(0, 20, size=10)
print(a)  # [10 13  4  1  5]
# 取一个值
print(a[0])
# 取多个值
print(a[[0, 1, 3]])
# 切片
print(a[1:3])  # 从索引1切到索引为3(包括前索引不包括后索引)
print(a[:3])  # 从开始位置切到索引3位置
print(a[3:])  # 从索引为3开始切到末尾
print(a[::3])  # 取全部数据，步长为3
print(a[::-1])  # 相当于反转数据

""" 输出
[14  1 14  7  4 10  4 15  6  7]
14
[14  1  7]
[ 1 14]
[14  1 14]
[ 7  4 10  4 15  6  7]
[14  7  4  7]
[ 7  6 15  4 10  4  7 14  1 14]
"""

• Multidimensional array indexing and slicing

​ For two-dimensional arrays or high-dimensional arrays, we can index according to previous knowledge. Of course, we can also pass in a comma-separated index list to select single or multiple elements.

• slice value

import numpy as np

a = np.arange(20)
print(a)
# [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]

b = a[3:7]
print(b)
# [3 4 5 6]

b[0] = 33333
print(a)
# [ 0  1  2  33333  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]
print(b)
# [33333  4  5  6]

When slicing, from the comparison of the returned data, slicing is not a deep copy, but a shallow copy.

• fancy index

import numpy as np

a = np.arange(20)
# 花式索引
b = a[[3, 4, 5, 6]]
print(a)
# [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]
print(b)
# [3 4 5 6]

b[0] = 33333
print(a)
# [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]
print(b)
# [33333     4     5     6]

When using fancy index, from the comparison of the returned data, the fancy index returns a deep copy and no longer affects the original value.

• Common indexing techniques

import numpy as np

a = np.random.randint(40, 100, size=(12, 3))
print(a)
"""
[[80 61 51]
 [88 98 66]
 [41 81 86]
 [45 50 82]
 [44 98 76]
 [51 56 93]
 [96 61 50]
 [77 77 46]
 [50 40 94]
 [97 48 45]
 [64 85 94]
 [81 82 80]]
"""
# 找出全部数值中大于等于60的值
cond = a >= 60
print(a[cond])
# [80 61 88 98 66 81 86 82 98 76 93 96 61 77 77 94 97 64 85 94 81 82 80]

# boolean True=1;False=0
print(cond[:])
"""
[[ True  True False]
 [ True  True  True]
 [False  True  True]
 [False False  True]
 [False  True  True]
 [False False  True]
 [ True  True False]
 [ True  True False]
 [False False  True]
 [ True False False]
 [ True  True  True]
 [ True  True  True]]
"""

# 找出每行3列数字都大于60的信息
# 相当于用cond每行的True和False去逐个相乘，如果都为True，赋值给cond_new，从而取出每行中元素都大于60的行
cond_new = cond[:, 0] * cond[:, 1] * cond[:, 2]
print(a[cond_new])
"""
[[88 98 66]
 [64 85 94]
 [81 82 80]]
"""

7.NumPy shape operations

• Array transformation - reshape

• Array transpose (column swap) .T

• Array stacking and merging

import numpy as np

a = np.random.randint(0, 10, size=(2, 3))
b = np.random.randint(0, 10, size=(2, 3))
print(a)
# [[0 5 0]
# [7 6 0]]
print(b)
# [[1 8 2]
# [5 1 8]]

# 水平方向-横向行叠加
# 方法1
c = np.concatenate([a, b], axis=0)
print(c)
# 方法2
print(np.vstack((a, b)))
"""
[[0 5 0]
 [7 6 0]
 [1 8 2]
 [5 1 8]]
"""

# 竖直方向-纵向列叠加
# 方法1
d = np.concatenate([a, b], axis=1)
print(d)
# 方法2
print(np.hstack((a, b)))
"""
[[0 5 0 1 8 2]
 [7 6 0 5 1 8]]
"""

• split array split

import numpy as np

a = np.random.randint(0, 15, size=(6, 6))
print(a)
"""
[[ 2  9  7  0  7 13]
 [ 2  9 10  8  6  7]
 [ 1  8  6  1  8 10]
 [14  6 12  9 13  2]
 [ 5  2  1  1 13  2]
 [ 0 14  8  3  3 12]]
"""
# 等分拆成3份，按照行拆
b = np.split(a, indices_or_sections=3, axis=0)
print(b)
print(np.vsplit(a, indices_or_sections=3))
"""
[array([[ 2,  9,  7,  0,  7, 13],
       [ 2,  9, 10,  8,  6,  7]]), 
 array([[ 1,  8,  6,  1,  8, 10],
       [14,  6, 12,  9, 13,  2]]), 
 array([[ 5,  2,  1,  1, 13,  2],
       [ 0, 14,  8,  3,  3, 12]])]
"""

# 等分拆成2份,按照列拆
c = np.split(a, indices_or_sections=2, axis=1)
print(c)
print(np.hsplit(a, indices_or_sections=2))
"""
[array([[ 2,  9,  7],
       [ 2,  9, 10],
       [ 1,  8,  6],
       [14,  6, 12],
       [ 5,  2,  1],
       [ 0, 14,  8]]), 
array([[ 0,  7, 13],
       [ 8,  6,  7],
       [ 1,  8, 10],
       [ 9, 13,  2],
       [ 1, 13,  2],
       [ 3,  3, 12]])]
"""

# 参数给列表，则根据列表中的索引来进行切片
d = np.split(a, indices_or_sections=[1, 2, 3])
print(d)
"""
[array([[ 2,  9,  7,  0,  7, 13]]), 
array([[ 2,  9, 10,  8,  6,  7]]), 
array([[ 1,  8,  6,  1,  8, 10]]), 
array([[14,  6, 12,  9, 13,  2],
       [ 5,  2,  1,  1, 13,  2],
       [ 0, 14,  8,  3,  3, 12]])]
"""

8.NumPy broadcast mechanism

​ When the shapes of the two arrays are not the same, we can implement operations such as addition, subtraction, and multiplication by extending the array. This mechanism is called broadcasting.

• One-dimensional array broadcast

​ It can be simply understood that a one-dimensional array is a line, and a two-dimensional array is a surface. Now merge the one-dimensional array into a two-dimensional one, and then merge a line into a surface, then this line needs to be extended vertically into a surface Then merge with the two-dimensional surface, the ultimate goal is to achieve a one-to-one correspondence effect, so that the corresponding positions can be merged.

import numpy as np

arr1 = np.sort(np.array([0, 1, 2, 3] * 3)).reshape(4, 3)
arr2 = np.array([1, 2, 3])
print(arr1)
"""
[[0 0 0]
 [1 1 1]
 [2 2 2]
 [3 3 3]]
"""
print(arr2)
"""
[1 2 3]
"""
print(arr1 + arr2)
"""
[[1 2 3]
 [2 3 4]
 [3 4 5]
 [4 5 6]]
"""

• Two-dimensional array broadcast

There are not enough columns, fill them vertically

import numpy as np

arr1 = np.sort(np.array([0, 1, 2, 3] * 3)).reshape(4, 3)
arr2 = np.array([[1], [2], [3], [4]])
print(arr1)
"""
[[0 0 0]
 [1 1 1]
 [2 2 2]
 [3 3 3]]
"""
print(arr2)
"""
[[1]
 [2]
 [3]
 [4]]
"""
print(arr1 + arr2)
"""
[[1 1 1]
 [3 3 3]
 [5 5 5]
 [7 7 7]]
"""

• 3D array broadcast

It is equivalent to adding different quantities of copies. You need to copy the single copy of data until the copies have the same number and then add them.

import numpy as np

arr1 = np.array([0, 1, 2, 3, 4, 5, 6, 7] * 3).reshape(3, 4, 2)
# arr1为3份4行2列的数组
print(arr1)
"""
[[[0 1]
  [2 3]
  [4 5]
  [6 7]]

 [[0 1]
  [2 3]
  [4 5]
  [6 7]]

 [[0 1]
  [2 3]
  [4 5]
  [6 7]]]
"""

arr2 = np.array([0, 1, 2, 3, 4, 5, 6, 7]).reshape(4, 2)
# arr2为1份4行2列的数组
print(arr2)
"""
[[0 1]
 [2 3]
 [4 5]
 [6 7]]
"""

print(arr1 + arr2)
"""
[[[ 0  2]
  [ 4  6]
  [ 8 10]
  [12 14]]

 [[ 0  2]
  [ 4  6]
  [ 8 10]
  [12 14]]

 [[ 0  2]
  [ 4  6]
  [ 8 10]
  [12 14]]]

"""

9.NumPy general functions

9-1. Element-level numerical functions

abs、sqrt、square、exp、log、sin、cos、tan，maxinmum、minimum、all、any、inner、clip、round、trace、ceil、floor

import numpy as np

arr = np.array([-1, -2, -3, -4, 4, 9, 16])
print(arr)
# [-1 -2 -3 -4  4  9 16]

print(np.abs(arr))
# 绝对值 [ 1  2  3  4  4  9 16]

print(np.sqrt(arr[4:]))
# 开平方 [2. 3. 4.]

print(np.square(arr))
# 算平方 [  1   4   9  16  16  81 256]

print(np.min(arr))
# 最小值 -4

print(np.max(arr))
# 最大值 16

arr2 = np.array([666, -45, 0, 1, 1, 124, -999])
print(np.minimum(arr, arr2))
# 逐个对比arr和arr2的元素,列出小的值[  -1  -45   -3   -4    1    9 -999]
print(np.maximum(arr, arr2))
# 逐个对比arr和arr2的元素,列出大的值[666  -2   0   1   4 124  16]

9-2.where function

where function, three parameters, selects an array of values ​​when the condition is true, selects an array of values ​​when the condition is false

import numpy as np

arr1 = np.array([1, 3, 5, 7, 9])
arr2 = np.array([2, 4, 6, 8, 10])
cond = np.array([True, False, True, True, False])
print(np.where(cond, arr1, arr2))
# [ 1  4  5  7 10]
"""
逐个去查看cond中的值，当cond的值为True时，对应位置取arr1中的值，当cond的值为False时，对应位置取arr2中的值
"""
arr3 = np.random.randint(0, 30, size=20)
# ⼩于25还是⾃身的值，⼤于25设置成NULL值
print(np.where(arr3 < 25, arr3, "NULL"))
# ['3' '19' '11' '18' '11' 'NULL' 'NULL' 'NULL' '20' 'NULL' '10' '23' '12'  'NULL' 'NULL' '15' '22' '5' '13' 'NULL']

9-3. Sorting method

np also provides a sorting method. The sorting method is in-place sorting, that is, directly changing the original array.

import numpy as np

# 方法1
arr1 = np.array([111, 3, 555, 7, 999])
arr1.sort()
print(np.sort(arr1))
# [  3   7 111 555 999]

# 方法2 - 通过排序后得到的索引
arr2 = np.array([111, 3, 555, 7, 999])
arr_index = arr2.argsort()
print(arr2[arr_index])
# [  3   7 111 555 999]

9-4. Set operation function

import numpy as np

A = np.array([2, 4, 6, 8])
B = np.array([3, 4, 5, 6])

# 交集
print(np.intersect1d(A, B))
# [4 6] (A和B中都有的值)

# 并集
print(np.union1d(A, B))
# [2 3 4 5 6 8] (A或B中只要其中一个数组中有)

# 差集
print(np.setdiff1d(A, B))
# [2 8] (A中有并且B中没有的值)

9-5. Mathematical and statistical functions

min、max、mean、median、sum、std、var、cumsum、cumprod、argmin、argmax、
argwhere、cov、corrcoef

import numpy as np

arr1 = np.array([-666, 7, 2, 19, 23, 0, 888, 11, 6, 11])
# 计算最⼩值 -666
print(arr1.min())

# 计算最⼤值的索引 6
print(arr1.argmax())

# 返回⼤于20的元素的索引[[4][6]]
print(np.argwhere(arr1 > 20))

# 计算累加和
print(np.cumsum(arr1))
# [-666 -659 -657 -638 -615 -615  273  284  290  301]

arr2 = np.random.randint(0, 10, size=(4, 5))
print(arr2)
"""
[[3 5 0 1 1]
 [0 4 1 5 5]
 [2 9 3 9 4]
 [5 0 9 1 1]]
"""


# 计算列的平均值
print(arr2.mean(axis=0))
# [2.5  4.5  3.25 4.   2.75]

# 计算⾏的平均值
print(arr2.mean(axis=1))
# [2.  3.  5.4 3.2]

# 协⽅差矩阵
print(np.cov(arr2, rowvar=True))
"""
[[ 4.    0.    2.5  -3.75]
 [ 0.    5.5   5.75 -7.25]
 [ 2.5   5.75 11.3  -9.1 ]
 [-3.75 -7.25 -9.1  14.2 ]]
"""

# 相关性系数
print(np.corrcoef(arr2, rowvar=True))
"""
[[ 1.          0.          0.37185257 -0.49757334]
 [ 0.          1.          0.72936896 -0.82037514]
 [ 0.37185257  0.72936896  1.         -0.71838623]
 [-0.49757334 -0.82037514 -0.71838623  1.        ]]
"""

10.NumPy linear algebra

The most important method of matrix multiplication is the general matrix product. It only makes sense if the number of columns of the first matrix is ​​the same as the number of rows of the second matrix. Generally speaking, when simply referring to matrix product, it refers to the general matrix product. An m×n matrix is ​​an array of m×n numbers arranged in m rows and n columns. Because it compactly brings together a lot of data, sometimes it can easily express some complex models.