Chapter 5: NumPy Fundamentals
MODULE 2 | WEEK 5
Chapter 5: NumPy Fundamentals
So far we have kept data in Python lists. But when the data runs into thousands or millions of numbers, lists become slow and awkward. NumPy (Numerical Python) is the solution to exactly this problem. It is the foundation of Data Science, ML and Deep Learning: Pandas and Scikit-learn are built on top of NumPy, and TensorFlow works hand in hand with it (arrays and tensors convert to each other easily).
5.1 What is NumPy and why is it better than a list?
NumPy is a library that provides a special data structure called an array . An array is a table in which all elements are of the same type (for example, all numbers). This is why NumPy runs very fast.
4 big advantages of NumPy
-
Speed: 10 to 100 times faster than a list (because it runs in the C language underneath).
-
Less memory: because all elements are of the same type, it takes up less space.
-
Vectorization: calculations on a whole array at once without writing a loop (see below).
-
Foundation: Pandas and Scikit-learn are built on NumPy, and TensorFlow works seamlessly with NumPy arrays.
To use NumPy, we import it. Worldwide, its short name is np . NumPy is already installed in Google Colab.
import numpy as np
# Multiply every number in the list by 2 (needs a loop)
nums = [1, 2, 3, 4, 5]
doubled = []
for n in nums:
doubled.append(n * 2)
print(doubled) # [2, 4, 6, 8, 10]
# NumPy needs no loop (vectorization)
arr = np.array([1, 2, 3, 4, 5])
print(arr * 2) # [ 2 4 6 8 10]
import numpy as np
import time
big_list = list(range(1_000_000))
big_arr = np.arange(1_000_000)
t = time.time()
result = [x * 2 for x in big_list]
print("List:", round(time.time() - t, 4), "seconds")
t = time.time()
result = big_arr * 2
print("NumPy:", round(time.time() - t, 4), "seconds")
# NumPy is usually much faster
5.2 Creating arrays
Array from a list
import numpy as np
a = np.array([10, 20, 30]) # 1D array (vector)
b = np.array([[1, 2, 3], [4, 5, 6]]) # 2D array (matrix): 2 rows, 3 columns
c = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]]) # 3D array
print(a)
print(b)
print(c.shape) # (2, 2, 2)
Functions that create arrays automatically
| Function | Purpose | Example |
|---|---|---|
| np.zeros(shape) | All 0s | np.zeros((2, 3)) |
| np.ones(shape) | All 1s | np.ones((2, 2)) |
| np.full(shape, v) | All one value v | np.full((2, 2), 7) |
| np.eye(n) | Identity matrix (1s on the diagonal) | np.eye(3) |
| np.arange(start, stop, step) | Like range, but creates an array | np.arange(0, 10, 2) |
| np.linspace(start, stop, n) | n evenly spaced numbers from start to stop | np.linspace(0, 1, 5) |
| np.random.rand(r, c) | Random numbers between 0 and 1 | np.random.rand(2, 3) |
| np.random.randint(a, b, size) | Random whole numbers from a up to b-1 (b is not included) | np.random.randint(1, 10, 5) |
| np.random.randn(r, c) | Random numbers from a normal distribution | np.random.randn(3) |
import numpy as np
print(np.zeros((2, 3)))
print(np.ones((2, 2)))
print(np.full((2, 2), 7))
print(np.eye(3))
print(np.arange(0, 10, 2)) # [0 2 4 6 8]
print(np.linspace(0, 1, 5)) # [0. 0.25 0.5 0.75 1. ]
Random numbers and seed
Random numbers are different every time. Setting a seed gives the same numbers every time, so your result can be reproduced . This is very important in ML.
import numpy as np
np.random.seed(42) # set the seed
r = np.random.randint(1, 100, size=5)
print(r.shape) # (5,)
np.random.seed(42)
r2 = np.random.randint(1, 100, size=5)
print((r == r2).all()) # True: same numbers both times
5.3 Array attributes and changing the shape
import numpy as np
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr.shape) # (2, 3) -> 2 rows, 3 columns
print(arr.ndim) # 2 -> number of dimensions
print(arr.size) # 6 -> total elements
print(arr.dtype) # int64 -> data type (can be int32 on some systems)
f = np.array([1, 2, 3], dtype=float) # choose the type yourself
print(f) # [1. 2. 3.]
print(arr.astype(float)) # change the type
reshape, flatten, transpose
import numpy as np
a = np.arange(12) # [0 1 2 ... 11]
m = a.reshape(3, 4) # 3 rows, 4 columns
print(m)
print(a.reshape(2, -1).shape) # (2, 6) -1 means: work it out yourself
print(m.flatten()) # back from 2D to 1D
print(m.T) # transpose: rows and columns swapped
print(m.T.shape) # (4, 3)
5.4 Indexing and Slicing
1D array
Exactly like a list: index starts at 0, negative index counts from the end, slicing is [start:stop:step] .
import numpy as np
a = np.array([10, 20, 30, 40, 50])
print(a[0], a[-1]) # 10 50
print(a[1:4]) # [20 30 40]
print(a[::2]) # [10 30 50]
a[0] = 99
print(a) # [99 20 30 40 50]
2D array: [row, column]
import numpy as np
m = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
print(m[0, 0]) # 1 first row, first column
print(m[1, 2]) # 6
print(m[0]) # [1 2 3] the whole first row
print(m[:, 1]) # [2 5 8] the whole second column
print(m[0:2, 1:3]) # [[2 3]
# [5 6]] first 2 rows, columns 1 and 2
Boolean indexing (masking): selecting data by a condition
This is one of the most powerful features of NumPy. First a True/False array (the mask) is created, then you get only the elements where it is True .
import numpy as np
marks = np.array([45, 82, 67, 91, 33, 78])
mask = marks > 60
print(mask) # [False True True True False True]
print(marks[mask]) # [82 67 91 78]
print(marks[marks > 60]) # you can write it directly too
print(marks[(marks > 60) & (marks < 90)]) # [82 67 78] both conditions
print(marks[(marks < 40) | (marks > 90)]) # [91 33] either condition
marks[marks < 40] = 40 # make everything below 40 equal to 40
print(marks) # [45 82 67 91 40 78]
Fancy indexing (selecting with a list of indexes)
import numpy as np
a = np.array([10, 20, 30, 40, 50])
print(a[[0, 2, 4]]) # [10 30 50]
print(a[[4, 0]]) # [50 10]
5.5 Array Operations
Element-wise operations (applied to each element separately)
import numpy as np
a = np.array([1, 2, 3])
b = np.array([10, 20, 30])
print(a + b) # [11 22 33]
print(b - a) # [ 9 18 27]
print(a * b) # [10 40 90] product of each pair
print(b / a) # [10. 10. 10.]
print(a ** 2) # [1 4 9]
print(a * 10) # [10 20 30] the scalar applies to every element
print(a > 1) # [False True True]
Aggregation functions: sum, mean, max and more
| Function | Purpose |
|---|---|
| arr.sum() | Total of all |
| arr.mean() | Average |
| np.median(arr) | Middle value |
| arr.std() / arr.var() | Standard deviation / variance |
| arr.min() / arr.max() | Smallest / largest |
| arr.argmin() / arr.argmax() | **Position (index)**of the smallest / largest |
| arr.cumsum() | Running total (cumulative sum) |
| np.percentile(arr, q) | Value at the q-th percentile |
import numpy as np
marks = np.array([45, 82, 67, 91, 33, 78])
print(marks.sum()) # 396
print(marks.mean()) # 66.0
print(np.median(marks)) # 72.5
print(marks.max(), marks.min()) # 91 33
print(marks.argmax()) # 3 (91 is at index 3)
print(marks.cumsum()) # [ 45 127 194 285 318 396]
print(round(marks.std(), 2)) # 20.64
axis: row-wise and column-wise calculation (very important)
In a 2D array we can tell NumPy the direction of the calculation:
-
axis=0: downwards (calculation for each column). Rows collapse .
-
axis=1: sideways (calculation for each row). Columns collapse .
Mathematical functions
import numpy as np
a = np.array([1, 4, 9, 16])
print(np.sqrt(a)) # [1. 2. 3. 4.]
print(np.exp(np.array([0, 1]))) # [1. 2.71828183]
print(np.log(np.array([1, np.e]))) # [0. 1.]
print(np.round(np.array([1.234, 5.678]), 1)) # [1.2 5.7]
print(np.abs(np.array([-3, 2, -1]))) # [3 2 1]
sort, unique, where
import numpy as np
a = np.array([5, 2, 9, 2, 5, 1])
print(np.sort(a)) # [1 2 2 5 5 9]
print(np.unique(a)) # [1 2 5 9]
vals, counts = np.unique(a, return_counts=True)
print(vals, counts) # [1 2 5 9] [1 2 2 1]
# np.where: if-else on the whole array
marks = np.array([45, 82, 67, 33])
result = np.where(marks >= 50, "Pass", "Fail")
print(result) # ['Fail' 'Pass' 'Pass' 'Fail']
print(np.where(marks >= 50)[0]) # [1 2] positions where the condition is true
Joining and splitting arrays
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.concatenate([a, b])) # [1 2 3 4 5 6]
x = np.array([[1, 2], [3, 4]])
y = np.array([[5, 6], [7, 8]])
print(np.vstack([x, y])) # joined below (rows increased)
print(np.hstack([x, y])) # joined at the side (columns increased)
parts = np.split(np.arange(6), 3) # into 3 equal parts
print(parts) # [array([0, 1]), array([2, 3]), array([4, 5])]
5.6 Broadcasting (operations on arrays of different shapes)
Normally, for an element-wise operation both arrays must have the same shape . But NumPy offers a smart way called broadcasting : the smaller array automatically stretches to take the shape of the bigger array (in reality no copy is made; it is only treated that way in the calculation).
The 2 rules of broadcasting
NumPy compares the shapes of the two arrays from the last dimension towards the first . At every dimension:
-
Either the sizes of both are equal ,
-
Or the size of one of them is 1 (it will stretch).
If neither of these is true, you get an error.
import numpy as np
a = np.ones((3, 4)) # shape (3, 4)
b = np.ones(4) # shape (4,) -> matches (3,4)
print((a + b).shape) # (3, 4)
c = np.ones(3) # shape (3,) -> last dimension 3 vs 4: no match
try:
a + c
except ValueError as e:
print("Error:", e)
The real use of broadcasting: Standardization (z-score)
In ML we often transform each column so that its mean is 0 and its standard deviation is 1 . Formula: (x - mean) / std . Thanks to broadcasting this takes just one line.
import numpy as np
data = np.array([[170, 65],
[160, 55],
[180, 80],
[175, 70]], dtype=float) # height, weight
mean = data.mean(axis=0) # mean of each column
std = data.std(axis=0) # std of each column
z = (data - mean) / std # broadcasting
print(np.round(z, 2))
print(np.round(z.mean(axis=0), 2)) # [0. 0.]
print(np.round(z.std(axis=0), 2)) # [1. 1.]
5.7 Hands-on with NumPy
First try it yourself, then look at the solution.
Exercise 1: Even numbers from 1 to 20
import numpy as np
a = np.arange(1, 21)
print(a[a % 2 == 0]) # [ 2 4 6 8 10 12 14 16 18 20]
Exercise 2: Analysing a marks table
4 students, 3 subjects. Find each student's average, the highest mark in each subject, and the index of the topper.
import numpy as np
marks = np.array([[78, 85, 90],
[60, 55, 70],
[92, 88, 95],
[45, 65, 50]])
student_avg = marks.mean(axis=1)
print(np.round(student_avg, 2)) # [84.33 61.67 91.67 53.33]
print(marks.max(axis=0)) # [92 88 95]
print("Topper index:", student_avg.argmax()) # 2
Exercise 3: Turn negative numbers into 0
import numpy as np
a = np.array([4, -2, 7, -9, 0, 3])
a[a < 0] = 0
print(a) # [4 0 7 0 0 3]
Exercise 4: Min-Max Scaling (bringing values between 0 and 1)
Formula: (x - min) / (max - min)
import numpy as np
x = np.array([10, 20, 30, 40, 50], dtype=float)
scaled = (x - x.min()) / (x.max() - x.min())
print(scaled) # [0. 0.25 0.5 0.75 1. ]
Exercise 5: Chessboard pattern (8x8)
import numpy as np
board = np.zeros((8, 8), dtype=int)
board[1::2, ::2] = 1 # odd rows, even columns
board[::2, 1::2] = 1 # even rows, odd columns
print(board)
Exercise 6: Distance between two points
Distance = sqrt((x2-x1)<sup>2</sup> + (y2-y1)<sup>2</sup> )
import numpy as np
p1 = np.array([1, 2])
p2 = np.array([4, 6])
distance = np.sqrt(((p2 - p1) ** 2).sum())
print(distance) # 5.0
Exercise 7: Dot product and Matrix multiplication
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.dot(a, b)) # 32 = 1*4 + 2*5 + 3*6
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A @ B)
# [[19 22]
# [43 50]]
Exercise 8: Create random data and make sense of it
Create random marks (30 to 100) for 100 students. How many passed (>= 50), what is the average, and how many students scored above the average?
import numpy as np
np.random.seed(0)
marks = np.random.randint(30, 101, size=100)
print("Pass:", (marks >= 50).sum())
print("Average:", round(marks.mean(), 2))
print("Above average:", (marks > marks.mean()).sum())
Challenge: Scale every subject of the Exercise 2 table between 0 and 1 (the Exercise 4 formula with axis=0 ), without any for loop.
KEY TAKEAWAYS
-
A NumPy array is fast, uses less memory, and gives vectorization (calculation without loops).
-
Ways to create arrays: array, zeros, ones, eye, arange, linspace, random. A seed makes the result repeatable.
-
shape (2,3) means 2 rows, 3 columns. reshape, flatten and T change the shape.
-
2D indexing m[row, col]. A slice is a view of the original array; use .copy() for a copy.
-
Boolean masking: arr[arr > 60]. Two conditions: & and |, each in brackets.
-
axis=0 gives column-wise calculation, axis=1 gives row-wise calculation.
-
Broadcasting: the smaller array stretches to the shape of the bigger one (shapes are matched from the end; sizes must be equal or 1).