Linear Algebra – Dr. Alaa Alghwiri

Introduction to this course

This course provides a detailed foundation in linear algebra with practical Python and R code examples. Linear Algebra is a fundamental mathematical discipline that plays a crucial role in various fields, including machine learning and data science. This course covers essential concepts such as vectors, matrices, systems of linear equations, eigenvalues, eigenvectors, singular value decomposition (SVD), and principal component analysis (PCA). By the end of this course, you will have a solid understanding of linear algebra and its applications in data science and machine learning.

What is Linear Algebra?

Definition

Linear algebra is a branch of mathematics that deals with vector spaces and linear mappings between these spaces. It provides a framework for representing and solving complex problems using vectors, matrices, and linear transformations.

Significance and Applications in various fields

Learning linear algebra is essential for understanding various data science and machine learning concepts, such as dimensionality reduction, regression, and support vector machines (SVM). Linear algebra has numerous applications in different fields, including: Physics, Engineering, Economics, Finance, Cryptography, Data Analysis, Operations Research, Quantum Computing, Statistics, and Natural Language Processing.

Installing and Loading Required Packages

NumPy (for numerical operations)
Scipy (for scientific computing)
Matplotlib (for visualization)

#! pip install numpy scipy matplotlib scikit-learn

import numpy as np
import scipy.linalg as linalg
import matplotlib.pyplot as plt

Installing and Loading Required Packages

Matrix (for advanced matrix operations)
pracma (for additional mathematical functions)
ggplot2 (for visualization)

# install.packages(c("Matrix", "pracma", "ggplot2"))

library(Matrix)
library(pracma)
library(ggplot2)

Vectors

Creating vectors

Vector addition, subtraction, and scalar multiplication

Python
R

# Creating vectors
v1 = np.array([1, 2, 3])
v2 = np.array([4, 5, 6])

# Vector addition
v_add = v1 + v2

# Vector subtraction
v_sub = v1 - v2

# Scalar multiplication
scalar = 3
v_scalar = scalar * v1

print("Vector Addition:", v_add)

Vector Addition: [5 7 9]

print("Vector Subtraction:", v_sub)

Vector Subtraction: [-3 -3 -3]

print("Scalar Multiplication:", v_scalar)

Scalar Multiplication: [3 6 9]

# Creating vectors
v1 <- c(1, 2, 3)
v2 <- c(4, 5, 6)

# Vector addition
v_add <- v1 + v2

# Vector subtraction
v_sub <- v1 - v2

# Scalar multiplication
scalar <- 3
v_scalar <- scalar * v1

print(paste("Vector Addition:", paste(v_add, collapse = ", ")))

[1] "Vector Addition: 5, 7, 9"

print(paste("Vector Subtraction:", paste(v_sub, collapse = ", ")))

[1] "Vector Subtraction: -3, -3, -3"

print(paste("Scalar Multiplication:", paste(v_scalar, collapse = ", ")))

[1] "Scalar Multiplication: 3, 6, 9"

Dot Product

Python
R

dot_product = np.dot(v1, v2)
print("Dot Product:", dot_product)

Dot Product: 32

dot_product <- sum(v1 * v2)
print(paste("Dot Product:", dot_product))

[1] "Dot Product: 32"

Norm of a Vector

Python
R

norm_v1 = np.linalg.norm(v1)
print("Norm of v1:", norm_v1)

Norm of v1: 3.7416573867739413

norm_v1 <- norm(v1, type = "2")
print(paste("Norm of v1:", norm_v1))

[1] "Norm of v1: 3.74165738677394"

Matrices

Creating matrices

Matrix addition, subtraction, and scalar multiplication

Python
R

# Creating matrices
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Matrix addition
A_add_B = A + B

# Matrix subtraction
A_sub_B = A - B

# Scalar multiplication
scalar = 2
A_scalar = scalar * A

print("Matrix Addition:\n", A_add_B)

Matrix Addition:
 [[ 6  8]
 [10 12]]

print("Matrix Subtraction:\n", A_sub_B)

Matrix Subtraction:
 [[-4 -4]
 [-4 -4]]

print("Scalar Multiplication:\n", A_scalar)

Scalar Multiplication:
 [[2 4]
 [6 8]]

# Creating matrices
A <- matrix(c(1, 3, 2, 4), nrow = 2, byrow = TRUE)
B <- matrix(c(5, 7, 6, 8), nrow = 2, byrow = TRUE)
# Matrix addition
A_add_B <- A + B

# Matrix subtraction
A_sub_B <- A - B

# Scalar multiplication
scalar <- 2
A_scalar <- scalar * A

print("Matrix Addition:")

[1] "Matrix Addition:"

print(A_add_B)

     [,1] [,2]
[1,]    6   10
[2,]    8   12

print("Matrix Subtraction:")

[1] "Matrix Subtraction:"

print(A_sub_B)

     [,1] [,2]
[1,]   -4   -4
[2,]   -4   -4

print("Scalar Multiplication:")

[1] "Scalar Multiplication:"

print(A_scalar)

     [,1] [,2]
[1,]    2    6
[2,]    4    8

Matrix Multiplication

Python
R

A_mul_B = np.dot(A, B)
print("Matrix Multiplication:\n", A_mul_B)

Matrix Multiplication:
 [[19 22]
 [43 50]]

A_mul_B <- A %*% B
print("Matrix Multiplication:")

[1] "Matrix Multiplication:"

print(A_mul_B)

     [,1] [,2]
[1,]   23   31
[2,]   34   46

Transpose of a Matrix

Python
R

A_transpose = A.T
print("Transpose of A:\n", A_transpose)

Transpose of A:
 [[1 3]
 [2 4]]

A_transpose <- t(A)
print("Transpose of A:")

[1] "Transpose of A:"

print(A_transpose)

     [,1] [,2]
[1,]    1    2
[2,]    3    4

Determinant and Inverse

Python
R

det_A = np.linalg.det(A)
inv_A = np.linalg.inv(A)

print("Determinant of A:", det_A)

Determinant of A: -2.0000000000000004

print("Inverse of A:\n", inv_A)

Inverse of A:
 [[-2.   1. ]
 [ 1.5 -0.5]]

det_A <- det(A)
inv_A <- solve(A)

print(paste("Determinant of A:", det_A))

[1] "Determinant of A: -2"

print("Inverse of A:")

[1] "Inverse of A:"

print(inv_A)

     [,1] [,2]
[1,]   -2  1.5
[2,]    1 -0.5

Systems of Linear Equations

Solving Systems of Equations

Python
R

# Example system: A * x = b
A = np.array([[3, 2], [1, 2]])
b = np.array([5, 5])

# Solving for x
x = np.linalg.solve(A, b)
print("Solution of the system:")

Solution of the system:

print(x)

[2.96059473e-16 2.50000000e+00]

# Example system: A * x = b
A <- matrix(c(3, 1, 2, 2), nrow = 2)
b <- c(5, 5)

# Solving for x
x <- solve(A, b)
print("Solution of the system:")

[1] "Solution of the system:"

print(x)

[1] 0.0 2.5

Eigenvalues and Eigenvectors

Calculating Eigenvalues and Eigenvectors

Python
R

eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:")

Eigenvalues:

print(eigenvalues)

[4. 1.]

print("Eigenvectors:")

Eigenvectors:

print(eigenvectors)

[[ 0.89442719 -0.70710678]
 [ 0.4472136   0.70710678]]

eigen_decomp <- eigen(A)
eigenvalues <- eigen_decomp$values
eigenvectors <- eigen_decomp$vectors

print("Eigenvalues:")

[1] "Eigenvalues:"

print(eigenvalues)

[1] 4 1

print("Eigenvectors:")

[1] "Eigenvectors:"

print(eigenvectors)

          [,1]       [,2]
[1,] 0.8944272 -0.7071068
[2,] 0.4472136  0.7071068

Singular Value Decomposition (SVD)

Understanding SVD

Python
R

U, S, Vt = np.linalg.svd(A)
print("U:\n", U)

U:
 [[-0.86491009 -0.50192682]
 [-0.50192682  0.86491009]]

print("S (Singular values):\n", S)

S (Singular values):
 [4.13064859 0.96837093]

print("Vt:\n", Vt)

Vt:
 [[-0.74967818 -0.66180256]
 [-0.66180256  0.74967818]]

svd_decomp <- svd(A)
U <- svd_decomp$u
S <- svd_decomp$d
Vt <- svd_decomp$v

print("U:")

[1] "U:"

print(U)

           [,1]       [,2]
[1,] -0.8649101 -0.5019268
[2,] -0.5019268  0.8649101

print("Singular values (S):")

[1] "Singular values (S):"

print(S)

[1] 4.1306486 0.9683709

print("Vt:")

[1] "Vt:"

print(Vt)

           [,1]       [,2]
[1,] -0.7496782 -0.6618026
[2,] -0.6618026  0.7496782

Principal Component Analysis (PCA)

Applying PCA for Dimensionality Reduction

Python
R

from sklearn.decomposition import PCA

# Example data: 3-dimensional points
X = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

# Apply PCA
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)
print("Reduced Data:\n", X_reduced)

Reduced Data:
 [[-5.19615242e+00  5.55006974e-17]
 [ 0.00000000e+00 -0.00000000e+00]
 [ 5.19615242e+00  5.55006974e-17]]

# Example data: 3-dimensional points
X <- matrix(c(1, 2, 3, 4, 5, 6, 7, 8, 9), nrow = 3, byrow = TRUE)

# Apply PCA
pca <- prcomp(X, center = TRUE, scale. = TRUE)
X_reduced <- pca$x[, 1:2]  # Reduced to 2 dimensions

print("Reduced Data:")

[1] "Reduced Data:"

print(X_reduced)

           PC1 PC2
[1,] -1.732051   0
[2,]  0.000000   0
[3,]  1.732051   0

Linear Transformations and Projections

Visualizing Linear Transformations

Python
R

# Define a transformation matrix
T = np.array([[2, 0], [0, 0.5]])

# Define points to transform
points = np.array([[1, 2, 3], [1, 2, 3]])

# Apply transformation
transformed_points = np.dot(T, points)

# Plot original and transformed points
plt.scatter(points[0], points[1], label='Original')
plt.scatter(transformed_points[0], transformed_points[1], label='Transformed')
plt.legend()
plt.title("Linear Transformation")
plt.xlabel("x")
plt.ylabel("y")
plt.show()

# Define a transformation matrix
T <- matrix(c(2, 0, 0, 0.5), nrow = 2)

# Define points to transform
points <- matrix(c(1, 1, 2, 2, 3, 3), nrow = 2)

# Apply transformation
transformed_points <- T %*% points

# Plot original and transformed points
df <- data.frame(x = c(points[1, ], transformed_points[1, ]), 
                  y = c(points[2, ], transformed_points[2, ]),
                  type = rep(c("Original", "Transformed"), each = 3))

ggplot(df, aes(x = x, y = y, color = type)) +
  geom_point(size = 3) +
  labs(title = "Linear Transformation", x = "x", y = "y") +
  theme_minimal()

Applications of Linear Algebra

Machine Learning

Linear Regression

Python
R

from sklearn.linear_model import LinearRegression

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1.1, 2.0, 2.9, 4.1, 5.0])

# Train model
model = LinearRegression()
model.fit(X, y)

LinearRegression()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Predictions
predictions = model.predict(X)

print("Predictions:", predictions)

Predictions: [1.04 2.03 3.02 4.01 5.  ]

# Sample data
X <- cbind(1, c(1, 2, 3, 4, 5))  # Adding intercept term
y <- c(1.1, 2.0, 2.9, 4.1, 5.0)

# Train linear regression model
model <- lm(y ~ X - 1)  # -1 to omit intercept term as it is included in X

# Predictions
predictions <- predict(model, newdata = data.frame(X))

print("Predictions:")

[1] "Predictions:"

print(predictions)

   1    2    3    4    5 
1.04 2.03 3.02 4.01 5.00

Advanced Topics (Optional)

Matrix Factorizations

LU Decomposition

QR Decomposition

Python
R

# LU Decomposition
P, L, U = linalg.lu(A)
print("P:\n", P)

P:
 [[1. 0.]
 [0. 1.]]

print("L:\n", L)

L:
 [[1.         0.        ]
 [0.33333333 1.        ]]

print("U:\n", U)

U:
 [[3.         2.        ]
 [0.         1.33333333]]

# QR Decomposition
Q, R = np.linalg.qr(A)
print("Q:\n", Q)

Q:
 [[-0.9486833  -0.31622777]
 [-0.31622777  0.9486833 ]]

print("R:\n", R)

R:
 [[-3.16227766 -2.52982213]
 [ 0.          1.26491106]]

# LU Decomposition
lu_decomp <- lu(A)
P <- lu_decomp$P
L <- lu_decomp$L
U <- lu_decomp$U

print("P:")

[1] "P:"

print(P)

NULL

print("L:")

[1] "L:"

print(L)

          [,1] [,2]
[1,] 1.0000000    0
[2,] 0.3333333    1

print("U:")

[1] "U:"

print(U)

     [,1]     [,2]
[1,]    3 2.000000
[2,]    0 1.333333

# QR Decomposition
qr_decomp <- qr(A)
Q <- qr.Q(qr_decomp)
R <- qr.R(qr_decomp)

print("Q:")

[1] "Q:"

print(Q)

           [,1]       [,2]
[1,] -0.9486833 -0.3162278
[2,] -0.3162278  0.9486833

print("R:")

[1] "R:"

print(R)

          [,1]      [,2]
[1,] -3.162278 -2.529822
[2,]  0.000000  1.264911

References

Mathematics for Machine Learning, Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong, Cambridge University Press, 2020.