Chapter 5: Machine Learning Algorithms

Learning Outcomes

By the end of this chapter, students will be able to:

• Differentiate between various types of machine learning methods
• Understand the concept behind each machine learning method
• Apply these methods to develop simple solutions for real-world problems
• Build up knowledge to apply during capstone project development

Theory

Machine Learning in a Nutshell

Machine learning is a subset of artificial intelligence that focuses on developing systems that can learn from data and improve their performance over time without being explicitly programmed. Instead of following rigid rules, ML systems identify patterns in data and make decisions based on those patterns.

Key Characteristics:

• Learning from experience (data)
• Improving performance over time
• Making predictions or decisions
• Handling complex, non-linear relationships

Types of Machine Learning

1. Supervised Learning

In supervised learning, the algorithm learns from labeled data (input-output pairs) to make predictions on new, unseen data.

Characteristics:

• Training data includes both features (inputs) and labels (outputs)
• Goal is to learn a mapping function from inputs to outputs
• Performance is measured against known correct answers

Applications:

• Email spam detection
• Medical diagnosis
• Credit scoring
• Image classification

2. Unsupervised Learning

In unsupervised learning, the algorithm finds patterns in data without labeled outputs.

Characteristics:

• Training data has no labels
• Algorithm discovers hidden structures
• Used for grouping similar data points

Applications:

• Customer segmentation
• Anomaly detection
• Market basket analysis
• Document clustering

3. Reinforcement Learning

In reinforcement learning, an agent learns by interacting with an environment and receiving rewards or penalties.

Characteristics:

• Learning through trial and error
• Reward-based feedback
• Sequential decision making
• Balance between exploration and exploitation

Applications:

• Game playing (Chess, Go)
• Robotics
• Autonomous vehicles
• Resource optimization

Supervised Learning Algorithms

Understanding Correlation

What is Correlation? Correlation measures the strength and direction of the relationship between two variables.

Pearson Correlation Coefficient (r):

• Ranges from -1 to +1
• +1: Perfect positive correlation
• 0: No correlation
• -1: Perfect negative correlation

Formula:

r = Σ(xi - x̄)(yi - ȳ) / √[Σ(xi - x̄)² × Σ(yi - ȳ)²]

Calculating in MS Excel:

=CORREL(data_range_x, data_range_y)

Example:

The correlation coefficient ≈ 0.98 (strong positive correlation)

Linear Regression

What is Linear Regression? Linear regression finds the best-fitting straight line through data points to predict continuous values.

The Line Equation:

y = mx + b

Where:

• y = predicted value
• x = input feature
• m = slope (coefficient)
• b = y-intercept (bias)

Finding the Best Line: The goal is to minimize the difference between predicted values and actual values (minimize error).

Mean Squared Error (MSE):

MSE = (1/n) × Σ(yi - ŷi)²

Linear Regression in MS Excel:

1. Select your data
2. Insert → Scatter chart
3. Click on data points → Add Trendline
4. Select “Linear” and “Display Equation on chart”

Linear Regression in Python:

from sklearn.linear_model import LinearRegression
import numpy as np

# Sample data
X = np.array([[1], [2], [3], [4], [5]])
y = np.array([45, 50, 55, 65, 70])

# Create and train model
model = LinearRegression()
model.fit(X, y)

# Model parameters
print(f"Slope (m): {model.coef_[0]:.2f}")
print(f"Intercept (b): {model.intercept_:.2f}")

# Make predictions
new_hours = np.array([[6], [7]])
predictions = model.predict(new_hours)
print(f"Predicted scores for 6 and 7 hours: {predictions}")

Classification

What is Classification? Classification assigns data points to predefined categories (classes).

Types of Classification:

• Binary Classification: Two classes (e.g., spam/not spam)
• Multi-class Classification: Multiple classes (e.g., animal types)

How Classification Works:

1. Feature extraction from training data
2. Learning decision boundaries
3. Assigning new data to classes based on boundaries

K-Nearest Neighbors (k-NN) Algorithm

What is k-NN? K-Nearest Neighbors classifies a data point based on the majority class of its k nearest neighbors.

How k-NN Works:

1. Choose the number of neighbors (k)
2. Calculate distance between new point and all training points
3. Find the k closest neighbors
4. Assign the class by majority vote

Distance Metrics:

• Euclidean Distance: √[(x₂-x₁)² + (y₂-y₁)²]
• Manhattan Distance: |x₂-x₁| + |y₂-y₁|

Choosing k:

• Small k: Sensitive to noise
• Large k: Smoother boundaries but may miss local patterns
• Common approach: Try odd values to avoid ties

k-NN in Python:

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
import numpy as np

# Sample data
X = np.array([[1, 2], [2, 3], [3, 1], [6, 5], [7, 7], [8, 6]])
y = np.array([0, 0, 0, 1, 1, 1])  # Two classes

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Create and train k-NN model
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)

# Make predictions
new_point = np.array([[4, 4]])
prediction = knn.predict(new_point)
print(f"Predicted class: {prediction[0]}")

# Evaluate accuracy
accuracy = knn.score(X_test, y_test)
print(f"Accuracy: {accuracy * 100:.2f}%")

Unsupervised Learning Algorithms

Clustering

What is Clustering? Clustering groups similar data points together without predefined labels.

Types of Clustering:

• Partition-based: k-means, k-medoids
• Hierarchical: Agglomerative, Divisive
• Density-based: DBSCAN
• Model-based: Gaussian Mixture Models

K-Means Clustering Algorithm

What is K-Means? K-Means partitions data into k clusters where each point belongs to the cluster with the nearest mean (centroid).

How K-Means Works:

1. Initialize: Randomly select k initial centroids
2. Assign: Assign each point to nearest centroid
3. Update: Recalculate centroids as mean of assigned points
4. Repeat: Continue until centroids don’t change

K-Means Algorithm Steps:

Step 1: Choose k (number of clusters)
Step 2: Initialize k centroids randomly
Step 3: Repeat until convergence:
    a. Assign each data point to nearest centroid
    b. Recalculate centroids as mean of cluster points
Step 4: Return final clusters

K-Means in Python:

from sklearn.cluster import KMeans
import numpy as np
import matplotlib.pyplot as plt

# Sample data
X = np.array([[1, 2], [1.5, 1.8], [5, 8], [8, 8], [1, 0.6], [9, 11]])

# Create k-means model
kmeans = KMeans(n_clusters=2, random_state=42)
kmeans.fit(X)

# Get cluster labels and centroids
labels = kmeans.labels_
centroids = kmeans.cluster_centers_

print(f"Cluster labels: {labels}")
print(f"Centroids:\n{centroids}")

# Visualize
plt.scatter(X[:, 0], X[:, 1], c=labels, cmap='viridis', s=100)
plt.scatter(centroids[:, 0], centroids[:, 1], c='red', marker='X', s=200)
plt.title('K-Means Clustering')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.savefig('kmeans_clustering.png')
plt.show()

Choosing k (Elbow Method):

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# Calculate inertia for different k values
inertias = []
K_range = range(1, 10)
for k in K_range:
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X)
    inertias.append(kmeans.inertia_)

# Plot elbow curve
plt.plot(K_range, inertias, 'bo-')
plt.xlabel('Number of clusters (k)')
plt.ylabel('Inertia')
plt.title('Elbow Method for Optimal k')
plt.savefig('elbow_method.png')
plt.show()

Summary of Algorithms

Practical Activities

Activity 1: Pearson Correlation in MS Excel

1. Open MS Excel with sample data (Study Hours vs Exam Scores)
2. Use =CORREL(A2:A10, B2:B10) to calculate correlation
3. Interpret the result

Activity 2: Linear Regression in MS Excel

1. Create a scatter plot with data
2. Add a trendline (Linear)
3. Display equation on chart
4. Use equation to predict new values

Activity 3: Linear Regression in Python (Advanced Learners)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Data: House size (sq ft) vs Price ($1000s)
X = np.array([[1000], [1500], [2000], [2500], [3000]])
y = np.array([150, 200, 250, 300, 350])

# Train model
model = LinearRegression()
model.fit(X, y)

# Predictions
X_pred = np.linspace(500, 3500, 100).reshape(-1, 1)
y_pred = model.predict(X_pred)

# Plot
plt.scatter(X, y, color='blue', label='Actual data')
plt.plot(X_pred, y_pred, color='red', label='Regression line')
plt.xlabel('House Size (sq ft)')
plt.ylabel('Price ($1000s)')
plt.title('House Price Prediction')
plt.legend()
plt.savefig('linear_regression.png')
plt.show()

Activity 4: k-NN Classification (Advanced Learners)

from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split

# Load iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Train k-NN
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)

# Evaluate
accuracy = knn.score(X_test, y_test)
print(f"Accuracy: {accuracy * 100:.2f}%")

Activity 5: K-Means Clustering (Advanced Learners)

from sklearn.cluster import KMeans
import pandas as pd
import matplotlib.pyplot as plt

# Customer data for segmentation
data = {
    'Annual Income': [15, 16, 17, 18, 19, 55, 56, 57, 58, 59],
    'Spending Score': [39, 81, 6, 77, 40, 5, 10, 23, 35, 37]
}
df = pd.DataFrame(data)

# K-Means clustering
kmeans = KMeans(n_clusters=2, random_state=42)
df['Cluster'] = kmeans.fit_predict(df)

# Visualize
plt.scatter(df['Annual Income'], df['Spending Score'], c=df['Cluster'], cmap='viridis')
plt.xlabel('Annual Income')
plt.ylabel('Spending Score')
plt.title('Customer Segmentation')
plt.savefig('customer_segments.png')
plt.show()

Competency-Based Questions

Example Questions

1. Explain the difference between supervised and unsupervised learning. (3 marks)
2. Write a Python program to implement linear regression using NumPy. (4 marks)
3. Describe how decision trees work and provide an example. (5 marks)
4. Discuss the advantages and limitations of K-Nearest Neighbors. (6 marks)

Answers to Example Questions

1. Answer:

   Aspect	Supervised Learning	Unsupervised Learning
   Data	Labeled (input-output pairs)	Unlabeled (input only)
   Goal	Predict outputs	Find patterns/structure
   Examples	Classification, Regression	Clustering, Dimensionality reduction
   Algorithms	Linear Regression, k-NN	K-Means, PCA

2. Answer:

   import numpy as np
   
   # Sample data
   X = np.array([1, 2, 3, 4, 5])
   y = np.array([2, 4, 5, 4, 5])
   
   # Calculate slope (m) and intercept (b)
   n = len(X)
   m = (n * np.sum(X * y) - np.sum(X) * np.sum(y)) / (n * np.sum(X**2) - np.sum(X)**2)
   b = (np.sum(y) - m * np.sum(X)) / n
   
   print(f"y = {m:.2f}x + {b:.2f}")
   
   # Predict for x = 6
   prediction = m * 6 + b
   print(f"Prediction for x=6: {prediction:.2f}")
   

3. Answer: Decision trees work by recursively splitting data based on feature values to create a tree-like model:

   • Root Node: Contains entire dataset
   • Splitting: Data divided based on best feature (using metrics like Gini impurity or information gain)
   • Branches: Each split creates branches
   • Leaf Nodes: Final predictions

   Example: Classifying whether to play tennis based on weather. Root splits on “Outlook” → if Sunny, checks “Humidity” → if High, decision is “Don’t Play.”

4. Answer: Advantages:

   • Simple to understand and implement
   • No training phase (lazy learning)
   • Works for classification and regression
   • Adapts to new data easily

   Limitations:

   • Slow for large datasets (calculates all distances)
   • Sensitive to irrelevant features
   • Requires feature scaling
   • Poor performance with high-dimensional data
   • Choice of k affects results significantly

Official Sample Paper Questions

1. What is the primary purpose of machine learning in AI? (2 marks)
2. Compare and contrast supervised and unsupervised learning. (3 marks)
3. Implement a KNN classifier to classify iris flowers using the scikit-learn library. (4 marks)
4. Explain how k-means clustering algorithm works with an example. (5 marks)

Answers to Official Sample Paper Questions

1. Answer: Machine learning enables AI systems to learn from data and improve performance without explicit programming. It allows systems to recognize patterns, make predictions, and automate decision-making by learning from experience.

2. Answer: Supervised learning uses labeled data to train models that predict known outputs (e.g., spam detection), while unsupervised learning finds hidden patterns in unlabeled data (e.g., customer segmentation). Supervised needs human-labeled examples; unsupervised discovers structure automatically.

3. Answer:

   from sklearn.datasets import load_iris
   from sklearn.neighbors import KNeighborsClassifier
   from sklearn.model_selection import train_test_split
   
   # Load data
   iris = load_iris()
   X_train, X_test, y_train, y_test = train_test_split(
       iris.data, iris.target, test_size=0.3, random_state=42
   )
   
   # Train k-NN
   knn = KNeighborsClassifier(n_neighbors=5)
   knn.fit(X_train, y_train)
   
   # Evaluate
   accuracy = knn.score(X_test, y_test)
   print(f"Accuracy: {accuracy * 100:.2f}%")
   

4. Answer: K-means clustering:

   1. Initialize: Randomly place k centroids
   2. Assign: Each point assigned to nearest centroid
   3. Update: Recalculate centroids as cluster means
   4. Repeat: Until centroids stabilize

   Example: Customer segmentation with k=3. Initial centroids placed randomly. Customers assigned to nearest centroid based on spending and income. Centroids recalculated. After iterations, three distinct customer segments emerge: low spenders, moderate spenders, high spenders.

Practice Questions

Multiple Choice Questions

1. Which algorithm is used for regression tasks? a) K-Nearest Neighbors b) K-Means Clustering c) Linear Regression d) Decision Tree Classification

2. In k-NN, what does ‘k’ represent? a) Number of features b) Number of neighbors c) Number of clusters d) Number of iterations

3. Which technique is used to find the optimal k in k-means? a) Cross-validation b) Elbow method c) Grid search d) Random search

4. What type of learning is k-means clustering? a) Supervised b) Unsupervised c) Reinforcement d) Semi-supervised

5. What does a correlation coefficient of -0.9 indicate? a) Strong positive correlation b) Weak positive correlation c) Strong negative correlation d) No correlation

Short Answer Questions

1. Define machine learning and explain its relationship to artificial intelligence.
2. Describe the steps involved in training a linear regression model.
3. How does the K-Nearest Neighbors algorithm work?
4. What is the purpose of the elbow method in k-means clustering?

Long Answer Questions

1. Discuss the applications of machine learning in healthcare and education.
2. Compare and contrast k-NN and k-means algorithms.
3. Design a machine learning pipeline for predicting student performance based on historical data.

Summary

Key Points

• Machine learning enables systems to learn from data and improve over time
• Three main types: Supervised, Unsupervised, and Reinforcement Learning
• Linear regression predicts continuous values using a best-fit line
• k-NN classifies data based on majority vote of nearest neighbors
• K-means clusters data by minimizing distance to cluster centroids
• Correlation measures the relationship between two variables

Important Terminologies

• Supervised Learning: Learning from labeled data
• Unsupervised Learning: Finding patterns in unlabeled data
• Reinforcement Learning: Learning through rewards and penalties
• Linear Regression: Predicting continuous values
• Classification: Assigning data to categories
• Clustering: Grouping similar data points
• Correlation: Measure of relationship strength
• Centroid: Center point of a cluster
• Overfitting: Model performs well on training but poorly on new data

Solutions to Practice Questions

Multiple Choice Answers

1. c) Linear Regression
2. b) Number of neighbors
3. b) Elbow method
4. b) Unsupervised
5. c) Strong negative correlation

Short Answer Model Answers

1. Machine learning is a subset of AI that focuses on developing systems that learn from data and improve their performance without explicit programming.
2. Steps: Collect and preprocess data, split into training/testing sets, initialize model, fit model to training data, evaluate on test data, tune parameters.
3. k-NN finds the k closest data points to a new point and assigns the majority class among those neighbors.
4. The elbow method plots inertia vs k and identifies the point where adding more clusters doesn’t significantly reduce inertia.

Long Answer Model Answers

1. In healthcare, ML enables disease diagnosis, drug discovery, and personalized treatment. In education, it powers adaptive learning systems, automated grading, and student performance prediction.
2. k-NN is a supervised algorithm for classification using labeled data, while k-means is an unsupervised algorithm for clustering without labels. k-NN uses distance to classify, k-means uses distance to cluster.
3. Pipeline: Collect student data (grades, attendance, demographics), preprocess (handle missing values, normalize), split data, train regression model, evaluate using MSE/R², deploy for predictions.

IBM Skills Build Integration

Complete the IBM Skills Build - Machine Learning with Python course to:

• Understand machine learning fundamentals
• Practice implementing ML algorithms
• Learn model evaluation techniques
• Earn a certification in machine learning

References

• CBSE Artificial Intelligence Curriculum for Class XI (2025-2026)
• IBM Skills Build - Machine Learning with Python
• Scikit-learn Documentation
• “Introduction to Machine Learning” by Ethem Alpaydin