Chapter 4: Data Literacy – Data Collection to Data Analysis

Learning Outcomes

By the end of this chapter, students will be able to:

• Explain the importance of data literacy in AI
• Identify different data collection methods and their applications
• Comprehend mathematical concepts related to matrices and their operations
• Apply basic data analysis techniques
• Visualize data using various techniques

Theory

What is Data Literacy?

Data literacy is the ability to read, understand, create, and communicate data as information. In AI, it’s crucial for preprocessing data, interpreting model outputs, and making data-driven decisions.

Key Components of Data Literacy:

• Understanding data types and structures
• Interpreting statistical measures
• Creating meaningful visualizations
• Making informed decisions based on data

Data Collection Methods

1. Surveys and Questionnaires

• Description: Structured data collection from large populations
• Applications: User feedback, market research, opinion polling
• Advantages: Scalable, standardized responses
• Limitations: Response bias, limited depth

2. Sensors and IoT Devices

• Description: Real-time data collection from physical systems
• Applications: Weather monitoring, health tracking, smart cities
• Advantages: Continuous, automated data collection
• Limitations: Equipment costs, maintenance requirements

3. Web Scraping

• Description: Extracting data from websites programmatically
• Applications: Price monitoring, content aggregation, research
• Advantages: Access to large datasets, automation
• Limitations: Legal considerations, website structure changes

4. Databases

• Description: Structured data storage and retrieval systems
• Applications: Enterprise systems, historical records, transactions
• Advantages: Organized, queryable, secure
• Limitations: Requires setup and maintenance

5. Crowdsourcing

• Description: Collecting data from a large group of people
• Applications: Image labeling, transcription, problem-solving
• Advantages: Diverse perspectives, scalable
• Limitations: Quality control challenges

Exploring Data

Levels of Measurement

Data can be classified into four levels of measurement:

Statistical Analysis of Data

Measures of Central Tendency

import numpy as np

data = [85, 90, 78, 92, 88, 76, 95, 89, 84, 91]

# Mean (Average)
mean = np.mean(data)
print(f"Mean: {mean}")  # Output: 86.8

# Median (Middle value)
median = np.median(data)
print(f"Median: {median}")  # Output: 88.5

# Mode (Most frequent value)
from scipy import stats
mode = stats.mode(data)
print(f"Mode: {mode.mode}")

Measures of Dispersion

import numpy as np

data = [85, 90, 78, 92, 88, 76, 95, 89, 84, 91]

# Standard Deviation
std_dev = np.std(data)
print(f"Standard Deviation: {std_dev}")

# Variance
variance = np.var(data)
print(f"Variance: {variance}")

# Range
data_range = max(data) - min(data)
print(f"Range: {data_range}")

Introduction to Matrices

What is a Matrix?

A matrix is a rectangular array of numbers arranged in rows and columns. Matrices are fundamental in AI for:

• Representing data
• Storing model weights
• Performing transformations

Matrix Notation

A matrix A with m rows and n columns:

A = | a₁₁  a₁₂  a₁₃ |
    | a₂₁  a₂₂  a₂₃ |
    | a₃₁  a₃₂  a₃₃ |

Matrix Operations

Addition and Subtraction

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Addition
C = A + B
print("A + B =\n", C)  # [[6, 8], [10, 12]]

# Subtraction
D = A - B
print("A - B =\n", D)  # [[-4, -4], [-4, -4]]

Scalar Multiplication

import numpy as np

A = np.array([[1, 2], [3, 4]])
scalar = 3

result = scalar * A
print("3 * A =\n", result)  # [[3, 6], [9, 12]]

Matrix Multiplication

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Matrix multiplication
C = np.dot(A, B)
# or C = A @ B
print("A × B =\n", C)  # [[19, 22], [43, 50]]

Transpose

import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6]])

# Transpose
A_T = A.T
print("Transpose of A =\n", A_T)
# [[1, 4],
#  [2, 5],
#  [3, 6]]

Data Visualization with Python

Using Matplotlib

import matplotlib.pyplot as plt
import numpy as np

# Sample data
categories = ['A', 'B', 'C', 'D', 'E']
values = [23, 45, 56, 78, 32]

Line Graph

import matplotlib.pyplot as plt

months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun']
sales = [150, 180, 170, 200, 220, 250]

plt.figure(figsize=(10, 6))
plt.plot(months, sales, marker='o', color='blue', linewidth=2)
plt.title('Monthly Sales Trend')
plt.xlabel('Month')
plt.ylabel('Sales')
plt.grid(True)
plt.savefig('line_graph.png')
plt.show()

Bar Graph

import matplotlib.pyplot as plt

categories = ['Math', 'Science', 'English', 'History', 'Art']
scores = [85, 92, 78, 88, 95]

plt.figure(figsize=(10, 6))
plt.bar(categories, scores, color='steelblue')
plt.title('Subject-wise Scores')
plt.xlabel('Subjects')
plt.ylabel('Scores')
plt.ylim(0, 100)
plt.savefig('bar_graph.png')
plt.show()

Histogram

import matplotlib.pyplot as plt
import numpy as np

# Generate random data
data = np.random.normal(70, 15, 1000)

plt.figure(figsize=(10, 6))
plt.hist(data, bins=30, color='green', edgecolor='black', alpha=0.7)
plt.title('Distribution of Student Scores')
plt.xlabel('Score')
plt.ylabel('Frequency')
plt.savefig('histogram.png')
plt.show()

Scatter Plot

import matplotlib.pyplot as plt
import numpy as np

# Sample data
study_hours = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
exam_scores = [45, 50, 55, 65, 70, 75, 82, 88, 92, 95]

plt.figure(figsize=(10, 6))
plt.scatter(study_hours, exam_scores, color='red', s=100)
plt.title('Study Hours vs Exam Scores')
plt.xlabel('Study Hours')
plt.ylabel('Exam Score')
plt.grid(True)
plt.savefig('scatter_plot.png')
plt.show()

Pie Chart

import matplotlib.pyplot as plt

labels = ['Python', 'Java', 'JavaScript', 'C++', 'Others']
sizes = [35, 25, 20, 10, 10]
colors = ['#ff9999', '#66b3ff', '#99ff99', '#ffcc99', '#ff99cc']
explode = (0.1, 0, 0, 0, 0)

plt.figure(figsize=(8, 8))
plt.pie(sizes, explode=explode, labels=labels, colors=colors, autopct='%1.1f%%')
plt.title('Programming Language Popularity')
plt.savefig('pie_chart.png')
plt.show()

Data Pre-processing

Handling Missing Values

import pandas as pd
import numpy as np

# Create DataFrame with missing values
df = pd.DataFrame({
    'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'Age': [25, np.nan, 30, 28],
    'Score': [85, 90, np.nan, 78]
})

# Check for missing values
print(df.isnull().sum())

# Drop rows with missing values
df_dropped = df.dropna()

# Fill missing values with mean
df['Age'].fillna(df['Age'].mean(), inplace=True)
df['Score'].fillna(df['Score'].mean(), inplace=True)

Normalization and Scaling

from sklearn.preprocessing import MinMaxScaler, StandardScaler
import numpy as np

data = np.array([[100, 0.001], [8, 0.05], [50, 0.005], [88, 0.07]])

# Min-Max Normalization (scales to 0-1)
min_max_scaler = MinMaxScaler()
normalized = min_max_scaler.fit_transform(data)
print("Min-Max Normalized:\n", normalized)

# Standardization (mean=0, std=1)
standard_scaler = StandardScaler()
standardized = standard_scaler.fit_transform(data)
print("Standardized:\n", standardized)

Data in Modelling and Evaluation

Data Splitting

from sklearn.model_selection import train_test_split

X = [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
y = [0, 0, 1, 1, 1]

# Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training set size: {len(X_train)}")
print(f"Testing set size: {len(X_test)}")

Practical Activities

Activity 1: Identification of Level of Measurement

Identify the level of measurement for each variable:

Activity 2: Statistical Analysis with Python

import numpy as np

# Sample dataset: Student exam scores
scores = [78, 85, 92, 76, 88, 95, 82, 79, 91, 87, 83, 90, 86, 74, 93]

# Calculate statistics
mean = np.mean(scores)
median = np.median(scores)
std_dev = np.std(scores)
variance = np.var(scores)

print(f"Mean: {mean:.2f}")
print(f"Median: {median:.2f}")
print(f"Standard Deviation: {std_dev:.2f}")
print(f"Variance: {variance:.2f}")

Activity 3: Data Visualization

Create visualizations using the rainfall.csv dataset:

import pandas as pd
import matplotlib.pyplot as plt

# Read the rainfall data
df = pd.read_csv('rainfall.csv')

# Line graph - Monthly rainfall trend
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.plot(df['Month'], df['Rainfall'], marker='o')
plt.title('Monthly Rainfall')
plt.xlabel('Month')
plt.ylabel('Rainfall (mm)')
plt.xticks(rotation=45)

# Bar graph - Comparison
plt.subplot(1, 2, 2)
plt.bar(df['Month'], df['Rainfall'], color='skyblue')
plt.title('Monthly Rainfall Comparison')
plt.xlabel('Month')
plt.ylabel('Rainfall (mm)')
plt.xticks(rotation=45)

plt.tight_layout()
plt.savefig('rainfall_analysis.png')
plt.show()

Competency-Based Questions

Example Questions

1. Explain the concept of data literacy and its relevance to AI. (2 marks)
2. List three methods of data collection and provide examples for each. (3 marks)
3. Write a Python program to calculate the mean of a dataset. (4 marks)
4. Using matplotlib, create a scatter plot to visualize the relationship between two variables. (5 marks)
5. Discuss how data preprocessing impacts the performance of AI models. (6 marks)

Answers to Example Questions

1. Answer: Data literacy is the ability to read, understand, create, and communicate data. In AI, it’s essential for: understanding data requirements, preprocessing data correctly, interpreting model results, and making data-driven decisions. Without data literacy, AI models may be built on flawed data leading to poor outcomes.

2. Answer:

   • Surveys: Questionnaires collecting user opinions (e.g., Google Forms for market research)
   • Sensors/IoT: Automated real-time data (e.g., weather stations, fitness trackers)
   • Web Scraping: Extracting data from websites (e.g., collecting product prices from e-commerce sites)

3. Answer:

   import numpy as np
   
   data = [78, 85, 92, 76, 88, 95, 82, 79, 91, 87]
   
   # Method 1: Using NumPy
   mean_np = np.mean(data)
   
   # Method 2: Manual calculation
   mean_manual = sum(data) / len(data)
   
   print(f"Mean: {mean_np}")
   

4. Answer:

   import matplotlib.pyplot as plt
   
   # Sample data
   study_hours = [1, 2, 3, 4, 5, 6, 7, 8]
   exam_scores = [45, 50, 55, 65, 70, 78, 85, 92]
   
   plt.figure(figsize=(8, 6))
   plt.scatter(study_hours, exam_scores, color='blue', s=100)
   plt.title('Study Hours vs Exam Scores')
   plt.xlabel('Study Hours')
   plt.ylabel('Exam Score')
   plt.grid(True)
   plt.savefig('scatter_plot.png')
   plt.show()
   

5. Answer: Data preprocessing significantly impacts AI model performance:

   • Handling Missing Values: Prevents errors and biased results
   • Normalization/Scaling: Ensures features contribute equally, improves convergence
   • Outlier Removal: Prevents extreme values from skewing model
   • Feature Engineering: Creates informative features improving accuracy
   • Data Cleaning: Removes noise and inconsistencies
   • Poor preprocessing leads to “garbage in, garbage out” - unreliable predictions

Official Sample Paper Questions

1. What are the key steps in data preprocessing for AI applications? (2 marks)
2. Write a Python program to read a CSV file and calculate the standard deviation of a column. (3 marks)
3. Explain the difference between descriptive and inferential statistics. (4 marks)
4. Create a histogram to represent the frequency distribution of a dataset. (5 marks)
5. How can data visualization aid in understanding AI model outputs? (6 marks)

Answers to Official Sample Paper Questions

1. Answer: Key preprocessing steps: (1) Handle missing values (remove/impute), (2) Remove duplicates, (3) Handle outliers, (4) Normalize/scale features, (5) Encode categorical variables, (6) Split into training/testing sets.

2. Answer:

   import pandas as pd
   
   df = pd.read_csv('data.csv')
   std_dev = df['column_name'].std()
   print(f"Standard Deviation: {std_dev}")
   

3. Answer:

   Aspect	Descriptive Statistics	Inferential Statistics
   Purpose	Summarize data	Make predictions
   Scope	Describes current data	Generalizes to population
   Methods	Mean, median, mode	Hypothesis testing, regression
   Output	Tables, charts, numbers	Conclusions, predictions

4. Answer:

   import matplotlib.pyplot as plt
   import numpy as np
   
   # Generate sample data
   data = np.random.normal(70, 15, 500)
   
   plt.figure(figsize=(10, 6))
   plt.hist(data, bins=20, color='steelblue', edgecolor='black')
   plt.title('Frequency Distribution of Scores')
   plt.xlabel('Score')
   plt.ylabel('Frequency')
   plt.savefig('histogram.png')
   plt.show()
   

5. Answer: Data visualization aids AI understanding by:

   • Pattern Discovery: Identifying trends, clusters, and relationships
   • Outlier Detection: Spotting anomalies visually
   • Model Evaluation: Comparing predicted vs actual values
   • Feature Importance: Visualizing which features matter most
   • Communication: Explaining results to non-technical stakeholders
   • Debugging: Understanding why models make certain predictions

Practice Questions

Multiple Choice Questions

1. Which of the following is NOT a data collection method? a) Surveys b) Web scraping c) Machine learning d) Sensor data

2. What is the primary purpose of data preprocessing? a) To increase dataset size b) To clean and transform data c) To train machine learning models d) To visualize data

3. Which statistical measure indicates the spread of data? a) Mean b) Median c) Standard deviation d) Mode

4. Which level of measurement has a true zero point? a) Nominal b) Ordinal c) Interval d) Ratio

5. In matrix multiplication, if A is a 2×3 matrix and B is a 3×4 matrix, what is the size of A×B? a) 2×4 b) 3×3 c) 2×3 d) 3×4

Short Answer Questions

1. Define data literacy and explain why it’s important in AI.
2. Compare and contrast surveys and sensor-based data collection.
3. Write a Python function to calculate the variance of a list of numbers.
4. Describe how data visualization helps in AI model interpretation.

Long Answer Questions

1. Discuss the process of data collection for an AI project aimed at predicting crop yields.
2. Explain how matrices are used in neural network computations.
3. Design a data analysis pipeline for an AI application in healthcare.
4. Evaluate the effectiveness of different data visualization techniques for a given dataset.

Summary

Key Points

• Data literacy is essential for effective AI development and interpretation
• Common data collection methods include surveys, sensors, web scraping, and databases
• Statistical measures (mean, median, mode, standard deviation) help understand data
• Matrices and their operations form the mathematical foundation for AI
• Data visualization helps in understanding patterns and communicating results
• Data preprocessing (handling missing values, normalization) is crucial for AI models

Important Terminologies

• Data Literacy: Ability to read, understand, create, and communicate data
• Nominal Data: Categorical data without inherent order
• Ordinal Data: Categorical data with inherent order
• Interval Data: Numerical data with equal intervals but no true zero
• Ratio Data: Numerical data with equal intervals and true zero
• Mean: Average of all values
• Median: Middle value when sorted
• Mode: Most frequently occurring value
• Standard Deviation: Measure of data spread
• Matrix: Rectangular array of numbers
• Normalization: Scaling data to a specific range

Solutions to Practice Questions

Multiple Choice Answers

1. c) Machine learning
2. b) To clean and transform data
3. c) Standard deviation
4. d) Ratio
5. a) 2×4

Short Answer Model Answers

1. Data literacy is the ability to understand and work with data. It’s crucial in AI for tasks like data preprocessing, model interpretation, and making informed decisions based on data insights.
2. Surveys collect structured data from people through questions, while sensors gather real-time data from physical systems automatically. Surveys are better for subjective data; sensors are better for objective measurements.

3. def calculate_variance(numbers):
       mean = sum(numbers) / len(numbers)
       variance = sum((x - mean) ** 2 for x in numbers) / len(numbers)
       return variance
   

4. Data visualization helps identify patterns, outliers, and trends in data, making it easier to understand complex AI model outputs and communicate findings effectively.

Long Answer Model Answers

1. Data collection for crop yield prediction would involve gathering historical weather data, soil samples, and yield records. This data would be preprocessed to handle missing values and normalized before being used to train a regression model.
2. Matrices are used in neural networks to represent weights and perform operations like forward propagation. Matrix multiplication is fundamental to calculating activations in each layer.
3. A healthcare data analysis pipeline might involve collecting patient data, preprocessing it to handle missing values, applying feature engineering, training a classification model, and visualizing results for medical professionals.
4. Different visualization techniques serve different purposes. Bar charts show comparisons, line graphs show trends, scatter plots show correlations, and pie charts show proportions.

IBM Skills Build Integration

Complete the IBM Skills Build - Data Visualization with Python (Modules 1, 2, 3) course to:

• Learn data visualization fundamentals
• Master matplotlib and other visualization libraries
• Practice creating various chart types
• Earn a certification in data visualization

References

• CBSE Artificial Intelligence Curriculum for Class XI (2025-2026)
• IBM Skills Build - Data Visualization with Python
• NumPy and Pandas Official Documentation
• Matplotlib Documentation