Software Automation

How Machine Learning Supports Automation

🎯 (SE-12-01, SE-12-03, SE-12-09)

📌 Investigate how machine learning (ML) supports automation through the use of DevOps, robotic process automation (RPA) and business process automation (BPA)

A trained model does nothing until it is built into a process that runs reliably, day after day. Three kinds of automation carry ML into real work: DevOps (building and releasing software), RPA (bots that do single tasks) and BPA (whole workflows). Behind them sits MLOps, the discipline of building, releasing and looking after the models themselves. This section covers MLOps first, because the Course Specifications name its stages, and then shows how ML supports each of the three.

MLOps: designing, training and deploying models by automation

The Course Specifications describe MLOps as the automated process of designing, training and deploying ML models. It borrows many of the principles and practices of DevOps, bringing together the teams that develop models and the operational teams that deploy and support them in production. You are expected to know its three stages.

Worked example: a courier's delivery-time predictor

A courier company gets many phone calls asking when a parcel will arrive. Follow the project through the three stages.

• Design. The business problem is "too many customers phone to ask where their parcel is". Refactored as an ML problem it becomes "predict the delivery time of each parcel", a regression task. The team sets a success metric, for example "predictions within 30 minutes for at least 85% of parcels", and researches the data: past delivery scans, distances, depots and weather records. They also check that using customers' addresses is allowed under the Privacy Act 1988.
• Model development. Data wrangling removes duplicate scans, fixes missing timestamps and puts times into one format. Feature engineering creates useful inputs such as distance, hour of day and whether it is a public holiday. The model is trained on past deliveries, then tested and validated on the most recent month, which it has not seen, to check it meets the success metric.
• Operations. The approved model is deployed behind the tracking app. The help desk is briefed on what the estimate means (supporting operations/use). Each week a script compares predictions with what actually happened (monitoring model performance).

The last step is where automation earns its keep. Suppose the company opens deliveries to new suburbs, or a holiday rush changes the pattern. The model has not changed but the world has, so accuracy quietly falls. This is called model drift (or data drift, when the incoming data no longer looks like the training data). A monitoring script catches it before customers do.

# Weekly check of a deployed spam filter. Each pair is (model's prediction, true label); 1 = spam.
ACCURACY_TARGET = 0.85           # the success metric agreed in the Design stage

weekly_results = {
    "Week 1": [(1, 1), (0, 0), (1, 1), (0, 0), (0, 0), (1, 1), (0, 0), (1, 1), (0, 0), (1, 1)],
    "Week 2": [(1, 1), (0, 0), (0, 0), (0, 0), (1, 1), (1, 1), (0, 1), (1, 1), (0, 0), (1, 1)],
    "Week 3": [(0, 1), (0, 0), (1, 1), (0, 1), (0, 0), (1, 1), (0, 1), (1, 1), (0, 0), (0, 0)],
}

def accuracy(pairs):
    correct = sum(1 for predicted, actual in pairs if predicted == actual)
    return correct / len(pairs)

for week, pairs in weekly_results.items():
    score = accuracy(pairs)
    if score >= ACCURACY_TARGET:
        print(f"{week}: accuracy {score:.0%} - OK, keep serving this model")
    else:
        print(f"{week}: accuracy {score:.0%} - BELOW TARGET, alert the team and start retraining")

Week 1: accuracy 100% - OK, keep serving this model
Week 2: accuracy 90% - OK, keep serving this model
Week 3: accuracy 70% - BELOW TARGET, alert the team and start retraining

The check reports week 3 at 70%, below the 85% target from the Design stage, so it raises the alert. In a full MLOps pipeline the alert would start the next automated loop: retrain on newer data, test, and deploy only if the new model beats the old one. In NESA pseudocode that decision is a nested selection:

BEGIN RetrainCheck
    accuracy ← accuracy of the live model on this week's results
    IF accuracy >= target THEN
        OUTPUT "Keep serving the current model"
    ELSE
        retrain the model on the newest data
        IF new model accuracy > accuracy THEN
            deploy the new model
            OUTPUT "New model deployed"
        ELSE
            OUTPUT "Alert the team"
        ENDIF
    ENDIF
END RetrainCheck

DevOps and Automation

DevOps is a software development practice that unifies development and operations teams to deliver software continuously and reliably. A typical DevOps pipeline is automated from end to end: a developer commits code, a server builds it, automated tests run, and if they pass the release is deployed. MLOps reuses this idea, but the pipeline has extra things to test and version, because an ML system changes when its data changes, not only when its code changes. The trained model becomes another build artefact, and a pipeline can rebuild and redeploy it when new data arrives. ML also supports DevOps itself, for example by learning which code changes are most likely to cause test failures so those tests run first.

Robotic Process Automation (RPA)

RPA uses software "bots" to automate repetitive, rule-based digital tasks that humans would otherwise perform manually, such as copying data between systems, filling online forms, or processing invoices. Traditional RPA bots follow rigid scripts and break when a form changes layout or a document arrives in an unexpected format. When ML is integrated, the result is Intelligent Automation: bots that can read unstructured data (such as handwritten invoice fields or natural language emails), extract meaning from it, and make routing decisions. For example, an intelligent RPA bot in a hospital can read an unstructured doctor's referral email, extract the patient's details and condition, check availability, and book the appropriate specialist appointment, a task that previously required a receptionist.

Business Process Automation (BPA)

BPA involves using technology to automate entire end-to-end business workflows: not just individual repetitive tasks, but complete multi-step processes involving multiple systems and decision points. ML-driven BPA integrates predictive intelligence into workflows. For example, an insurance claims BPA system uses ML to assess a submitted photo of car damage, classify the severity, cross-reference the policy, calculate the repair estimate, and either auto-approve small claims or route complex cases to a human adjudicator, compressing a multi-day process into minutes. The key difference from RPA is that BPA automates the whole process, while RPA automates individual steps within a process.

Distinguishing AI from ML

🎯 (SE-12-03)

📌 Distinguish between artificial intelligence (AI) and ML

The differences, side by side

The same task done both ways

Both approaches below flag spam by counting exclamation marks. The first is AI without ML: a programmer picks the rule. The second is ML: the program chooses its own threshold from labelled examples. Then the spammers change their style, and the two approaches react differently.

# Task: flag spam using the number of exclamation marks in an email.
# Each example is (exclamation marks, label) where 1 = spam and 0 = not spam.
training = [(0, 0), (1, 0), (0, 0), (2, 0), (1, 0), (4, 1), (5, 1), (3, 1), (6, 1), (2, 0)]

# Approach 1: AI without ML. A programmer chooses the rule by hand.
def rule_based(marks):
    return 1 if marks >= 4 else 0

# Approach 2: ML. The program picks the threshold that makes the fewest mistakes on the data.
def learn_threshold(examples):
    best_threshold, fewest_mistakes = 0, len(examples) + 1
    for threshold in range(0, 10):
        mistakes = sum(1 for marks, label in examples if (1 if marks >= threshold else 0) != label)
        if mistakes < fewest_mistakes:
            best_threshold, fewest_mistakes = threshold, mistakes
    return best_threshold

def accuracy(predict, examples):
    return sum(1 for marks, label in examples if predict(marks) == label) / len(examples)

learned = learn_threshold(training)
print("Learned threshold:", learned)
print(f"Hand-written rule on training data: {accuracy(rule_based, training):.0%}")
print(f"Learned rule on training data:      {accuracy(lambda m: 1 if m >= learned else 0, training):.0%}")

# Spammers change style: new spam uses fewer exclamation marks. Same rule, new data.
new_data = [(0, 0), (1, 0), (0, 0), (2, 1), (3, 1), (2, 1), (1, 0), (3, 1), (2, 1), (0, 0)]
print(f"Hand-written rule on new data: {accuracy(rule_based, new_data):.0%}")
relearned = learn_threshold(new_data)
print("Re-learned threshold:", relearned)
print(f"Re-trained model on new data: {accuracy(lambda m: 1 if m >= relearned else 0, new_data):.0%}")

Learned threshold: 3
Hand-written rule on training data: 90%
Learned rule on training data:      100%
Hand-written rule on new data: 50%
Re-learned threshold: 2
Re-trained model on new data: 100%

The hand-written rule (four or more marks) misses the spam with three marks, so it is only 90% accurate on the data it was designed for. The learned threshold is 3 and scores 100%. When new spam uses fewer exclamation marks, the fixed rule drops to 50% and stays there until a programmer edits the code. The ML program only has to be run again on fresh data: the same code learns a new threshold of 2. That is what "learning from data" means in practice. It is not magic, though. Both programs look at a single feature, and the ML version is only as good as the examples it was given.

Common confusions

• "AI means robots." Robots use AI, but so does a chatbot or a route planner with no body at all.
• "ML and AI are the same." ML is one way of building AI. A rule-based expert system is AI with no learning.
• "Deep learning is something else." Deep learning is ML that uses neural networks with many layers, so it sits inside ML.
• "Generative AI is not ML." Chatbots and image generators are trained on huge datasets, so they are ML, and specifically deep learning.
• "The system understands." A trained model finds statistical patterns in its data. It can be confidently wrong when the input is unlike what it saw in training.

Models of Training ML

🎯 (SE-12-03)

📌 Explore models of training ML

• supervised learning
• unsupervised learning
• semi-supervised learning
• reinforcement learning

Worked examples

Each program below uses only the Python standard library and is short enough to trace by hand. They are simplified teaching versions of real algorithms, but the training idea in each is genuine. NESA does not ask you to code them: the aim is to see what "labelled", "unlabelled" and "reward" mean in practice.

Supervised learning: learn from labelled examples

The mango grader is given six mangoes already labelled ripe or unripe, each scored 0 to 10 for sweetness and softness. "Training" finds the average point for each label, and a new mango takes the label of the nearest average (a nearest-centroid classifier).

# Supervised learning: labelled examples of mangoes as (sweetness, softness), each scored 0-10.
labelled = [((8, 7), "ripe"), ((9, 8), "ripe"), ((7, 8), "ripe"),
            ((3, 2), "unripe"), ((2, 3), "unripe"), ((4, 1), "unripe")]

def centroid(points):
    return tuple(round(sum(values) / len(points), 1) for values in zip(*points))

def distance(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5

# "Training": find the average point of each labelled group.
centroids = {label: centroid([p for p, l in labelled if l == label]) for label in ("ripe", "unripe")}
print(centroids)

# "Prediction": choose the label whose centroid is nearest to the new mango.
def predict(point):
    return min(centroids, key=lambda label: distance(point, centroids[label]))

print(predict((6, 6)), predict((3, 4)))

{'ripe': (8.0, 7.7), 'unripe': (3.0, 2.0)}
ripe unripe

The mango scoring (6, 6) is about 2.6 units from the ripe average and 5 from the unripe one, so it is graded ripe. The mango scoring (3, 4) is 2 units from the unripe average, so it is graded unripe. The correct answers in the training data are what made this possible.

Unsupervised learning: find groups without labels

A shop has twelve customers' weekly spend and no labels. The k-means algorithm repeats two steps: put each value in the group with the nearest centre, then move each centre to the average of its group.

# Unsupervised learning: group customers by weekly spend (dollars). There are NO labels.
spend = [12, 15, 14, 11, 95, 102, 88, 110, 47, 52, 50, 13]

# k-means with k = 3: start the centres at the lowest, middle and highest values, then repeat.
centres = [min(spend), sorted(spend)[len(spend) // 2], max(spend)]
for step in range(10):
    groups = [[] for _ in centres]
    for value in spend:
        nearest = min(range(len(centres)), key=lambda i: abs(value - centres[i]))
        groups[nearest].append(value)
    centres = [sum(g) / len(g) for g in groups]

for i, group in enumerate(groups):
    print(f"Group {i + 1}: centre {centres[i]:.1f}, members {sorted(group)}")

Group 1: centre 13.0, members [11, 12, 13, 14, 15]
Group 2: centre 49.7, members [47, 50, 52]
Group 3: centre 98.8, members [88, 95, 102, 110]

The program found three groups of shoppers: small, medium and big spenders. Nobody told it what the groups mean. A person still has to choose how many groups to look for (k = 3 here) and decide what the groups are called.

Semi-supervised learning: a few labels, many unlabelled examples

Here only two points are labelled and six are not. Self-training repeatedly gives the unlabelled point closest to an already-labelled point the same label, so the labels spread outwards.

# Semi-supervised learning: 2 labelled points and 6 unlabelled points on a 0-10 scale.
labelled = {1.0: "low risk", 9.0: "high risk"}
unlabelled = [2.0, 3.0, 4.5, 6.0, 7.5, 8.0]

# Self-training: repeatedly label the unlabelled point closest to something already labelled.
while unlabelled:
    value = min(unlabelled, key=lambda u: min(abs(u - known) for known in labelled))
    nearest = min(labelled, key=lambda known: abs(value - known))
    labelled[value] = labelled[nearest]
    unlabelled.remove(value)
    print(f"{value} gets the label '{labelled[value]}' from its neighbour {nearest}")

2.0 gets the label 'low risk' from its neighbour 1.0
3.0 gets the label 'low risk' from its neighbour 2.0
8.0 gets the label 'high risk' from its neighbour 9.0
7.5 gets the label 'high risk' from its neighbour 8.0
4.5 gets the label 'low risk' from its neighbour 3.0
6.0 gets the label 'high risk' from its neighbour 7.5

Two hand-labelled points were enough to label all eight. The risk is visible too: if one early guess were wrong, every later label built on it would inherit the mistake, which is why real projects have experts check a sample of the machine-made labels.

Reinforcement learning: learn from rewards

An agent walks along a corridor of five cells and must find the goal at cell 4. It gets +10 for reaching the goal and −1 for every other move. It keeps a table of how good each action is in each cell (a Q-table), and 30% of the time it tries a random move to explore instead of using what it already knows.

import random

# Reinforcement learning: an agent in a corridor of 5 cells (0 to 4). The goal is cell 4.
# Reward: +10 for reaching the goal, -1 for every other move. Actions: 0 = left, 1 = right.
rng = random.Random(1)
q = [[0.0, 0.0] for _ in range(5)]        # Q-table: expected future reward for each state and action
ALPHA, GAMMA, EPSILON = 0.5, 0.9, 0.3     # learning rate, discount, chance of exploring

def best_action(state):
    return max((0, 1), key=lambda a: (q[state][a], a))

for episode in range(200):
    state = 0
    while state != 4:
        action = rng.randint(0, 1) if rng.random() < EPSILON else best_action(state)
        next_state = max(0, state - 1) if action == 0 else state + 1
        reward = 10 if next_state == 4 else -1
        q[state][action] += ALPHA * (reward + GAMMA * max(q[next_state]) - q[state][action])
        state = next_state

for state in range(4):
    print(f"Cell {state}: left {q[state][0]:6.2f}  right {q[state][1]:6.2f}  -> {'right' if best_action(state) else 'left'}")

Cell 0: left   3.12  right   4.58  -> right
Cell 1: left   3.12  right   6.20  -> right
Cell 2: left   4.58  right   8.00  -> right
Cell 3: left   6.20  right  10.00  -> right

No one told the agent the answer, yet it learned to go right from every cell. The values show the reward spreading backwards from the goal: 10.00 for stepping into the goal from cell 3, then 8.00 (−1 + 0.9 × 10) from cell 2, then 6.20 from cell 1. The 0.9 is the discount that makes nearer rewards worth more.

Common Applications of ML Algorithms

🎯 (SE-12-03, SE-12-05)

📌 Investigate common applications of key ML algorithms

• data analysis and forecasting
• virtual personal assistants
• image recognition

Data Analysis and Forecasting

ML algorithms applied to data analysis and forecasting use patterns in historical data to predict future outcomes. Regression algorithms (linear, polynomial) identify trends and project them forward. Time-series models (e.g., ARIMA, LSTM neural networks) specifically handle data ordered over time — such as stock prices, weather measurements, or website traffic. Key applications include:

Virtual Personal Assistants

Virtual personal assistants (VPAs) such as Apple's Siri, Amazon's Alexa, and Google Assistant combine multiple ML domains into a seamless conversational interface:

Image Recognition

Image recognition uses Convolutional Neural Networks (CNNs) — a specialised deep learning architecture designed to process grid-structured data (pixels). CNNs learn hierarchical visual features: early layers detect edges and colours; deeper layers recognise shapes and textures; the final layers identify objects. Key applications include:

Comparing the three applications

How a voice assistant answers, in order

1. A small model on the device listens only for the wake word.
2. Speech recognition turns the audio into text.
3. Natural language processing extracts the intent and entities: "Set a timer for 10 minutes" becomes SetTimer with 10 minutes.
4. The assistant carries out the action, for example by calling the clock service.
5. A text reply is generated and text-to-speech reads it aloud.

Investigating an application: questions to ask

• Accuracy in real conditions. Was it tested on data like the real world (noisy audio, poor lighting, new customers), or only on tidy samples?
• Data. Where did the training data come from, and does it represent everyone who will use the system?
• Human oversight. Who checks a wrong or unexpected output before it affects someone?
• Privacy and consent. What personal data is collected, and is the person told?
• Cost and benefit. Does the gain in speed or accuracy justify the risks and the running cost?

Design Models: Decision Trees and Neural Networks

🎯 (SE-12-03, SE-12-08)

📌 Research models used by software engineers to design and analyse ML

• decision trees
• neural networks

Decision Trees

A decision tree is a supervised learning model that represents a series of binary questions (decisions) as a tree structure, branching at each internal node based on a feature value, and arriving at a prediction at each leaf node. Decision trees are highly interpretable — engineers and non-technical stakeholders can read the tree logic directly. They are also the foundation for more powerful ensemble methods such as Random Forests (many decision trees voting together) and Gradient Boosted Trees (iteratively correcting prediction errors).

How a tree chooses its questions

A tree-building algorithm tries every available question and keeps the one that separates the labels best. A common measure is Gini impurity: 0 means a group holds only one label and 0.5 means an even mix of two. The best question leaves the lowest impurity, averaged over the two groups it creates and weighted by their sizes. Below, eight labelled emails ("spam", or "ham" for genuine mail) are tested against three yes/no questions.

# Choosing the root question of a decision tree with Gini impurity.
# Each email: (says "free", has a link, sender is a contact, label)
emails = [
    (1, 1, 0, "spam"), (1, 0, 0, "spam"), (1, 1, 0, "spam"), (0, 1, 0, "spam"),
    (0, 0, 1, "ham"),  (0, 1, 1, "ham"),  (1, 0, 1, "ham"),  (0, 0, 0, "ham"),
]
questions = ["says 'free'", "has a link", "sender is a contact"]

def gini(rows):
    """Impurity of a group: 0 = all one label, 0.5 = an even mix of two labels."""
    if not rows:
        return 0.0
    share_spam = sum(1 for r in rows if r[-1] == "spam") / len(rows)
    return 1 - share_spam ** 2 - (1 - share_spam) ** 2

def split_impurity(rows, feature):
    yes = [r for r in rows if r[feature] == 1]
    no = [r for r in rows if r[feature] == 0]
    return (len(yes) * gini(yes) + len(no) * gini(no)) / len(rows)

print(f"Impurity before any question: {gini(emails):.3f}")
for feature, question in enumerate(questions):
    print(f"Split on '{question}': weighted impurity {split_impurity(emails, feature):.3f}")

best = min(range(len(questions)), key=lambda f: split_impurity(emails, f))
print("Best root question:", questions[best])

Impurity before any question: 0.500
Split on 'says 'free'': weighted impurity 0.375
Split on 'has a link': weighted impurity 0.375
Split on 'sender is a contact': weighted impurity 0.200
Best root question: sender is a contact

Check the winner by hand. Three emails come from a contact and all are ham, so that group's impurity is 0. The other five hold four spam and one ham, so the impurity is 1 − 0.8² − 0.2² = 0.32. Weighted by size, (3 × 0 + 5 × 0.32) ÷ 8 = 0.200, the lowest of the three, so "sender is a contact" becomes the root. The algorithm then repeats the process inside each group until the leaves are pure or a limit such as a maximum depth is reached. Limits and pruning matter: a tree grown until every leaf is pure has just memorised its training emails (overfitting).

Neural Networks

Artificial Neural Networks (ANNs) are designed to loosely mimic the biological structure of the human brain. They consist of layers of interconnected artificial neurones (nodes), each receiving weighted input signals, applying an activation function, and passing an output signal to the next layer.

Following NESA's description

The Course Specifications describe a neural network as a series of interconnected nodes (artificial neurones), each able to accept a binary input signal and potentially output a signal to connected nodes. Two ideas from that description matter for the code in Part 2.

• Weightings and threshold values. Each connection has a weighting and each node has a threshold. A node fires when the weighted total of its inputs reaches its threshold. In NESA's diagram the thicker lines are the stronger weightings: those inputs count for more in the final output.
• Training cycle, then execution cycle. In the training cycle the network is shown inputs with known answers and its weightings and thresholds are adjusted; regular exposure improves accuracy and pattern matching. In the execution cycle those values are fixed and used to produce the output.

The Specifications say training uses "linear regression with backward chaining" to work out the values step by step. Most textbooks call the same idea backpropagation with gradient descent: the error at the output is passed backwards through the network and each weighting is nudged to reduce it. Use NESA's wording if a question does, and know the standard term as well.

A worked forward pass shows the execution cycle. A node has weightings 1.0 and 0.5 and a threshold of 1.5. For the inputs (1, 1) the weighted total is 1.0 × 1 + 0.5 × 1 = 1.5, which reaches the threshold, so the node fires and outputs 1. For (1, 0) the total is 1.0, below 1.5, so it outputs 0. That node behaves as an AND gate, and Part 2 shows the training cycle that finds those numbers.

Types of Algorithms Associated with ML

🎯 (SE-12-02, SE-12-03)

📌 Describe types of algorithms associated with ML

• linear regression
• logistic regression
• K-nearest neighbour

Linear Regression

Linear regression is a supervised learning algorithm used to predict a continuous numerical value based on one or more input features. It fits a straight line (or hyperplane in multiple dimensions) through the training data that minimises the total squared distance between each data point and the line — known as the Ordinary Least Squares (OLS) method. The resulting equation takes the form: y = mx + c, where m is the slope (coefficient) and c is the y-intercept. For example, a linear regression model trained on study hours and exam scores would output a predicted score for any given number of study hours.

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

# Sample data: study hours vs exam scores
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
y = np.array([45, 55, 60, 65, 70, 75, 80, 88])

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the model
model = LinearRegression()
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
print(f"R² Score: {r2_score(y_test, y_pred):.3f}")
print(f"MSE: {mean_squared_error(y_test, y_pred):.2f}")
print(f"Predicted score for 9 hours: {model.predict([[9]])[0]:.1f}")

R² Score: 0.963
MSE: 3.70
Predicted score for 9 hours: 93.3

The output comes from only two test points, so the scores are rough. The idea to take from it is the workflow: split the data, fit on the training part, and judge the model on data it has not seen.

Behind mean_squared_error is one of the simplest standard algorithms: a counting loop with an accumulator. It adds up each squared error, then divides by the number of items. The trace below shows the summing loop on its own. Enter some squared errors to see how the total builds.

Logistic Regression

Despite its name, logistic regression is a classification algorithm — not used for predicting continuous values. It calculates the probability that an input belongs to a particular class (typically binary: 0 or 1, True or False). The sigmoid function squashes any input value into a probability between 0 and 1. A threshold (typically 0.5) converts this probability into a class prediction. For example, a logistic regression model assessing whether a bank transaction is fraudulent outputs a probability of fraud; if it exceeds 0.5 (or a risk-adjusted threshold), the transaction is flagged. Logistic regression is highly interpretable — each feature's coefficient indicates its influence on the predicted probability.

A large negative sum gives a probability near 0, a sum of 0 gives exactly 0.5 (the usual decision boundary), and a large positive sum gives a probability near 1. That is why logistic regression can turn any score into a class.

K-Nearest Neighbour (KNN)

K-Nearest Neighbour (KNN) is a simple, instance-based supervised learning algorithm used for both classification and regression. Rather than learning a model during training, KNN stores the entire training dataset. At prediction time, it finds the K training examples closest to the new input (using a distance metric such as Euclidean distance), and assigns the majority class (classification) or average value (regression) of those K neighbours as the prediction. KNN requires no explicit training phase, but is computationally expensive at prediction time on large datasets.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler

# Features: [age, income], Labels: [0=no loan, 1=loan approved]
X_train = [[25, 30000], [35, 55000], [45, 80000], [22, 25000], [50, 90000]]
y_train = [0, 1, 1, 0, 1]

# Scale features (important for KNN!)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)

# Train with k=3
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_scaled, y_train)

# Predict for new customer
new_customer = scaler.transform([[30, 45000]])
prediction = knn.predict(new_customer)
print(f"Loan approved: {'Yes' if prediction[0] == 1 else 'No'}")

Loan approved: No

ML Regression Models Using OOP

🎯 (SE-12-02, SE-12-08)

📌 Design, develop and apply ML regression models using an OOP to predict numeric values

• linear regression
• polynomial regression
• logistic regression

The Course Specifications say students should know how to design programs that use and apply regression algorithms, and are not expected to implement (or code) these complex algorithms themselves. Linear and polynomial regression predict values in a continuous range, and logistic regression is used for classification. This section therefore does two jobs. It shows the scikit-learn style that the Specifications use, and it builds each model from scratch as a Python class so that you can see exactly what fit() does. The from-scratch code is optional depth, but it is the best way to understand why the library calls work.

Designing a regression program

Choose the model from the shape of the answer you need and the pattern in the data, then follow the same six steps whichever model you pick.

1. Collect and inspect the data (plot it: is it a line or a curve?).
2. Split it into training data and test data.
3. Create the model object.
4. Call fit() on the training data.
5. Call predict() on new inputs.
6. Measure the error on the test data and decide if it is good enough.

Linear regression in Python (OOP via scikit-learn)

The scikit-learn library implements regression models as Python classes. Each model is an object with fit() and predict() methods. This is a direct application of OOP: instantiating a model object, calling methods to train it, and calling methods to use it. Install it with pip install scikit-learn; it also needs NumPy.

import numpy as np
from sklearn.linear_model import LinearRegression

# Training data: hours studied vs exam score
X_train = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
y_train = np.array([45, 50, 55, 60, 65, 70, 75, 80])

# Instantiate the model object (OOP: creating an instance)
model = LinearRegression()

# Train the model by calling the fit() method on the object
model.fit(X_train, y_train)

# Predict exam score for 9 hours of study
prediction = model.predict([[9]])
print(f"Predicted score for 9 hours: {prediction[0]:.1f}")  # ~85.0

# Inspect the learned parameters (slope and intercept)
print(f"Slope (coefficient): {model.coef_[0]:.2f}")       # ~5.0
print(f"Intercept: {model.intercept_:.2f}")                # ~40.0

Predicted score for 9 hours: 85.0
Slope (coefficient): 5.00
Intercept: 40.00

The Course Specifications use the same calls on eight points that lie exactly on the line y = x − 1. Because the relationship is perfectly linear, predict([[4]]) returns 3 (a known point) and predict([[4.5]]) returns 3.5, a value the model was never shown. Real data is noisy, so a real model only gets close.

Linear regression from first principles

What does fit() compute? For a line y = mx + c, ordinary least squares chooses the slope and intercept that make the sum of the squared errors as small as possible. The formulas are m = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)² and c = ȳ − m·x̄, where x̄ and ȳ are the means. Here it is worked on eight students' study hours and marks (invented data):

The slope is 230 ÷ 42 = 5.476 marks per extra hour, and the intercept is 65 − 5.476 × 4.5 = 40.357. In NESA pseudocode the training step is a single loop:

BEGIN FitLine (xs, ys)
    n ← number of values in xs
    meanX ← sum of xs / n
    meanY ← sum of ys / n
    sxy ← 0
    sxx ← 0
    FOR i = 1 TO n STEP 1
        sxy ← sxy + (xs[i] - meanX) * (ys[i] - meanY)
        sxx ← sxx + (xs[i] - meanX) * (xs[i] - meanX)
    NEXT i
    slope ← sxy / sxx
    intercept ← meanY - slope * meanX
    RETURN slope, intercept
END FitLine (xs, ys)

Now the same calculation as a class. The design uses OOP in three ways. State: each object keeps its own slope and intercept as attributes. Behaviour: fit() sets that state and predict() uses it. Inheritance: the base class RegressionModel supplies mse() and r_squared(), so any model that provides predict() inherits them.

class RegressionModel:
    """Base class: every model learns with fit() and answers with predict()."""

    def fit(self, xs, ys):
        raise NotImplementedError

    def predict(self, x):
        raise NotImplementedError

    def mse(self, xs, ys):
        """Mean squared error: the average of the squared gaps between prediction and truth."""
        errors = [(self.predict(x) - y) ** 2 for x, y in zip(xs, ys)]
        return sum(errors) / len(errors)

    def r_squared(self, xs, ys):
        """1.0 = perfect fit; 0.0 = no better than always guessing the mean."""
        mean_y = sum(ys) / len(ys)
        total = sum((y - mean_y) ** 2 for y in ys)
        residual = sum((y - self.predict(x)) ** 2 for x, y in zip(xs, ys))
        return 1 - residual / total


class LinearRegression(RegressionModel):
    """A straight line, y = slope * x + intercept, fitted by ordinary least squares."""

    def __init__(self):
        self.slope = 0.0
        self.intercept = 0.0

    def fit(self, xs, ys):
        n = len(xs)
        mean_x = sum(xs) / n
        mean_y = sum(ys) / n
        s_xy = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
        s_xx = sum((x - mean_x) ** 2 for x in xs)
        self.slope = s_xy / s_xx
        self.intercept = mean_y - self.slope * mean_x
        return self

    def predict(self, x):
        return self.slope * x + self.intercept


hours = [1, 2, 3, 4, 5, 6, 7, 8]
marks = [45, 52, 58, 61, 68, 74, 77, 85]

model = LinearRegression()
model.fit(hours, marks)
print(f"slope = {model.slope:.3f}, intercept = {model.intercept:.3f}")
print(f"Predicted mark for 9 hours: {model.predict(9):.1f}")
print(f"MSE on the training data: {model.mse(hours, marks):.3f}")
print(f"R squared: {model.r_squared(hours, marks):.3f}")

slope = 5.476, intercept = 40.357
Predicted mark for 9 hours: 89.6
MSE on the training data: 1.060
R squared: 0.993

The slope and intercept match the hand working, and scikit-learn's LinearRegression gives the same 5.476 and 40.357 for this data. The MSE of 1.060 means predictions are typically about one mark out (the square root of the MSE is about 1.03). An R² of 0.993 says the line explains nearly all the variation in marks.

Least squares has a direct formula, but neural networks do not, so most ML training uses gradient descent instead: start with a guess, measure how the error changes as each parameter changes, and step downhill. This method finds the same line:

# Continues from the previous block: gradient descent finds the same line by repeated small steps.
def fit_by_gradient_descent(xs, ys, learning_rate=0.01, epochs=20000):
    slope, intercept = 0.0, 0.0
    n = len(xs)
    for epoch in range(epochs):
        # how much the MSE changes if slope or intercept increases a little
        slope_gradient = sum(2 * (slope * x + intercept - y) * x for x, y in zip(xs, ys)) / n
        intercept_gradient = sum(2 * (slope * x + intercept - y) for x, y in zip(xs, ys)) / n
        slope -= learning_rate * slope_gradient
        intercept -= learning_rate * intercept_gradient
    return slope, intercept

slope, intercept = fit_by_gradient_descent(hours, marks)
print(f"gradient descent: slope = {slope:.3f}, intercept = {intercept:.3f}")

gradient descent: slope = 5.476, intercept = 40.357

Explore it: fit a line yourself

Drag the sliders to place a line through the eight points and watch the mean squared error. Then use Gradient step to see training improve the line one step at a time, and Snap to least squares to jump to the answer.

Judging a model

• Mean squared error is the average squared gap between predictions and true values. Smaller is better, and squaring punishes big misses.
• R squared compares the model with always guessing the mean: 1 is perfect, 0 is no better than the mean.
• Test on unseen data. A model can score well on the data it trained on and badly on new data (overfitting), so hold some data back.
• Do not extrapolate blindly. The line above predicts 89.6 for 9 hours, which is sensible. For 20 hours it predicts a mark of about 150, which is impossible, because 20 hours lies far outside the 1 to 8 hours the model saw.

Polynomial Regression (Non-Linear Relationships)

Polynomial regression extends linear regression to model curved (non-linear) relationships between variables. When a scatter graph shows a curve rather than a straight line, a polynomial fit is more appropriate. The trick is to add powers of the input (x², x³ and so on) as extra features, then fit an ordinary linear model to them. Fuel use against speed is a good example: it falls as a car leaves slow traffic, then rises again at high speed, which no straight line can follow.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Speed (km/h) against fuel use (L/100 km): a curved relationship (invented sample data)
X = np.array([[40], [60], [80], [100], [120], [140]])
y = np.array([8.4, 6.5, 5.9, 6.6, 8.3, 11.2])

# Transform features to include x and x squared (degree 2 polynomial)
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

# Fit a linear regression model to the transformed features
model = LinearRegression()
model.fit(X_poly, y)

# Predict fuel use at 110 km/h
X_test = poly.transform([[110]])
print(f"Predicted fuel use at 110 km/h: {model.predict(X_test)[0]:.1f} L/100km")

Predicted fuel use at 110 km/h: 7.3 L/100km

Here is the same idea built from scratch. Fitting the powers by least squares gives a set of "normal equations", which the solve() function handles by Gaussian elimination. The class reuses RegressionModel and LinearRegression from the previous program, so it inherits mse() for free.

def solve(a, b):
    """Solve the linear equations a * w = b by Gaussian elimination."""
    n = len(b)
    for col in range(n):
        pivot = max(range(col, n), key=lambda r: abs(a[r][col]))
        a[col], a[pivot] = a[pivot], a[col]
        b[col], b[pivot] = b[pivot], b[col]
        for row in range(col + 1, n):
            factor = a[row][col] / a[col][col]
            for k in range(col, n):
                a[row][k] -= factor * a[col][k]
            b[row] -= factor * b[col]
    w = [0.0] * n
    for row in range(n - 1, -1, -1):
        w[row] = (b[row] - sum(a[row][k] * w[k] for k in range(row + 1, n))) / a[row][row]
    return w


class PolynomialRegression(RegressionModel):
    """y = c0 + c1*x + c2*x**2 + ... : linear regression on the powers of x."""

    def __init__(self, degree=2):
        self.degree = degree
        self.coefficients = []

    def _powers(self, x):
        return [x ** p for p in range(self.degree + 1)]

    def fit(self, xs, ys):
        rows = [self._powers(x) for x in xs]
        size = self.degree + 1
        # Least squares gives the "normal equations": (X^T X) w = X^T y
        a = [[sum(r[i] * r[j] for r in rows) for j in range(size)] for i in range(size)]
        b = [sum(r[i] * y for r, y in zip(rows, ys)) for i in range(size)]
        self.coefficients = solve(a, b)
        return self

    def predict(self, x):
        return sum(c * p for c, p in zip(self.coefficients, self._powers(x)))


speed = [40, 60, 80, 100, 120, 140]                # km/h
fuel = [8.4, 6.5, 5.9, 6.6, 8.3, 11.2]             # litres per 100 km (invented sample data)

straight = LinearRegression().fit(speed, fuel)
curve = PolynomialRegression(degree=2).fit(speed, fuel)
print(f"Straight line MSE: {straight.mse(speed, fuel):.3f}")
print(f"Quadratic MSE:     {curve.mse(speed, fuel):.3f}")
print(f"Fuel use at 110 km/h: line {straight.predict(110):.1f}, curve {curve.predict(110):.1f} L/100 km")

Straight line MSE: 2.189
Quadratic MSE:     0.002
Fuel use at 110 km/h: line 8.4, curve 7.3 L/100 km

The curve's error is close to zero, while the straight line misses badly, and its prediction at 110 km/h (7.3) agrees with scikit-learn's. A higher degree is not automatically better. With six points, a degree-5 polynomial can pass through every point exactly and then swing wildly between them, which is overfitting again.

Logistic Regression — Binary Classification

Logistic regression classifies inputs into one of two categories by predicting a probability using the sigmoid function. The OOP interface is identical to linear regression: instantiate, fit, predict.

from sklearn.linear_model import LogisticRegression
import numpy as np

# Training data: [study_hours, assignments_completed] → pass (1) or fail (0)
X_train = np.array([[1,2],[2,3],[3,4],[4,5],[5,6],[6,7],[2,1],[1,1],[3,2],[4,3]])
y_train = np.array([0, 0, 0, 1, 1, 1, 0, 0, 1, 1])

# Instantiate and train the logistic regression classifier
classifier = LogisticRegression()
classifier.fit(X_train, y_train)

# Predict outcome for a student: 5 study hours, 5 assignments completed
prediction = classifier.predict([[5, 5]])
probability = classifier.predict_proba([[5, 5]])
print(f"Prediction: {'Pass' if prediction[0] == 1 else 'Fail'}")
print(f"Pass probability: {probability[0][1]:.2%}")

Prediction: Pass
Pass probability: 93.49%

Building it by hand shows what happens inside fit(). For each training example the class predicts a probability, measures how far it is from the truth (the label 0 or 1), and moves the weight and bias to reduce the gap. That is gradient descent again, with the sigmoid inside the loop.

import math


class LogisticRegression:
    """Predicts the probability of class 1 with a sigmoid, trained by gradient descent."""

    def __init__(self, learning_rate=0.1, epochs=5000):
        self.learning_rate = learning_rate
        self.epochs = epochs
        self.weight = 0.0
        self.bias = 0.0

    @staticmethod
    def sigmoid(z):
        return 1 / (1 + math.exp(-z))

    def predict_proba(self, x):
        return self.sigmoid(self.weight * x + self.bias)

    def fit(self, xs, labels):
        for epoch in range(self.epochs):
            for x, label in zip(xs, labels):
                error = self.predict_proba(x) - label      # prediction minus truth
                self.weight -= self.learning_rate * error * x
                self.bias -= self.learning_rate * error
        return self

    def predict(self, x, threshold=0.5):
        return 1 if self.predict_proba(x) >= threshold else 0


study_hours = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
passed      = [0, 0, 0, 1, 0, 1, 1, 0, 1, 1]      # 1 = passed the test (invented sample data)

classifier = LogisticRegression().fit(study_hours, passed)
print(f"weight = {classifier.weight:.3f}, bias = {classifier.bias:.3f}")
for hours_studied in (2, 5.5, 8):
    p = classifier.predict_proba(hours_studied)
    print(f"{hours_studied} hours: P(pass) = {p:.2f} -> {'Pass' if classifier.predict(hours_studied) else 'Fail'}")

weight = 1.239, bias = -6.068
2 hours: P(pass) = 0.03 -> Fail
5.5 hours: P(pass) = 0.68 -> Pass
8 hours: P(pass) = 0.98 -> Pass

Check one prediction by hand. For 5.5 hours the weighted sum is z = 1.239 × 5.5 − 6.068 = 0.747, and the sigmoid of 0.747 is about 0.68. That is above the 0.5 threshold, so the class predicts a pass, though with only 68% confidence. The data overlaps (the student with 5 hours failed and the student with 6 passed), so a sensible model does not claim certainty.

Neural Network Models Using OOP

🎯 (SE-12-02, SE-12-08)

📌 Apply neural network models using an OOP to make predictions

A neural network is a natural fit for OOP. Its weightings and thresholds are the object's state, and the two cycles in the Course Specifications become methods: the training cycle adjusts the state using examples with known answers, and the execution cycle uses that state to make a prediction. This section builds a single neurone and then a small network from scratch, using only the standard library, and then shows the scikit-learn version that a real project would use. You are not expected to write these from memory in the exam. The aim is to see every step of the two cycles.

A single neurone as a class

The simplest neural network is one neurone, called a perceptron. It multiplies each input by a weighting, adds the results, and fires (outputs 1) if the total reaches its threshold. Like NESA's neurone, it takes binary inputs. It learns by the training cycle: for each example it compares its output with the known answer, and if it is wrong it nudges each weighting and the threshold.

BEGIN TrainNeurone (input1, input2, target, learningRate)
    weight1 ← 0
    weight2 ← 0
    threshold ← 0
    REPEAT
        mistakes ← 0
        FOR i = 1 TO 4 STEP 1
            total ← weight1 * input1[i] + weight2 * input2[i]
            IF total >= threshold THEN
                output ← 1
            ELSE
                output ← 0
            ENDIF
            error ← target[i] - output
            IF error != 0 THEN
                mistakes ← mistakes + 1
            ENDIF
            weight1 ← weight1 + learningRate * error * input1[i]
            weight2 ← weight2 + learningRate * error * input2[i]
            threshold ← threshold - learningRate * error
        NEXT i
    UNTIL mistakes = 0
END TrainNeurone (input1, input2, target, learningRate)

The class below is that algorithm. predict() is the execution cycle (a forward pass), train_step() is one pass of the training cycle for one example, and train() repeats it for whole epochs. The learning rate is 0.5, which keeps every number exact in binary arithmetic so the output is the same on every machine.

class Perceptron:
    """One artificial neurone: weighted inputs, a threshold and a step output."""

    def __init__(self, n_inputs, learning_rate=0.5):
        self.weights = [0.0] * n_inputs
        self.threshold = 0.0
        self.learning_rate = learning_rate

    def predict(self, inputs):
        """Execution cycle: the forward pass. Fire (1) if the weighted sum reaches the threshold."""
        total = 0.0
        for weight, x in zip(self.weights, inputs):
            total += weight * x
        return 1 if total >= self.threshold else 0

    def train_step(self, inputs, target):
        """Training cycle: predict, measure the error, then nudge every weight."""
        error = target - self.predict(inputs)
        for i, x in enumerate(inputs):
            self.weights[i] += self.learning_rate * error * x
        self.threshold -= self.learning_rate * error
        return error

    def train(self, examples, max_epochs=20, show=True):
        for epoch in range(1, max_epochs + 1):
            mistakes = sum(abs(self.train_step(inputs, target)) for inputs, target in examples)
            if show:
                w = [round(v, 2) for v in self.weights]
                print(f"Epoch {epoch}: {mistakes} mistakes, weights {w}, threshold {self.threshold:.2f}")
            if mistakes == 0:
                return epoch
        return None


AND = [((0, 0), 0), ((0, 1), 0), ((1, 0), 0), ((1, 1), 1)]

neurone = Perceptron(2)
neurone.train(AND)
for inputs, target in AND:
    print(inputs, "->", neurone.predict(inputs), "(expected", target, ")")

Epoch 1: 2 mistakes, weights [0.5, 0.5], threshold 0.00
Epoch 2: 3 mistakes, weights [1.0, 0.5], threshold 0.50
Epoch 3: 3 mistakes, weights [1.0, 0.5], threshold 1.00
Epoch 4: 2 mistakes, weights [1.0, 1.0], threshold 1.00
Epoch 5: 1 mistakes, weights [1.0, 0.5], threshold 1.50
Epoch 6: 0 mistakes, weights [1.0, 0.5], threshold 1.50
(0, 0) -> 0 (expected 0 )
(0, 1) -> 0 (expected 0 )
(1, 0) -> 0 (expected 0 )
(1, 1) -> 1 (expected 1 )

Trace the first epoch by hand from weights (0, 0) and threshold 0:

Two mistakes, ending on weights (0.5, 0.5) and threshold 0, which matches the first line of output. The first example was wrong only because the neurone started out saying 1 to everything, and the fourth because it then said 0 to everything. After six epochs the weightings are 1.0 and 0.5 and the threshold is 1.5: only when both inputs are 1 does the total (1.5) reach the threshold, which is exactly AND.

Explore it: step a perceptron through training

Pick a function, then press Step to watch the weighted total, error and weight changes for one example at a time. The dividing line on the right is where the total equals the threshold. Try AND and OR, then try XOR.

What one neurone cannot do

The same class learns OR in four epochs but never learns XOR ("one or the other, but not both"):

OR  = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 1)]
XOR = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]

for name, examples in (("OR", OR), ("XOR", XOR)):
    neurone = Perceptron(2)
    epochs = neurone.train(examples, max_epochs=100, show=False)
    if epochs:
        print(f"{name}: learned in {epochs} epochs, weights {[round(w, 2) for w in neurone.weights]}, threshold {neurone.threshold:.2f}")
    else:
        print(f"{name}: still making mistakes after 100 epochs")

OR: learned in 4 epochs, weights [0.5, 0.5], threshold 0.50
XOR: still making mistakes after 100 epochs

A single neurone draws one straight line through the input space. AND and OR can be separated by a straight line, but no straight line separates XOR's 1s from its 0s. This is why real networks have hidden layers: the hidden neurones each draw a line and the output neurone combines them.

A network with a hidden layer

The next class has two inputs, two hidden neurones and one output neurone. It uses a smooth sigmoid instead of a hard threshold (the bias plays the part of NESA's threshold), because smooth outputs let the error be traced backwards. predict() is again the execution cycle. train_step() is the training cycle: a forward pass, then how far the output is from the target, then that error is shared backwards with each hidden neurone in proportion to the weighting that connects it (backpropagation), and finally every weighting is adjusted a little.

import math
import random


def sigmoid(z):
    return 1 / (1 + math.exp(-z))


class Neuron:
    def __init__(self, n_inputs, rng):
        self.weights = [rng.uniform(-1, 1) for _ in range(n_inputs)]
        self.bias = rng.uniform(-1, 1)          # plays the part of the threshold
        self.inputs = []
        self.output = 0.0
        self.delta = 0.0                        # this neurone's share of the error

    def forward(self, inputs):
        self.inputs = inputs
        z = sum(w * x for w, x in zip(self.weights, inputs)) + self.bias
        self.output = sigmoid(z)
        return self.output


class NeuralNetwork:
    """A network with one hidden layer, trained by backpropagation."""

    def __init__(self, n_inputs, n_hidden, seed=1, learning_rate=0.5):
        rng = random.Random(seed)
        self.hidden = [Neuron(n_inputs, rng) for _ in range(n_hidden)]
        self.output = Neuron(n_hidden, rng)
        self.learning_rate = learning_rate

    def predict(self, inputs):
        """Execution cycle: pass the signals forward, layer by layer."""
        hidden_outputs = [neuron.forward(inputs) for neuron in self.hidden]
        return self.output.forward(hidden_outputs)

    def train_step(self, inputs, target):
        """Training cycle: forward pass, then push the error backwards and adjust weights."""
        prediction = self.predict(inputs)
        self.output.delta = (target - prediction) * prediction * (1 - prediction)
        for i, neuron in enumerate(self.hidden):
            neuron.delta = neuron.output * (1 - neuron.output) * self.output.weights[i] * self.output.delta
        for neuron in [self.output] + self.hidden:
            for j, x in enumerate(neuron.inputs):
                neuron.weights[j] += self.learning_rate * neuron.delta * x
            neuron.bias += self.learning_rate * neuron.delta
        return (target - prediction) ** 2

    def train(self, examples, epochs):
        for epoch in range(1, epochs + 1):
            loss = sum(self.train_step(inputs, target) for inputs, target in examples) / len(examples)
            if epoch in (1, 1000, 5000, 10000):
                print(f"Epoch {epoch:>5}: mean squared error {loss:.4f}")


XOR = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]

network = NeuralNetwork(n_inputs=2, n_hidden=2, seed=1)
network.train(XOR, epochs=10000)
for inputs, target in XOR:
    print(inputs, f"-> {network.predict(inputs):.3f} (expected {target})")

Epoch     1: mean squared error 0.2707
Epoch  1000: mean squared error 0.2542
Epoch  5000: mean squared error 0.0010
Epoch 10000: mean squared error 0.0003
(0, 0) -> 0.018 (expected 0)
(0, 1) -> 0.983 (expected 1)
(1, 0) -> 0.979 (expected 1)
(1, 1) -> 0.017 (expected 0)

The mean squared error falls from 0.27 to 0.0003, and the four outputs end up close to 0, 1, 1 and 0. Two things are worth noticing. Training is not smooth: at epoch 1000 the error had barely moved, then it dropped quickly. And the starting weights are random, seeded here so the run repeats. A different start can matter. With seed 7 this same network stayed stuck with two examples wrong (error 0.129 after 10,000 epochs), which is why real projects train several times or add more hidden neurones. Small differences in the last digit are possible on other machines.

Multi-Layer Perceptron (MLP) with scikit-learn

The MLPClassifier (Multi-Layer Perceptron) is a neural network class available in scikit-learn. It uses the same OOP interface as other models: instantiate with architecture parameters, call fit() to train, and call predict() to classify. The hidden_layer_sizes parameter defines the number of hidden layers and neurons per layer.

from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
import numpy as np

# Training data: [temperature, humidity] → weather type
# 0 = Sunny, 1 = Cloudy, 2 = Rainy
X_train = np.array([
    [30, 20], [32, 25], [28, 30],   # Sunny
    [22, 55], [20, 60], [18, 65],   # Cloudy
    [15, 85], [12, 90], [10, 95]    # Rainy
])
y_train = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

# Scale features (important for neural networks)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)

# Instantiate MLP with 2 hidden layers: 8 neurons, then 4 neurons
nn_model = MLPClassifier(
    hidden_layer_sizes=(8, 4),
    activation='relu',       # Rectified Linear Unit activation function
    max_iter=1000,
    random_state=42
)

# Train the neural network (backpropagation runs internally)
nn_model.fit(X_scaled, y_train)

# Predict weather for 25°C temperature and 40% humidity
new_data = scaler.transform([[25, 40]])
prediction = nn_model.predict(new_data)
labels = {0: 'Sunny', 1: 'Cloudy', 2: 'Rainy'}
print(f"Predicted weather: {labels[prediction[0]]}")

Predicted weather: Sunny

Neural Network Training and Execution Cycles

The Impact of Automation

🎯 (SE-12-05)

📌 Assess the impact of automation on the individual, society and the environment

• safety of workers
• people with disability
• the nature and skills required for employment
• production efficiency, waste and the environment
• the economy and distribution of wealth

Safety of Workers

Automation and robotics are progressively replacing human workers in physically dangerous environments, with significant positive safety outcomes. Automated drones now perform inspection of high-voltage power lines and offshore oil rigs, eliminating worker exposure to electrocution, heights, and toxic atmospheres. Robotic welding and material handling systems in manufacturing have reduced repetitive strain injuries and burn risks. In Western Australian mining, autonomous haul trucks carry ore in remote pits without drivers, and Rio Tinto's AutoHaul network runs driverless trains across the Pilbara. Taking people out of the most dangerous tasks reduces their exposure to harm, but new risks appear, such as maintenance workers servicing machines that move on their own. The displacement of workers from these roles also creates economic vulnerability for the communities that depend on them.

People with Disability

AI-powered automation is providing unprecedented levels of independence and accessibility for people with disability. Computer vision systems enable real-time audio descriptions of the surrounding environment for visually impaired users (e.g., Microsoft's Seeing AI app describes people, currency, and written text through a smartphone camera). Speech-to-text and natural language interfaces allow people with motor disabilities to control computers and dictate documents without physical input. Autonomous vehicles could fundamentally transform mobility for people who cannot drive due to physical or visual impairment. AI captioning provides real-time speech-to-text conversion for deaf individuals during live conversations, video calls, and broadcasts. These advances promote social inclusion and reduce dependence on others, improving quality of life. The risk is that if these systems are not designed with universal design principles, the digital divide — where those without access to technology are further marginalised — may widen.

Nature and Skills Required for Employment

The adoption of automation is causing a significant structural shift in labour markets. Routine cognitive tasks (data entry, basic reporting, invoice processing) and routine physical tasks (assembly line work, packaging, basic quality inspection) are being automated at scale, displacing workers in these roles. Simultaneously, demand is growing rapidly for workers with skills in MLOps, data science, AI ethics, robotic maintenance, and human-AI collaboration — roles that require understanding of automation systems rather than performing the automated tasks themselves. Workers must engage in continuous reskilling and upskilling throughout their careers. Governments and educational institutions face the challenge of retraining displaced workers for emerging roles. The transition may exacerbate inequality if the benefits of automation accrue primarily to capital owners rather than being distributed across the workforce.

Production Efficiency, Waste, and the Environment

Automation offers significant environmental benefits through efficiency gains but also introduces new environmental costs. Optimised logistics and supply chains powered by ML reduce unnecessary transport and warehousing, cutting fuel consumption and emissions. Predictive maintenance extends the lifespan of industrial equipment, reducing material waste from premature replacements. Smart energy management systems use ML to optimise building heating, cooling, and lighting, reducing energy waste. Conversely, training large deep learning models, such as those behind chatbots, uses a great deal of electricity. Published estimates vary widely between models and studies, so treat any single figure with caution. Data centres that run AI services also need large amounts of electricity and often use water for cooling. Engineers must evaluate the full lifecycle environmental cost of AI systems alongside their operational benefits.

The Economy and Distribution of Wealth

Automation increases productivity and creates new industries and wealth. However, the distribution of that wealth is a critical societal concern. Historically, automation has created new types of employment alongside the jobs it displaces — but the transition period creates significant hardship for displaced workers. The current wave of AI-driven automation is occurring much faster than previous industrial revolutions, leaving less time for natural retraining. There is a risk of increasing economic polarisation: high-skill workers who can work alongside AI systems see wages rise, while low-skill workers face wage stagnation or displacement. Policy responses under discussion include universal basic income (UBI), robot taxes, and mandatory corporate investment in worker retraining programs.

Weighing the five areas

To assess an impact, set the benefits against the harms, say who carries each, and reach a judgement. The same technology can land differently on different people.

Australian context: ethics principles and law

The Australian Government published Australia's AI Ethics Principles in 2019. They are voluntary guidance for the people who design, build and use AI, not law. There are eight:

1. Human, societal and environmental wellbeing: AI should benefit people, society and the environment, and its impacts should be assessed.
2. Human-centred values: respect human rights, diversity and the autonomy of individuals.
3. Fairness: be inclusive and accessible, and avoid unfair discrimination.
4. Privacy protection and security: protect personal data and protect the system from attack.
5. Reliability and safety: work as intended, with testing and monitoring throughout the system's life.
6. Transparency and explainability: people should be able to tell when AI is affecting them and understand the outcome.
7. Contestability: there should be a way for people to challenge a significant decision.
8. Accountability: people remain responsible for AI systems and their outcomes at every stage.

Several laws and bodies apply too. Personal information used to train or run a system is covered by the Privacy Act 1988 and the Australian Privacy Principles, regulated by the OAIC. Anti-discrimination law applies whether a person or a program makes the decision. The NSW AI Assessment Framework is a mandatory risk self-assessment that NSW Government agencies must complete for AI projects.

The Australian case that best shows what goes wrong without these safeguards is Robodebt. From the mid-2010s a government scheme automatically raised debts against welfare recipients by averaging their income data, and left people to prove the debts were wrong. It used rule-based automation rather than machine learning, but it is the standard example of an automated decision made at scale with too little human checking and no easy way to contest it. The Royal Commission into the Robodebt Scheme reported in 2023 and found the scheme unlawful. Read it against the principles above: contestability, accountability and human-centred values were all missing.

How Human Behaviour Influences ML and AI

🎯 (SE-12-04, SE-12-05)

📌 Explore by implementation how patterns in human behaviour influence ML and AI software development

• psychological responses
• patterns related to acute stress response
• cultural protocols
• belief systems

ML systems are shaped by people at both ends. They learn from what people do (clicks, purchases, recordings), and they are used by people who are tired, stressed, trusting or sceptical. The syllabus asks you to explore by implementation, so each pattern below comes with a short Python program that shows how the pattern changes a design decision. All of them use only the standard library, and the output shown is real.

Psychological Responses

Human psychology profoundly shapes how AI systems must be designed. Users must understand when they are interacting with AI systems, and designers must balance automation with appropriate levels of human oversight and control. Two responses matter most. Automation bias is the tendency to trust an automated suggestion and stop checking it, so a confident but wrong output goes unchallenged. And people's habits become training data: a system that learns from clicks learns what people did, not necessarily what they would choose if they saw more.

The program below is a recommender that fills a news feed using each category's share of the person's past clicks. The simulated person clicks a sport story half the time it appears, science 40%, news 30% and music 20%. Run for 2000 rounds, once with no exploration and once with 20% of stories chosen at random.

import random

CATEGORIES = ["sport", "science", "news", "music"]
# How likely this person is to click a story from each category when it appears in the feed.
CLICK_CHANCE = {"sport": 0.50, "science": 0.40, "news": 0.30, "music": 0.20}


class Recommender:
    """Fills a feed using each category's share of the person's past clicks."""

    def __init__(self, explore=0.0, seed=1):
        self.clicks = {c: 1 for c in CATEGORIES}      # start with one click each, so all are equal
        self.explore = explore                        # chance of showing a random category instead
        self.rng = random.Random(seed)

    def pick_category(self):
        if self.rng.random() < self.explore:
            return self.rng.choice(CATEGORIES)
        point = self.rng.random() * sum(self.clicks.values())
        for category in CATEGORIES:
            point -= self.clicks[category]
            if point < 0:
                return category
        return CATEGORIES[-1]

    def record_click(self, category):
        self.clicks[category] += 1

    def share(self):
        total = sum(self.clicks.values())
        return {c: self.clicks[c] / total for c in CATEGORIES}


def simulate(explore, rounds=2000):
    recommender = Recommender(explore)
    person = random.Random(7)
    for _ in range(rounds):
        category = recommender.pick_category()               # the feed shows one story
        if person.random() < CLICK_CHANCE[category]:          # the person may click it
            recommender.record_click(category)
    return recommender.share()


for explore in (0.0, 0.2):
    shares = simulate(explore)
    print(f"explore = {explore}: " + ", ".join(f"{c} {shares[c]:.0%}" for c in CATEGORIES))

explore = 0.0: sport 34%, science 64%, news 1%, music 1%
explore = 0.2: sport 59%, science 21%, news 13%, music 7%

The person likes sport best, yet the click-driven feed ended up 64% science with almost no news or music. Early random clicks snowballed, because whatever is shown gets clicked, and whatever is clicked gets shown more. This is a feedback loop of the kind described in the bias section. With 20% exploration every category kept a place, and sport, the person's real favourite, came out on top. Designers respond to these psychological patterns with exploration, a visible "not interested" control, labels that say when content is machine-selected, and confidence indicators that discourage blind trust.

Patterns Related to Acute Stress Response

When individuals are operating under acute psychological stress (for example medical emergencies, financial crises or natural disasters), their cognitive performance changes significantly: attention narrows, working memory capacity decreases, and decision-making becomes impaired. AI interfaces designed for critical environments, such as intensive care monitoring systems, emergency dispatch platforms or incident management dashboards, must account for these changes. Systems should prioritise the most critical information, minimise cognitive load, and use clear visual communication to guide operators during stressful situations.

This class shows the idea as code. In normal use the panel lists every alert with its detail. When the system is told that an emergency is under way, it shows only the top three alerts, as short commands, and says how many it has hidden.

class AlertPanel:
    """Decides what an operator sees. Under stress, show fewer, shorter, clearer messages."""

    def __init__(self, stress_limit=3):
        self.stress_limit = stress_limit

    def render(self, alerts, high_stress):
        ranked = sorted(alerts, key=lambda a: a["priority"], reverse=True)   # most urgent first
        if not high_stress:
            return [f'P{a["priority"]}  {a["title"]}: {a["detail"]}' for a in ranked]
        top = ranked[:self.stress_limit]                                   # narrow attention: top few only
        lines = [f'{a["title"].upper()}: {a["action"]}' for a in top]
        hidden = len(ranked) - len(top)
        if hidden:
            lines.append(f"({hidden} lower-priority alerts hidden. Press M for more.)")
        return lines


alerts = [
    {"priority": 2, "title": "Filter change due", "detail": "Ward 4 air filter is 80% used", "action": "Book maintenance"},
    {"priority": 9, "title": "Oxygen low", "detail": "Bed 12 saturation 86% for 3 minutes", "action": "Check patient in bed 12"},
    {"priority": 5, "title": "Battery low", "detail": "Infusion pump 7 at 15%", "action": "Plug in pump 7"},
    {"priority": 7, "title": "Heart rate high", "detail": "Bed 3 at 138 bpm", "action": "Check patient in bed 3"},
    {"priority": 1, "title": "Software update", "detail": "Monitor firmware 2.4 available", "action": "Schedule update"},
]

panel = AlertPanel()
print("--- Normal shift ---")
print("\n".join(panel.render(alerts, high_stress=False)))
print("--- Emergency in progress ---")
print("\n".join(panel.render(alerts, high_stress=True)))

--- Normal shift ---
P9  Oxygen low: Bed 12 saturation 86% for 3 minutes
P7  Heart rate high: Bed 3 at 138 bpm
P5  Battery low: Infusion pump 7 at 15%
P2  Filter change due: Ward 4 air filter is 80% used
P1  Software update: Monitor firmware 2.4 available
--- Emergency in progress ---
OXYGEN LOW: Check patient in bed 12
HEART RATE HIGH: Check patient in bed 3
BATTERY LOW: Plug in pump 7
(2 lower-priority alerts hidden. Press M for more.)

The design choices are all visible in the output: alerts ranked by priority, fewer items, imperative wording ("Check patient in bed 12" instead of a sentence of readings), and a way to see the rest. Nothing is deleted, so a careful operator can still find the low-priority items later.

Cultural Protocols

ML and AI systems deployed across communities must be engineered to respect diverse cultural norms, communication expectations and social protocols. A conversational AI that is appropriate in one cultural context may be offensive or ineffective in another. Engineers must consider varying expectations around formality, language and culturally sensitive topics. Systems must also comply with the privacy and data protection laws of each jurisdiction. Ignoring cultural protocols leads to reputational damage, legal liability and loss of trust.

An Australian example: in some Aboriginal and Torres Strait Islander communities it is culturally sensitive to see the name or image of a person who has died, and broadcasters and archives commonly add a warning before showing such material. A photo-memories feature that learns which pictures people engage with could easily surface exactly that content. The program below adds a protocol setting that the family or community chooses.

# Cultural protocols: a photo-memories feature that respects a setting chosen by the family or community.
def memory_card(photo, protocols):
    if "avoid_images_of_deceased" in protocols and photo["person_deceased"]:
        return f'Content warning: this photo shows {photo["person"]}, who has died. Show anyway?'
    return f'On this day: {photo["caption"]}'

photo = {"person": "Grandpa Lee", "person_deceased": True, "caption": "Fishing at the lake"}
print(memory_card(photo, protocols=set()))
print(memory_card(photo, protocols={"avoid_images_of_deceased"}))

On this day: Fishing at the lake
Content warning: this photo shows Grandpa Lee, who has died. Show anyway?

The important design point is who decides. The protocol comes from the people concerned, not from the developer's assumptions, and the safe default is to warn first and let the person choose.

Belief Systems

Human belief systems, including religious beliefs, moral frameworks and ethical worldviews, influence both how AI is perceived and how it must be designed. In some religious and cultural traditions there are deep concerns about AI making life-or-death decisions (for example autonomous weapons or medical triage) without human moral accountability. Engineers must engage with these perspectives during design rather than treat them as obstacles. AI systems for healthcare, justice and education should offer explainability, the ability to give human-understandable reasons for a decision, so people can challenge outcomes that conflict with their values. Training data drawn from populations with particular belief systems will also encode those beliefs, so models need testing for fair and respectful outputs across belief communities.

Many beliefs are not preferences to be predicted but rules to be obeyed, such as halal, kosher or vegetarian eating. In this meal recommender, the model's predicted score ranks the dishes, but the diner's declared requirement is applied first as a hard filter that no score can override.

# Belief systems: hard rules that a learned model must not override.
menu = [
    {"dish": "Pork ribs",      "tags": {"pork"},               "predicted_score": 0.94},
    {"dish": "Lamb kofta",     "tags": {"halal"},              "predicted_score": 0.88},
    {"dish": "Chickpea curry", "tags": {"vegetarian", "halal"}, "predicted_score": 0.81},
    {"dish": "Beef burger",    "tags": set(),                  "predicted_score": 0.79},
]

def suggest(menu, must_have=(), must_avoid=()):
    allowed = [item for item in menu
               if set(must_have) <= item["tags"] and not (set(must_avoid) & item["tags"])]
    ranked = sorted(allowed, key=lambda item: item["predicted_score"], reverse=True)
    return [item["dish"] for item in ranked]

print("Model's raw ranking:   ", suggest(menu))
print("Diner needs halal food:", suggest(menu, must_have=["halal"]))
print("Diner is vegetarian:   ", suggest(menu, must_have=["vegetarian"]))

Model's raw ranking:    ['Pork ribs', 'Lamb kofta', 'Chickpea curry', 'Beef burger']
Diner needs halal food: ['Lamb kofta', 'Chickpea curry']
Diner is vegetarian:    ['Chickpea curry']

The model's favourite dish, pork ribs, disappears for a diner who needs halal food. Notice that the belief is declared by the person. A system should not try to infer someone's religion from what they eat or watch: under the Privacy Act 1988, information about religious beliefs is "sensitive information", which needs the person's consent to collect.

Human and Dataset Source Bias

🎯 (SE-12-04, SE-12-05)

📌 Investigate the effect of human and dataset source bias in the development of ML and AI solutions

A model learns whatever pattern is in its data, whether or not that pattern is fair. Bias in ML means systematically worse outcomes for some people or groups. It can enter through the people who build and use a system (human bias) and through the data the system learns from (dataset source bias). It rarely announces itself: overall accuracy can look excellent while one group is badly served.

Sources of Bias

Bias in ML systems can arise from multiple sources. Training data that reflects historical prejudices will cause AI systems to perpetuate those discriminatory patterns. When training datasets do not accurately represent the real-world population where the system will be used, the model produces systematically unfair outcomes for underrepresented groups. Developers may unconsciously select or weight data in ways that confirm their pre-existing assumptions. And when a biased model's predictions influence the collection of future data, it creates self-reinforcing cycles that entrench the original bias even further.

A worked example: a loan model that copies unfair history

The program below builds 4000 synthetic applicants (invented data). Everyone with an income of $50,000 or more would repay a loan, in both group A and group B. But in the historical records, human loan officers approved 95% of the good applicants from group A and only 40% of the good applicants from group B. Group is not given to the models. Postcode is, and 90% of group A live in the north while 90% of group B live in the south, so postcode leaks the group. The fair answer is that every applicant who would repay gets a loan.

• Model 1 learns from the past decisions, using postcode and income band.
• Model 2 learns from what really happened, whether the person repays, using income band only.

import random

rng = random.Random(42)


def make_applicant():
    group = "A" if rng.random() < 0.5 else "B"
    # Postcode is a proxy: 90% of group A live in the north and 90% of group B in the south.
    home = "north" if group == "A" else "south"
    other = "south" if home == "north" else "north"
    postcode = home if rng.random() < 0.9 else other
    income = rng.randint(20, 100)                     # thousands of dollars a year
    repays = income >= 50                             # what would really happen if a loan was made
    # HISTORY: human officers approved 95% of good applicants from A but only 40% from B.
    if repays:
        past_decision = rng.random() < (0.95 if group == "A" else 0.40)
    else:
        past_decision = False
    return {"group": group, "postcode": postcode, "income": income,
            "repays": repays, "past_decision": past_decision}


applicants = [make_applicant() for _ in range(4000)]


def train(label_of, features):
    """Learn the share of applicants with each combination of features who got label 1."""
    counts = {}
    for a in applicants:
        positive, total = counts.get(features(a), (0, 0))
        counts[features(a)] = (positive + label_of(a), total + 1)
    return {key: positive / total for key, (positive, total) in counts.items()}


by_postcode = lambda a: (a["postcode"], a["income"] // 10)    # the group is NOT a feature
by_income = lambda a: a["income"] // 10

table1 = train(lambda a: a["past_decision"], by_postcode)     # Model 1: copy past decisions
table2 = train(lambda a: a["repays"], by_income)              # Model 2: learn who repays


def report(name, decides):
    rates = []
    for group in ("A", "B"):
        good = [a for a in applicants if a["repays"] and a["group"] == group]
        rates.append(sum(1 for a in good if decides(a)) / len(good))
    print(f"{name:<38} A {rates[0]:4.0%}   B {rates[1]:4.0%}   gap {abs(rates[0] - rates[1]) * 100:2.0f} points")


print("Share of applicants who WOULD repay that get a loan (fair = same for A and B):")
report("Past human decisions", lambda a: a["past_decision"])
report("Model 1: trained on past decisions", lambda a: table1[by_postcode(a)] >= 0.5)
report("Model 2: trained on repayment", lambda a: table2[by_income(a)] >= 0.5)

Share of applicants who WOULD repay that get a loan (fair = same for A and B):
Past human decisions                   A  95%   B  42%   gap 53 points
Model 1: trained on past decisions     A  92%   B  31%   gap 61 points
Model 2: trained on repayment          A 100%   B 100%   gap  0 points

Read the three rows in turn. The officers' history already has a 53-point gap. Model 1 never saw the group column, yet its gap is 61 points, even wider, because postcode is a proxy and the model turns a tendency into a rule: a whole southern postcode band is now declined. This is why "we removed the sensitive column" is not enough. Model 2 has no gap, because its label measures the real outcome rather than someone's past choice. One caution: in real life repayment is known only for people who were approved (selection bias), so building Model 2 is much harder than this simulation suggests.

The measure used here asks: among people who would repay, does each group have the same chance of approval? Fairness can be defined in several ways, and they cannot all be satisfied at once, so engineers must choose a definition, state it and test against it.

Engineering Practices to Mitigate Bias

Australian context

Bias is a legal as well as an engineering problem. Australia's anti-discrimination laws, such as the Racial Discrimination Act 1975 and the Sex Discrimination Act 1984, protect people from unfair treatment, and an organisation cannot escape responsibility by saying "the algorithm decided". Fairness and contestability are also two of the eight Australian Government AI Ethics Principles described in the impact section above, and the NSW AI Assessment Framework requires NSW Government agencies to assess and mitigate risks such as bias before an AI project goes ahead.