Lesson 2: Neural Networks Fundamentals

What is a Neural Network?

A Neural Network is a computational model inspired by the way biological neural networks in animal brains process information. Just like how neurons in your brain connect and communicate to help you think, artificial neural networks consist of interconnected nodes (artificial neurons) that work together to solve complex problems.

Think of a neural network as a team of specialists, where each member (neuron) receives information, processes it, and passes the result to others. Through this collaboration, the network can recognize patterns, make decisions, and learn from experience.

Basic Neural Network Structure

x₁

Input 1

Neuron

Output

Data flows from inputs → through neurons → to outputs

The Anatomy of a Neuron

Key Components of an Artificial Neuron

📥 Inputs (x₁, x₂, x₃...)

The data or signals that the neuron receives. Each input represents a feature or piece of information.

⚖️ Weights (w₁, w₂, w₃...)

Numbers that determine the importance of each input. Higher weights mean more influence on the output.

➕ Bias (b)

An additional parameter that helps the neuron make better decisions by shifting the activation threshold.

🔢 Summation Function

Combines all weighted inputs plus bias into a single value (weighted sum).

🎯 Activation Function

Determines whether the neuron should "fire" (activate) based on the weighted sum.

📤 Output

The final result that gets passed to the next layer or becomes the network's prediction.

The Mathematical Process

Here's how a neuron processes information mathematically:

Step 1: Weighted Sum

$$z = w_1x_1 + w_2x_2 + w_3x_3 + ... + w_nx_n + b$$

Step 2: Activation

$$output = f(z)$$

where f() is the activation function

A Simple Example

Let's say we have a neuron deciding whether to recommend a movie:

Input 1 (x₁): Movie rating = 8.5, Weight (w₁) = 0.6
Input 2 (x₂): Genre match = 1, Weight (w₂) = 0.4
Bias (b): -3

Calculation:

z = (8.5 × 0.6) + (1 × 0.4) + (-3) = 5.1 + 0.4 - 3 = 2.5

If activation function returns 1 when z > 0: Output = 1 (Recommend!)

Activation Functions

Activation functions determine how a neuron responds to its inputs. Here are the most common ones:

Step Function

f(x) = 1 if x ≥ 0, else 0

Simple on/off switch. Output is either 0 or 1.

Sigmoid

f(x) = 1/(1 + e^(-x))

Smooth curve from 0 to 1. Great for probabilities.

ReLU

f(x) = max(0, x)

Most popular! Output is 0 for negative, x for positive.

Tanh

f(x) = (e^x - e^(-x))/(e^x + e^(-x))

Similar to sigmoid but outputs from -1 to 1.

Types of Neural Networks

1. Single Layer Perceptron

The simplest form with just input and output layers. Can only solve linearly separable problems (like AND, OR logic gates).

2. Multi-Layer Perceptron (MLP)

Has one or more hidden layers between input and output. Can solve complex, non-linear problems like image recognition.

3. Deep Neural Networks

Networks with many hidden layers (typically 3+). The "deep" in "deep learning" refers to these deep architectures.

Why Layers Matter

🎯 Input Layer

Receives raw data (pixels, text, numbers). No processing happens here.

🧠 Hidden Layers

Where the "magic" happens. Each layer learns increasingly complex patterns.

📊 Output Layer

Produces final predictions or classifications based on learned patterns.

Learning Process: Weights and Biases

Initially, weights and biases are set randomly. During training, the network:

Makes predictions with current weights
Compares predictions to correct answers
Calculates errors and adjusts weights
Repeats until predictions improve

This process is like learning to play basketball - you start with random shots, see where you miss, and gradually adjust your aim until you're hitting the target consistently.

Interactive MNIST Digit Recognition

See how a neuron learns to recognize handwritten digits! This neuron is trained to detect the digit "8":

🔢 MNIST Digit "8" Detector

Pixel Intensities

Top curve pixels: 0.9 Dark pixels forming top loop of "8"

Middle crossing: 0.8 Dark pixels in middle intersection

Bottom curve pixels: 0.7 Dark pixels forming bottom loop of "8"

Learned Weights

Top loop weight: 0.6 How important top curve is for "8"

Middle weight: 0.8 How important middle crossing is

Bottom loop weight: 0.5 How important bottom curve is

Detection Threshold

Bias (b): -0.3 How confident neuron needs to be

Digit Recognition Process

Feature Detection:
z = 0.9×0.6 + 0.8×0.8 + 0.7×0.5 + (-0.3) = 1.08

Confidence Score:
confidence = 1/(1 + e^(-z)) = 0.746

Is this digit "8"? 74.6% confident

Prediction: Likely an "8"

Real-World Applications

Neural networks power many technologies you use daily:

Image Recognition: Photo tagging on social media
Speech Recognition: Siri, Alexa, Google Assistant
Language Translation: Google Translate
Recommendation Systems: Netflix, Spotify, Amazon
Medical Diagnosis: Detecting diseases in X-rays
Autonomous Vehicles: Object detection and decision making

Knowledge Check

Test your understanding of neural network fundamentals

1. What is the purpose of weights in a neural network?

A) To store the final output

B) To determine the importance of each input

C) To activate the neuron

D) To provide bias to the network

2. Which activation function is most commonly used in modern deep learning?

A) Step Function

B) Sigmoid

C) ReLU

D) Linear

3. What does the bias term do in a neuron?

A) Multiplies the inputs

B) Shifts the activation threshold

C) Determines the learning rate

D) Normalizes the output

4. In a multi-layer perceptron, where does the actual "learning" happen?

A) Input layer only

B) Output layer only

C) Hidden layers

D) Activation functions

5. What is the mathematical operation performed in the summation function?

A) Multiply all inputs

B) Find the maximum input

C) Calculate weighted sum plus bias

D) Average all inputs

6. What makes a neural network "deep"?

A) Large number of inputs

B) Many hidden layers

C) Complex activation functions

D) High computational power

7. Which layer receives the raw input data?

A) Hidden layer

B) Output layer

C) Input layer

D) Activation layer

8. The ReLU activation function outputs:

A) Values between 0 and 1

B) Values between -1 and 1

C) Either 0 or 1

D) 0 for negative inputs, x for positive inputs

Quiz Complete!

0/8

Great job! You're mastering neural networks.