Why so exciting?

• Aside from having a name that sounds like it came straight out of Neuromancer, why is everyone so excited about Neural Networks?

Why so exciting?

• Aside from having a name that sounds like it came straight out of Neuromancer, why is everyone so excited about Neural Networks?

• Imagine trying to map a complex pattern to some outcome. Maybe you're trying to recognize whether an image is a dog, or a blueberry muffin.

Why so exciting?

• How do we find the dog/muffin functional form?

Why so exciting?

• How do we find the dog/muffin functional form? What kind of expert dog-ology field can we turn to in order to solve our problem?

Why so exciting?

• How do we find the dog/muffin functional form? What kind of expert dog-ology field can we turn to in order to solve our problem?

• However, you can recognize which is which, right?

Why so exciting?

• How do we find the dog/muffin functional form? What kind of expert dog-ology field can we turn to in order to solve our problem?

• However, you can recognize which is which, right?

Have you taken a course in dog-ology?

Why so exciting?

You've seen dogs, you've seen muffins, and somehow your brain has found an unknown way to tell the difference. What about letters/numbers?

Why so exciting?

You've seen dogs, you've seen muffins, and somehow your brain has found an unknown way to tell the difference. What about letters/numbers?

Why so exciting?

You've seen dogs, you've seen muffins, and somehow your brain has found an unknown way to tell the difference. What about letters/numbers?

That's what neural networks are trying to imitate - that process of taking one set of sensory inputs, and through repetition, finding patterns that map to some internal set of labels.

The building block: perceptron

Where did the first 'neural network' come from

• Neural networks (in their most basic form) are actually one of the oldest (if not THE oldest) machine learning tool still in use today.

The building block: perceptron

Where did the first 'neural network' come from

• Neural networks (in their most basic form) are actually one of the oldest (if not THE oldest) machine learning tool still in use today.

• Invented by McCulloch and Pitts in 1943*, the perceptron (the most fundamental element of a neural network) conceptually predate Alan Turing's machine.

The building block: perceptron

Where did the first 'neural network' come from

• Neural networks (in their most basic form) are actually one of the oldest (if not THE oldest) machine learning tool still in use today.

• Invented by McCulloch and Pitts in 1943*, the perceptron (the most fundamental element of a neural network) conceptually predate Alan Turing's machine.

You might ask yourself, why would anyone bother with designing an algorithm that takes hours of compute time on a high-end machine TODAY when at the time, they lacked even a rudimentary computer?

The building block: perceptron

In order to understand how the intuition for how a perceptron works mathematically, having a basic understanding of the biological mechanics of a neuron is useful.

The building block: perceptron

Ok, so how do we model this process using only the mathematical tools we have available?

The building block: perceptron

Ok, so how do we model this process using only the mathematical tools we have available?

• We need a function that takes a consistent set of inputs and maps them efficiently to some (potentially unknown) outputs.

The building block: perceptron

Ok, so how do we model this process using only the mathematical tools we have available?

• We need a function that takes a consistent set of inputs and maps them efficiently to some (potentially unknown) outputs.

• Let's start by simplifying the problem, and only modeling 'activated/not activated' once some threshold of charge is reached.

Perceptron, formally

Put another way:

Perceptron, formally

Put another way:

We're going to:

Perceptron, formally

Put another way:

We're going to:

• take Xs (inputs), multiply each $x_i$ by a weight $w_i$*, and then scale that value from 0-1, using a step function where $\sum_i w_i = \Theta$

Perceptron, formally

Put another way:

We're going to:

• take Xs (inputs), multiply each $x_i$ by a weight $w_i$*, and then scale that value from 0-1, using a step function where $\sum_i w_i = \Theta$

Perceptron, formally

Perceptron, formally

Two things to notice:

Perceptron, formally

Two things to notice:

• we now have an activation function that replaces $\Theta$.

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

• The answer:

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

• The answer: we guess. We start with some random set of weights.

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

• The answer: we guess. We start with some random set of weights.

• Then we update, using information from our accuracy to infer how well our weights are informing those guesses. Eventually, we'll get something meaningful

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

• The answer: we guess. We start with some random set of weights.

• Then we update, using information from our accuracy to infer how well our weights are informing those guesses. Eventually, we'll get something meaningful

• This is why the change from the original step-wise function is so important:

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

• The answer: we guess. We start with some random set of weights.

• Then we update, using information from our accuracy to infer how well our weights are informing those guesses. Eventually, we'll get something meaningful

• This is why the change from the original step-wise function is so important:

The linear activation function buys us something that the simple step-wise function could not:

Perceptron, how do they work?

• You might have wondered: how in the world do we solve this thing?

• The answer: we guess. We start with some random set of weights.

• Then we update, using information from our accuracy to infer how well our weights are informing those guesses. Eventually, we'll get something meaningful

• This is why the change from the original step-wise function is so important:

The linear activation function buys us something that the simple step-wise function could not: differentiability.

Perceptron, how do they work?

Lucky for us, we can define $\frac{1}{2}*SSE$ as our loss function, and then $\frac{\partial{L}}{\partial{w^t_i}}$ is fairly easy.

for each training sample... in each iteration

Perceptron, how do they work?

Lucky for us, we can define $\frac{1}{2}*SSE$ as our loss function, and then $\frac{\partial{L}}{\partial{w^t_i}}$ is fairly easy.

for each training sample... in each iteration

1.) $z = w^t_0 + w^t_1x_1 \dots w^t_nx_n = (W^t)^Tx.$

$w^t_0\ \text{is a scalar (bias). You will sometimes see this as}\ b^t$

Perceptron, how do they work?

Lucky for us, we can define $\frac{1}{2}*SSE$ as our loss function, and then $\frac{\partial{L}}{\partial{w^t_i}}$ is fairly easy.

for each training sample... in each iteration

1.) $z = w^t_0 + w^t_1x_1 \dots w^t_nx_n = (W^t)^Tx.$

$w^t_0\ \text{is a scalar (bias). You will sometimes see this as}\ b^t$

2.) $\phi(z^{(i)}) = \widehat{y}^{(i)}$

Perceptron, how do they work?

Lucky for us, we can define $\frac{1}{2}*SSE$ as our loss function, and then $\frac{\partial{L}}{\partial{w^t_i}}$ is fairly easy.

for each training sample... in each iteration

1.) $z = w^t_0 + w^t_1x_1 \dots w^t_nx_n = (W^t)^Tx.$

$w^t_0\ \text{is a scalar (bias). You will sometimes see this as}\ b^t$

2.) $\phi(z^{(i)}) = \widehat{y}^{(i)}$

3.) $\eta = \text{learning rate}$

Perceptron, how do they work?

Lucky for us, we can define $\frac{1}{2}*SSE$ as our loss function, and then $\frac{\partial{L}}{\partial{w^t_i}}$ is fairly easy.

for each training sample... in each iteration

1.) $z = w^t_0 + w^t_1x_1 \dots w^t_nx_n = (W^t)^Tx.$

$w^t_0\ \text{is a scalar (bias). You will sometimes see this as}\ b^t$

2.) $\phi(z^{(i)}) = \widehat{y}^{(i)}$

3.) $\eta = \text{learning rate}$

Loss function: $J(w_i^t) =\frac{1}{2}\sum_i(y^{(i)}-\phi(z)^{(i)})^2$

Perceptron, how do they work?

Let's see this in action:

Perceptron, how do they work?

You'll notice that the linear 'activation function' produces a linear decision boundary.

Perceptron, how do they work?

You'll notice that the linear 'activation function' produces a linear decision boundary.

• So what's the big deal? We could do that, and more, with SVMs.

Perceptron, how do they work?

You'll notice that the linear 'activation function' produces a linear decision boundary.

• So what's the big deal? We could do that, and more, with SVMs.

• Unlike with SVM, perceptrons improve over time.

Perceptron, how do they work?

You'll notice that the linear 'activation function' produces a linear decision boundary.

• So what's the big deal? We could do that, and more, with SVMs.

• Unlike with SVM, perceptrons improve over time. Because a perceptron can hypothetically continue to improve so long as it keeps seeing any loss, the only upper limit to a perceptron's performance is how flexible the activation function is.

Perceptron, how do they work?

You'll notice that the linear 'activation function' produces a linear decision boundary.

• So what's the big deal? We could do that, and more, with SVMs.

• Unlike with SVM, perceptrons improve over time. Because a perceptron can hypothetically continue to improve so long as it keeps seeing any loss, the only upper limit to a perceptron's performance is how flexible the activation function is.

But there's more than that: because we're using derivatives to update the weights, we can link a bunch of neurons together and use the chain rule to optimize the model weights $W$ as they work together to find the pattern that maps $X \to Y$.

Perceptron, how do they work?

You'll notice that the linear 'activation function' produces a linear decision boundary.

• So what's the big deal? We could do that, and more, with SVMs.

• Unlike with SVM, perceptrons improve over time. Because a perceptron can hypothetically continue to improve so long as it keeps seeing any loss, the only upper limit to a perceptron's performance is how flexible the activation function is.

But there's more than that: because we're using derivatives to update the weights, we can link a bunch of neurons together and use the chain rule to optimize the model weights $W$ as they work together to find the pattern that maps $X \to Y$.

Well what do you get when you glue a bunch of neurons together?

Neural Networks

By breaking down a complex mapping task into a series of steps, we can use large collections of modified perceptrons to universally approximate any functional form.

Neural Networks

By breaking down a complex mapping task into a series of steps, we can use large collections of modified perceptrons to universally approximate any functional form.

• You already know how to do this*, you can extend the model yourself!

Neural Networks

By breaking down a complex mapping task into a series of steps, we can use large collections of modified perceptrons to universally approximate any functional form.

• You already know how to do this*, you can extend the model yourself!

• Imagine making several perceptrons that each learn patterns in parallel that simultaneously are trying to minimize your loss function.

Layer?

Layer seems to imply we could have multiple sets of perceptrons?

Layer?

Layer seems to imply we could have multiple sets of perceptrons?

• We can!

Layer?

Layer seems to imply we could have multiple sets of perceptrons?

• We can!

All we do is make the $y_0,\ y_1,\ y_2$ feed into a new set of perceptrons and have those new perceptrons find the patterns in the output of our old ones.

Layer?

Layer seems to imply we could have multiple sets of perceptrons?

• We can!

All we do is make the $y_0,\ y_1,\ y_2$ feed into a new set of perceptrons and have those new perceptrons find the patterns in the output of our old ones.

that makes the output a little more complex, because now $\widehat{y}$ is equal to $\sigma_2(\sigma_1(X))$

Backpropagation

How do we update our weights using this new function? The main difference is that we need a way of creating an error for EVERY layer. This requires...

Backpropagation

How do we update our weights using this new function? The main difference is that we need a way of creating an error for EVERY layer. This requires...

• the chain rule (the scariest of rules)

$$F'(g(x)) = f'(g(x))g'(x)$$

Backpropagation

How do we update our weights using this new function? The main difference is that we need a way of creating an error for EVERY layer. This requires...

• the chain rule (the scariest of rules)

$$F'(g(x)) = f'(g(x))g'(x)$$

For the output layer, we can define our error $(\delta^L_j)$ by

$$\delta^L_j = \frac{\partial{C}}{\partial{a^L_j}}\sigma'(z^L_j)$$

Backpropagation

$$\delta^L_j = \frac{\partial{C}}{\partial{a^L_j}}\sigma'(z^L_j)$$

$\frac{\partial{C}}{\partial{a^L_j}} = (a^L_j - y_j)$ (for our cost function)

Backpropagation

Now all that's left is to find the derivatives for the other layer errors - however, those are going to be functions of the output error- let's look at some arbitrary layer $l \in \{1,\dots, L\}$ where $l$ is the layer before $l + 1$

$$\delta^l = ((w^{l+1})^T\delta^{l+1}) \circ \sigma'(z^l)$$

Where $w^{l+1} \equiv \text{weight matrix for layer }l+1$

Backpropagation

Now all that's left is to find the derivatives for the other layer errors - however, those are going to be functions of the output error- let's look at some arbitrary layer $l \in \{1,\dots, L\}$ where $l$ is the layer before $l + 1$

$$\delta^l = ((w^{l+1})^T\delta^{l+1}) \circ \sigma'(z^l)$$

Where $w^{l+1} \equiv \text{weight matrix for layer }l+1$

Now, we can use what we found to calculate $\delta^L$, which can be plugged in to find $\delta^{L-1}$ and so on, until we reach the first layer.

Backpropagation

In order to update the weights, we really need to know how our costs are changing as a result of our chosen weights. This is actually super simple to do given what we already know:

• $\frac{\partial{C}}{\partial{w^l_{j,k}}} = a^{l-1}_k\delta^l_j$

Or,

Backpropagation

In order to update the weights, we really need to know how our costs are changing as a result of our chosen weights. This is actually super simple to do given what we already know:

• $\frac{\partial{C}}{\partial{w^l_{j,k}}} = a^{l-1}_k\delta^l_j$

Or,

• $\frac{\partial{C}}{\partial{w^l}} = a_{input}\delta^l_{output}$

Where $a_{input}$ is the input to the weight for layer $l$ and $\delta_{output}$ is the error of the output from the weight $w^l$

Backpropagation

In order to update the weights, we really need to know how our costs are changing as a result of our chosen weights. This is actually super simple to do given what we already know:

• $\frac{\partial{C}}{\partial{w^l_{j,k}}} = a^{l-1}_k\delta^l_j$

Or,

• $\frac{\partial{C}}{\partial{w^l}} = a_{input}\delta^l_{output}$

Where $a_{input}$ is the input to the weight for layer $l$ and $\delta_{output}$ is the error of the output from the weight $w^l$

Thus, the MSE for the guess $\widehat{y}$ of $y$ will trickle through the weights and update them as the algorithm learns.

Your errors

You can think of this process as finding the bottom of a bowl by rolling a ball (with no momentum) down the side of it. The method we used before is called gradient descent, but there are others (we'll see those later.)

Your errors

You can think of this process as finding the bottom of a bowl by rolling a ball (with no momentum) down the side of it. The method we used before is called gradient descent, but there are others (we'll see those later.)

As the ball gets close to the bottom of the bowl, it will slow down and eventually stop at the lowest point.

Other Elements

Unlike other machine learning models we've seen so far, Neural Networks learn over what are called epochs.

Other Elements

Unlike other machine learning models we've seen so far, Neural Networks learn over what are called epochs.

• dfn Epoch: One full pass of the model over your training data set (updating the weights for each pass)

Some pitfalls of neural networks

• They are extremely flexible - and as we know already, that means they are prone to overfitting. Even more so than the algorithms you've seen so far.

Some pitfalls of neural networks

• this makes cross-validation much more difficult. They are much more reliant on either guessing and checking (bad) or having experience with them (better).

Some pitfalls of neural networks

• this makes cross-validation much more difficult. They are much more reliant on either guessing and checking (bad) or having experience with them (better).

• Further, because they climb to an optimum, there's no guarantee that the solution you find is the best one you could have found for your data. It's highly dependent on how you're updating your weights (optimization method) and where you started.

Some pitfalls of neural networks

• this makes cross-validation much more difficult. They are much more reliant on either guessing and checking (bad) or having experience with them (better).

• Further, because they climb to an optimum, there's no guarantee that the solution you find is the best one you could have found for your data. It's highly dependent on how you're updating your weights (optimization method) and where you started.

How we've updated our weights so far has used something called gradient descent.

Common Optimization Techniques

An example, Stochastic Gradient Descent:

Rather than calculate the error across all points in the sample, only calculate the error for a single point at a time to speed up the process and avoid overfit/getting stuck at the same time.

Let's see how some of these weird functions act next to gradient descent red ball.

Programming a Neural Network

It is perfectly possible to code a neural network by hand, but by far the most common tool used to write neural networks for production is to use Tensorflow and keras.

Programming a Neural Network

It is perfectly possible to code a neural network by hand, but by far the most common tool used to write neural networks for production is to use Tensorflow and keras.

The main downside of these is that they're both written under the hood in python, which means it may take some wrangling to get tensorflow for Rstudio to work on your computer.

Programming a Neural Network

It is perfectly possible to code a neural network by hand, but by far the most common tool used to write neural networks for production is to use Tensorflow and keras.

The main downside of these is that they're both written under the hood in python, which means it may take some wrangling to get tensorflow for Rstudio to work on your computer.

Both of these work together and use a strange representation of data called a 'tensor'.

Programming a Neural Network

It is perfectly possible to code a neural network by hand, but by far the most common tool used to write neural networks for production is to use Tensorflow and keras.

The main downside of these is that they're both written under the hood in python, which means it may take some wrangling to get tensorflow for Rstudio to work on your computer.

Both of these work together and use a strange representation of data called a 'tensor'. I don't have enough time to explain what a tensor is, but luckily this adorable human does a much better job than I ever could: tensors

Tensorflow & Keras

Like in tidymodels, tensorflow starts with a model-type object.

Tensorflow & Keras

Like in tidymodels, tensorflow starts with a model-type object. For our purposes, this is keras_model_sequential().

Tensorflow & Keras

Like in tidymodels, tensorflow starts with a model-type object. For our purposes, this is keras_model_sequential().

This tells tensorflow we're going to build our neural network sequentially.

MNIST dataset

First, let's remind ourselves what our neural network is trying to do:

MNIST dataset

First, let's remind ourselves what our neural network is trying to do:

• We want our neural network to read handwritten numbers and tell us what number they are supposed to represent.

MNIST

Ok, that's great, but remember we need to set up this problem so we can turn a picture into an outcome (the numbers 0-9). How do we do it?

MNIST

Ok, that's great, but remember we need to set up this problem so we can turn a picture into an outcome (the numbers 0-9). How do we do it?

Well, we can squish the data so that each of these matrices in just a very long row of Xs. (28*28 = 784 different variables.)

Tensorflow & Keras

Ok- let's do this in order.

model <- keras_model_sequential() %>%
  layer_flatten(input_shape = c(28,28))

Tensorflow & Keras

Ok- let's do this in order.

model <- keras_model_sequential() %>%
  layer_flatten(input_shape = c(28,28)) %>%
  layer_dense(units = 128, activation = "relu")

Tensorflow & Keras

Ok- let's do this in order.

model <- keras_model_sequential() %>%
  layer_flatten(input_shape = c(28,28)) %>%
  layer_dense(units = 128, activation = "relu")

This is our first layer in our Neural network! It creates 128 different 'neurons' (or perceptrons) and sets their activation function to be a relu.

Tensorflow & Keras

Ok- let's do this in order.

model <- keras_model_sequential() %>%
  layer_flatten(input_shape = c(28,28)) %>%
  layer_dense(units = 128, activation = "relu", name= 'hiddenlayer') %>%
  layer_dense(10, activation = "softmax", name = 'outputlayer')

Tensorflow & Keras

Ok- let's do this in order.

model <- keras_model_sequential() %>%
  layer_flatten(input_shape = c(28,28)) %>%
  layer_dense(units = 128, activation = "relu", name= 'hiddenlayer') %>%
  layer_dense(10, activation = "softmax", name = 'outputlayer')

This is our output layer, and the activation it's using is simply - "classify this into one of 10 (number of nodes) classes, and guess the one with the largest probability."

Tensorflow & Keras

you can see how many parameters they have and check - it should be $784\text{(28pix*28pix)}*128\text{(number of hidden neurons)} + 128(w_0 \ or \ b) = 100,480$

Tensorflow & Keras

you can see how many parameters they have and check - it should be $784\text{(28pix*28pix)}*128\text{(number of hidden neurons)} + 128(w_0 \ or \ b) = 100,480$

That is what's called a:

Tensorflow & Keras

Now we need to prepare the model.

Tensorflow & Keras

Now we need to prepare the model.

This is done with a compile command. We need to give that command a loss function to use, an optimizer (we'll use Adam which I described earlier) and any metrics we're interested in (let's look at accuracy.)

Tensorflow

And that's how you run a neural network! Ours is getting a ~ 92% accuracy rate classifying pictures into one of 10 different categories. If you run this for 40 epochs, you'll get ~ 95% accuracy.

Tensorflow

And that's how you run a neural network! Ours is getting a ~ 92% accuracy rate classifying pictures into one of 10 different categories. If you run this for 40 epochs, you'll get ~ 95% accuracy.

It's super easy.

Tensorflow

And that's how you run a neural network! Ours is getting a ~ 92% accuracy rate classifying pictures into one of 10 different categories. If you run this for 40 epochs, you'll get ~ 95% accuracy.

It's super easy.

It's too easy.

Other types of networks

There are TONS of other cool structures, but they all work exactly like these under the hood - they just add fancy math that allows us to process our inputs in a slightly different way.

Other types of networks

There are TONS of other cool structures, but they all work exactly like these under the hood - they just add fancy math that allows us to process our inputs in a slightly different way.

CNN: Convolutional Neural Network. You know that flattening step? That kind of sucks. We're actually losing information on the location of the pixels when we do that. If only we could hold onto that information...

Other types of networks

There are TONS of other cool structures, but they all work exactly like these under the hood - they just add fancy math that allows us to process our inputs in a slightly different way.

CNN: Convolutional Neural Network. You know that flattening step? That kind of sucks. We're actually losing information on the location of the pixels when we do that. If only we could hold onto that information...

Other types of networks

What about timeseries? Wouldn't it be nice to be able to tell a neural network the 'order' the data should occur in. Handled by a class of estimators called RNN (or Transformers, more recently).

Other types of networks

What about timeseries? Wouldn't it be nice to be able to tell a neural network the 'order' the data should occur in. Handled by a class of estimators called RNN (or Transformers, more recently).

LSTM/GRU: Long-Short Term Memory/ Gated Recurrent Unit. Both of these adapt our neurons so they can 'hold onto' data further in the learning process and change it as they like. The diagram is ugly, but...

Other types of Networks

What if we literally don't know anything about the data, just how each object relates to one another (think DNA structures)?

Other types of Networks

Oh yeah, remember this?

Make Bob Ross Nightmare Fuel

The only difference between these is what data the model is trained on.

GAN training process:

With great power...

These models are powerful.

With great power...

These models are powerful.

However, they aren't interpretable (yet, and they're getting MUCH better at this every year.)

With great power...

These models are powerful.

However, they aren't interpretable (yet, and they're getting MUCH better at this every year.)

They also use hundreds of thousands of parameters for even very simple models.

With great power...

These models are powerful.

However, they aren't interpretable (yet, and they're getting MUCH better at this every year.)

They also use hundreds of thousands of parameters for even very simple models.

That means you have to be super careful in how you evaluate them.