[Lecture Notes] The Simplest Neural Network Ever

This is the updated version for the class of the Fall semester, 2020.

The most simple neural network that I can imagine consists of 5 lines of TensorFlow code. It looks like more than 5 lines, but you can make it in 5 lines.

import tensorflow as tf
my_model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=[1])
])
my_model.compile(
    loss = 'mse',
    optimizer = 'adam'
)
my_model.fit([0], [1])
my_model.predict([0.5])

It is the most bare-bone structure of machine learning. Throughout a couple of writings, I will explain how these 5 lines of code can be extended to various machine learning models.

Before I fly, I will explain each line of code briefly. The first line is

import tensorflow as tf

It imports the TensorFlow module in the Python environment and reserves tf as the shortcut name to tensorflow. Since Colab serves the version 2 of TensorFlow, you don’t have to change or import anything as long as you are in Colab.

Now I am going to build a neural network model with TensorFlow. Especially I am going to use Keras, which provides simple yet intuitive functions for neural networks. There are two different approaches to define Keras networks: Sequential vs. Functional.

The sequential model is much easier to work with. So I will start my model with the sequential model. The model I am considering is depicted here. My model has one layer and inside the layer, there is only one node. This is the simplest possible structure of neural networks.

So the code is

my_model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=[1])
])

Note that ‘Sequential’ and ‘Dense’ start with capital letters. Why? These are just rules of TensorFlow, so I have to accept them.

Now I can get the summary of my model by typing my_model.summary().

Model: "sequential" _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= dense (Dense)                (None, 1)                 2          ================================================================= Total params: 2 
Trainable params: 2 
Non-trainable params: 0 _________________________________________________________________

Before I explain the summary output, I will tweak numbers in code so that we can compare the outputs.

my_model = tf.keras.Sequential([
    tf.keras.layers.Dense(2, input_shape=[1])
])

I changed the number of nodes in the layer from 1 to 2, and my_model.summary() shows

Model: "sequential_1" _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= dense_1 (Dense)              (None, 2)                 4          ================================================================= Total params: 4 
Trainable params: 4 
Non-trainable params: 0 _________________________________________________________________

We note that the Model name has been changed as well as the Layer name. These are TensorFlow’s naming conventions. Every time I create a model, TensorFlow puts new names on the model, so that I can distinguish from the previous ones.

We also note that the output shape and the number of parameters have been changed. The output shape was (None, 1) since I had one node in the layer, now it is (None, 2) since I have two nodes in the layer.

The number of parameters was 2 since the formula inside the node was f₁ = w₁ x + b₁. Now we have two nodes so the formula inside the first node is f₁ = w₁ x + b₁ and the formula inside the second node is f₂ = w₂ x+ b₂. So we have w₁, w₂, b₁, b₂, which count up to 4.

So here is a quick question. If I increase the number of nodes in the layer to 10, then how many parameters should we get?

my_model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, input_shape=[1])
])

Now I will change the input shape in my model. Here is the code.

my_model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=[2])
])

What does it mean by having input_shape=[2]? Before I answer this question, take a look at my_model.summary().

Model: "sequential_22" _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= dense_25 (Dense)             (None, 1)                 3          ================================================================= Total params: 3 
Trainable params: 3 
Non-trainable params: 0 _________________________________________________________________

The only change is the number of parameters, from 2 to 3. It does not affect the output shape. The same is true for any positive number of input_shape[x] . How should I interpret this?

a20200517_14.py

What is the math behind the summary?

• The dimension of output shape is equaled to the number of nodes in the layer. Right?
• The number of parameters can be obtained by

(number of nodes) * (dimension of input) + (number of nodes)

So,

• 1 * 1 + 1 = 2 for the base model,
• 2 * 1 + 2 = 4 for 2 nodes,
• 1 * 2 + 1 = 3 for 1 node and input_shape[2]

[TensorFlow 2] Garbage In, Garbage Out