Neural Networks Application: Transfer learning with CNNs (R)

library(tidyverse)
library(magrittr)
library(keras)

Using large pretrained models as your foundation

• This notebook is based on https://www.learnopencv.com/keras-tutorial-fine-tuning-using-pre-trained-models/
• Training large CNNs is costly (compute, time) and often we don’t have enough data to bring these models to reasonable performance levels. This is where transfer-learning comes in.
• Assuming that the early layers of a CNN learn very basic features (edges, lines etc.) we can consider a strategy, where we take a large pretrained model, discard the last couple of layers (where we would assume the more abstract information) and train it with our new data.
• This is how you, for instance, could fine-tune a CNN model to some very specific medical imaging data where you simply cannot have so many examples.

Load the data

It’s the same cats&dog data…

# Let's start by downloading and exploring the data
temp <- tempfile()
download.file('https://storage.googleapis.com/sds-file-transfer/dataset.zip',temp)

trying URL 'https://storage.googleapis.com/sds-file-transfer/dataset.zip'
Content type 'application/zip' length 228470636 bytes (217.9 MB)
==================================================
downloaded 217.9 MB

unzip(temp)
unlink(temp)

Preprocessing

Same as in the former notebook

library(tidyverse)
library(magrittr)

library(keras)

# how much data?
train_size <- c(list.files(path = 'dataset/training_set/cats'), list.files(path = 'dataset/training_set/dogs')) %>% length()
test_size <- c(list.files(path = 'dataset/test_set/cats'), list.files(path = 'dataset/test_set/dogs')) %>% length()

train_data_gen = image_data_generator(rescale = 1/255, shear_range = 0.2, zoom_range = 0.2, horizontal_flip = TRUE)
test_data_gen <- image_data_generator(rescale = 1/255)  

train_image_array_gen <- flow_images_from_directory(train_files_path, train_data_gen, target_size = target_size, class_mode = "binary", 
                                                    classes = class_list,batch_size = 32, seed = 1337)

Found 8000 images belonging to 2 classes.

test_image_array_gen <- flow_images_from_directory(test_files_path, test_data_gen, target_size = target_size, class_mode = "binary",
                                          classes = class_list, batch_size = 32, seed = 1337)

Found 2000 images belonging to 2 classes.

Loading pretrained model

Keras has some great built-in applications - basically pretrained models with some nice functional overhead. We will start by loadeing VGG16 (a rather large model with many many many layers that has been trained on imagenet)

# create the base pre-trained model
base_model <- application_vgg16(
  include_top = FALSE,
  weights='imagenet',
  input_shape = c(img_height, img_width, channels))

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5

    8192/58889256 [..............................] - ETA: 1s
  106496/58889256 [..............................] - ETA: 28s
  475136/58889256 [..............................] - ETA: 12s
 1425408/58889256 [..............................] - ETA: 6s 
 3178496/58889256 [>.............................] - ETA: 3s
 3817472/58889256 [>.............................] - ETA: 4s
 4898816/58889256 [=>............................] - ETA: 3s
 6299648/58889256 [==>...........................] - ETA: 5s
 8167424/58889256 [===>..........................] - ETA: 4s
 8396800/58889256 [===>..........................] - ETA: 4s
 9707520/58889256 [===>..........................] - ETA: 4s
10428416/58889256 [====>.........................] - ETA: 4s
13721600/58889256 [=====>........................] - ETA: 3s
14147584/58889256 [======>.......................] - ETA: 3s
16785408/58889256 [=======>......................] - ETA: 3s
18046976/58889256 [========>.....................] - ETA: 3s
20930560/58889256 [=========>....................] - ETA: 2s
22208512/58889256 [==========>...................] - ETA: 2s
25174016/58889256 [===========>..................] - ETA: 3s
26370048/58889256 [============>.................] - ETA: 3s
29188096/58889256 [=============>................] - ETA: 2s
30564352/58889256 [==============>...............] - ETA: 2s
33947648/58889256 [================>.............] - ETA: 1s
34004992/58889256 [================>.............] - ETA: 2s
36298752/58889256 [=================>............] - ETA: 1s
37543936/58889256 [==================>...........] - ETA: 1s
40148992/58889256 [===================>..........] - ETA: 1s
42115072/58889256 [====================>.........] - ETA: 1s
43048960/58889256 [====================>.........] - ETA: 1s
46161920/58889256 [======================>.......] - ETA: 0s
46833664/58889256 [======================>.......] - ETA: 0s
49881088/58889256 [========================>.....] - ETA: 0s
50651136/58889256 [========================>.....] - ETA: 0s
53993472/58889256 [==========================>...] - ETA: 0s
56467456/58889256 [===========================>..] - ETA: 0s
57712640/58889256 [============================>.] - ETA: 0s
58892288/58889256 [==============================] - 3s 0us/step

• Now we got the model, where we can add some own layers

predictions <- base_model$output %>% 
  layer_global_average_pooling_2d() %>% 
  layer_dense(units = 1024, activation = 'relu') %>% 
  layer_dense(units = 1, activation = 'sigmoid')

# this is the model we will train
model <- keras_model(inputs = base_model$input, outputs = predictions)

The trick is, to “freeze” most of the initial layers and thereby preserve the majority of the information embedded in the model already. We will only train the added new dense layer as well as an output layer that fit’s our purpose with the model - finding cats and dogs

# first: train only the top layers 
for (layer in base_model$layers)
  layer$trainable <- FALSE

# compile the model (should be done *after* setting layers to non-trainable)
model %>% compile(optimizer = 'adam', 
                  loss = 'binary_crossentropy',
                  metric = 'accuracy')

# train the model on the new data for a few epochs
model %>% fit_generator(
  train_image_array_gen,
  steps_per_epoch = 800,
  epochs = 2, 
  validation_data = test_image_array_gen,
  validation_steps = 100,
  verbose = FALSE,
  )

• at this point, the top layers are well trained and we can start fine-tuning convolutional layers.
• We will freeze the bottom N layers nd train the remaining top layers.

# let's visualize layer names and layer indices to see how many layers we should freeze:
layers <- base_model$layers
for (i in 1:length(layers))
  cat(i, layers[[i]]$name, "\n")

1 input_1 
2 block1_conv1 
3 block1_conv2 
4 block1_pool 
5 block2_conv1 
6 block2_conv2 
7 block2_pool 
8 block3_conv1 
9 block3_conv2 
10 block3_conv3 
11 block3_pool 
12 block4_conv1 
13 block4_conv2 
14 block4_conv3 
15 block4_pool 
16 block5_conv1 
17 block5_conv2 
18 block5_conv3 
19 block5_pool 

# we chose to train the top 2 inblocks, i.e. we will freeze the first layers and unfreeze the rest:
for (i in 1:18)
  layers[[i]]$trainable <- FALSE
for (i in 19:length(layers))
  layers[[i]]$trainable <- TRUE

# we need to recompile the model for these modifications to take effect
model %>% compile(optimizer = 'adam', 
                  loss = 'binary_crossentropy',
                  metric = 'accuracy')

# we train our model again (this time fine-tuning the top 2  blocks
# alongside the top Dense layers
# train the model on the new data for a few epochs
model %>% fit_generator(
  train_image_array_gen,
  steps_per_epoch = 800,
  epochs = 2, 
  verbose = FALSE,
  )

hist %>% plot()

Evaluation returns loss and accuracy

model %>% evaluate_generator(test_image_array_gen, steps=500)

     loss  accuracy 
0.4224312 0.8051285 

Ok, now we can delete all the images again…

# unlink("dataset", recursive = TRUE)