Neural Networks Application: Recurrent Neural Networks (R)

library(tidyverse)
library(magrittr)

library(keras)

RNN basics

Loops

To make these notions absolutely unambiguous, let’s write a naive R implementation of the forward pass of the simple RNN.

# We define a function that produces a "random" array
random_array <- function(dim) {
  array(runif(prod(dim)), dim = dim)
}

# We define some constants as arguments for the functions to come
timesteps <- 100 # Number of timesteps in the input sequence
input_features <- 32 # Dimensionality of the input feature space
output_features <- 64 # Dimensionality of the output feature space

# We create the initial inputs
inputs <- random_array(dim = c(timesteps, input_features)) # Input data: random noise for the sake of the example
state_t <- rep_len(0, length = c(output_features)) # Initial state t: an all-zero vector

# We create some random matrices for W, U, b
W <- random_array(dim = c(output_features, input_features)) # Creates random weight matrices: W
U <- random_array(dim = c(output_features, output_features)) # Creates random weight matrices: U
b <- random_array(dim = c(output_features, 1)) # Creates random weight matrices: b

# We create an empty array for the output sequence
output_sequence <- array(0, dim = c(timesteps, output_features))

dim(output_sequence)

[1] 100  64

head(output_sequence[,1:5])

     [,1] [,2] [,3] [,4] [,5]
[1,]    0    0    0    0    0
[2,]    0    0    0    0    0
[3,]    0    0    0    0    0
[4,]    0    0    0    0    0
[5,]    0    0    0    0    0
[6,]    0    0    0    0    0

So, we just made up an random inputs array, lets take a little look:

dim(inputs)

[1] 100  32

head(inputs[,1:5])

          [,1]       [,2]      [,3]       [,4]      [,5]
[1,] 0.4374455 0.65958703 0.8908022 0.30223137 0.1398738
[2,] 0.4993365 0.04578802 0.6532554 0.87700350 0.9450578
[3,] 0.8464301 0.60858042 0.3293128 0.99067676 0.8256690
[4,] 0.8522938 0.63519020 0.3309987 0.09233357 0.6345082
[5,] 0.5475497 0.60797047 0.4218896 0.34651074 0.6111947
[6,] 0.6299324 0.19615404 0.2303865 0.55745211 0.9669684

There we go. We created an 2d tensor, where we have an vector of 32 features over 100 timesteps. Likewise, we created a random weight matrix W (weights for t) and U (weights for t-1) of dimensionality 64x32 (we want 64 outputs), and a bias vector of lenght 64.

dim(W)

[1] 64 32

head(W[,1:5])

           [,1]      [,2]        [,3]      [,4]       [,5]
[1,] 0.05131798 0.4428372 0.047965156 0.9116978 0.05780573
[2,] 0.41738887 0.7932492 0.647385459 0.9647217 0.46568377
[3,] 0.11388474 0.9096280 0.009517771 0.1819730 0.78334617
[4,] 0.31752881 0.7400719 0.889261930 0.1602679 0.80351720
[5,] 0.49609406 0.1952676 0.134865081 0.4120722 0.97223370
[6,] 0.29625469 0.9341890 0.563066539 0.1283034 0.63314607

All set up, lets run a loop, where we apply some activation function (here tanh()) on the weighted input_t, but we add the (in another way) weighted state_t (the lagged value of input_t \(\rightarrow\) input_t-1).

Note: I use tanh() just as an example for whatever activation function you might want to apply to your inputs. tanh() (hyperbolic tangent) is popular for RNNs, since it squishes input values between a range of [-1,1]

for (i in 1:nrow(inputs)) {
  input_t <- inputs[i,]                                                # input_t is a vector of shape (input_features)
  output_t <- tanh(as.numeric((W %*% input_t) + (U %*% state_t) + b))  # Combines the input with the current state (the previous output) 
  output_sequence[i,] <- as.numeric(output_t)                          # Updates the result matrix
  state_t <- output_t                                                  # Updates the state of the network for the next timestep
}

glimpse(output_sequence)

 num [1:100, 1:64] 1 1 1 1 1 ...

Note: In U %*% state_t, the %*% operator performs a real matrix multiplication (every element of U gets multiplied with every element of state_t), not the dotproduct (cell-wise multiplication), as U * state_t would.

Easy enough: in summary, an RNN is an neural network application of a for() (reuses values computed during the previous iteration of the loop), nothing more. Of course, there are many different RNNs fitting this definition that you could build—this example is one of the simplest RNN formulations. Anyhow, In case we would not have to train the weights, we are done by here.

RNNs are characterized by their step function, such as the following function in this case

output_t <- tanh(as.numeric((W %*% input_t) + (U %*% state_t) + b))

glimpse(output_t)

 num [1:64] 1 1 1 1 1 1 1 1 1 1 ...

So, that’s pretty much how to construct a recurrent layer “by hand”. Most probably you will never have to, but I like to believe it demystifies the whole process.Moving on…

Recurrent Layers in Keras

The process you just naively implemented in R corresponds to an actual Keras layer: layer_simple_rnn

layer_simple_rnn(units = 32)

Like all recurrent layers in Keras, layer_simple_rnn can be run in two different modes:

1. it can return either the full sequences of successive outputs for each timestep (a 3D tensor of shape (batch_size, timesteps, output_features))
2. or only the last output for each input sequence (a 2D tensor of shape (batch_size, output_features)).

These two modes are controlled by the return_sequences constructor argument. Let’s look at an example that uses layer_simple_rnn and returns only the output at the last timestep:

model <- keras_model_sequential() %>%
  layer_embedding(input_dim = 10000, output_dim = 32) %>% # About this type of layer, we talk later
  layer_simple_rnn(units = 32)

2020-11-26 13:34:36.818008: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-11-26 13:34:36.837779: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8aa920f6c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-26 13:34:36.837835: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

summary(model)

Model: "sequential"
______________________________________________________________________________________________________________
Layer (type)                                     Output Shape                                Param #          
==============================================================================================================
embedding (Embedding)                            (None, None, 32)                            320000           
______________________________________________________________________________________________________________
simple_rnn (SimpleRNN)                           (None, 32)                                  2080             
==============================================================================================================
Total params: 322,080
Trainable params: 322,080
Non-trainable params: 0
______________________________________________________________________________________________________________

The following example in turn returns the full state sequence (return_sequences = TRUE):

model_sequence <- keras_model_sequential() %>%
  layer_embedding(input_dim = 10000, output_dim = 32) %>%
  layer_simple_rnn(units = 32, return_sequences = TRUE)

summary(model_sequence)

Model: "sequential_1"
______________________________________________________________________________________________________________
Layer (type)                                     Output Shape                                Param #          
==============================================================================================================
embedding_1 (Embedding)                          (None, None, 32)                            320000           
______________________________________________________________________________________________________________
simple_rnn_1 (SimpleRNN)                         (None, None, 32)                            2080             
==============================================================================================================
Total params: 322,080
Trainable params: 322,080
Non-trainable params: 0
______________________________________________________________________________________________________________

It’s sometimes useful to stack several recurrent layers one after the other in order to increase the representational power of a network. In such a setup, you have to get all of the intermediate layers to return full sequences. More on that later.

model_sequence_stacked <- keras_model_sequential() %>%
  layer_embedding(input_dim = 10000, output_dim = 32) %>%
  layer_simple_rnn(units = 32, return_sequences = TRUE) %>%
  layer_simple_rnn(units = 32, return_sequences = FALSE)

summary(model_sequence_stacked)

Model: "sequential_2"
______________________________________________________________________________________________________________
Layer (type)                                     Output Shape                                Param #          
==============================================================================================================
embedding_2 (Embedding)                          (None, None, 32)                            320000           
______________________________________________________________________________________________________________
simple_rnn_3 (SimpleRNN)                         (None, None, 32)                            2080             
______________________________________________________________________________________________________________
simple_rnn_2 (SimpleRNN)                         (None, 32)                                  2080             
==============================================================================================================
Total params: 324,160
Trainable params: 324,160
Non-trainable params: 0
______________________________________________________________________________________________________________

Recurrent Neural Networks (a text example)

So, lets start applying a first RNN, and since you are already warmed up, we do it with text data. It is not hard to see why for making sense of text data, sequential models would be a good idea.

The IMDB dataset

You’ll work with the IMDB dataset: a set of 50,000 highly polarized movie reviews from the Internet Movie Database, labeled by sentiment (positive/negative). They’re split into 25,000 reviews for training and 25,000 reviews for testing, each set consisting of 50% negative and 50% positive reviews. Read more details here. in case you are interested.

Just like the MNIST dataset, the IMDB dataset comes packaged with Keras, and is already neathly preprocessed. Each review is encoded as a sequence of word indexes (integers), where each integer stands for a specific word in a dictionary. For convenience, words are indexed by overall frequency in the dataset, so that for instance the integer 3 encodes the 3rd most frequent word in the data. This allows for quick filtering operations such as: “only consider the top 10,000 most common words, but eliminate the top 20 most common words” (Note: 0 is the encoding for unknown words).

Load the data

imdb <- dataset_imdb(num_words = 10000)

#c(c(train_data, train_labels), c(test_data, test_labels)) %<-% imdb

Note: The datasets built into Keras are all nested lists of training and test data. Here, we use the multi-assignment operator %<-% from the zeallot package to unpack the list into a set of distinct variables. This is only a convenience function, and could equally be written as follows:

train_data <- imdb$train$x
train_labels <- imdb$train$y

test_data <- imdb$test$x
test_labels <- imdb$test$y

The argument num_words = 10000 means we keep only the top 10,000 most frequently occurring words in the training data. Rare words will be discarded. This allows you to work with vector data of a manageable size.

The variables train_data and test_data are lists of reviews; each review is a list of word indices (encoding a sequence of words). train_labels and test_labels are lists of 0s and 1s, where 0 stands for negative and 1 stands for positive:

glimpse(train_labels[1:5])

 int [1:5] 1 0 0 1 0

glimpse(train_data[1:5])

List of 5
 $ : int [1:218] 1 14 22 16 43 530 973 1622 1385 65 ...
 $ : int [1:189] 1 194 1153 194 8255 78 228 5 6 1463 ...
 $ : int [1:141] 1 14 47 8 30 31 7 4 249 108 ...
 $ : int [1:550] 1 4 2 2 33 2804 4 2040 432 111 ...
 $ : int [1:147] 1 249 1323 7 61 113 10 10 13 1637 ...

Because you’re restricting yourself to the top 10,000 most frequent words, no word index will exceed 10,000:

train_data %>% map_int(max) %>% max()

[1] 9999

Sidenote: Notice that all words are not appearing anymore as strings, but as a numeric index. This is very pratical to do as preprocesing of text data before using them for ML workflows. However, we cannot read the reviews anymore manyally. Only for fun, here’s how you can quickly decode one of these reviews back to English words:

word_index <- dataset_imdb_word_index() # word_index is a named list mapping words to an integer index.

word_index %>% head()

word_index %<>% as_tibble() %>%
  pivot_longer(everything())

word_index %>% head()

review_words <- train_data[[1]] %>% as_tibble() %>%
  left_join(word_index, by = 'value')

review_words %>% pull(name) %>% paste(collapse = ' ')

[1] "the as you with out themselves powerful lets loves their becomes reaching had journalist of lot from anyone to have after out atmosphere never more room and it so heart shows to years of every never going and help moments or of every chest visual movie except her was several of enough more with is now current film as you of mine potentially unfortunately of you than him that with out themselves her get for was camp of you movie sometimes movie that with scary but and to story wonderful that in seeing in character to of 70s musicians with heart had shadows they of here that with her serious to have does when from why what have critics they is you that isn't one will very to as itself with other and in of seen over landed for anyone of and br show's to whether from than out themselves history he name half some br of and odd was two most of mean for 1 any an boat she he should is thought frog but of script you not while history he heart to real at barrel but when from one bit then have two of script their with her nobody most that with wasn't to with armed acting watch an for with heartfelt film want an"

Preprocessing

So, we have our data ready. However, we cannot can’t feed lists of integers (keep in mind, they are supposed to not represent numbers but words) into a neural network. You have to turn your lists into tensors. There are two ways to do that:

1. One-hot encode your lists to turn them into vectors of 0s and 1s. This would mean, for instance, turning the sequence [3, 5] into a 10,000-dimensional vector that would be all 0s except for indices 3 and 5, which would be 1. Then you could use as the first layer in your network a dense layer, capable of handling floating-point vector data.
2. Pad your lists so that they all have the same length, turn them into an integer tensor of shape (samples, word_indices), and then use as the first layer in your network a layer capable of handling such integer tensors (the “embedding” layer, comming soon).

While the first approach sounds computationally very inefficient, it is also the most intuitive to operationalize in terms of data-munging. Let’s warm up with this first solution and vectorize the data, which we will do manually for maximum clarity.

# Again, a small function that creates a 0-matrix, and replaces the corresponding words with 1.
vectorize_sequences <- function(sequences, dimension) {
  results <- matrix(0, nrow = length(sequences), ncol = dimension) # Creates an all-zero matrix of shape (length(sequences), dimension)
  for(i in 1:length(sequences)){
    results[i, sequences[[i]]] <- 1 # Sets specific indices of results[i] to 1s
  }
  return(results)
}

x_train <- train_data %>% vectorize_sequences(dimension = 10000) 
x_test <- test_data %>% vectorize_sequences(dimension = 10000) 

Here’s what the samples look like now:

str(x_train[1,])

 num [1:10000] 1 1 0 1 1 1 1 1 1 0 ...

Hint: The keras function to_categorical(), which does exactly the same more convenient when the data is in a dataframe shape.

We now only recode the outcomes as numerical.

y_train <- as.numeric(train_labels)
y_test <- as.numeric(test_labels)

Now the data is ready to be fed into a neural network.

A baseline feed-forward ANN

So far so good, let’s run a “normal” feed-forward ANN to predict the review sentiment. We do so, just to warm up and get a sense how good it performs. A little reminder, to build and run a Keras model, you need to:

1: Define the architecture (Layers, shape & activation) 2: Compile the model (choosing optimizer, loss function, and metric) 3: Run the model

Model architecture

We use, like before, a simple feed forward architecture with an input layer of shape 10.000 (number of words in our review vector), 2 inetrmediate hidden layers with 16 units each, and an ouput layer with only 1 unit (since we perform a binary classification). For the hidden layers, we use the standard relu activation function, for the output we use sigmoid, as common for binary classification. Note that all architecture choices in this example are standard, but not necessarily informed by our data. Just a good rule-of-thumb starting point.

model <- keras_model_sequential() %>%
  layer_dense(units = 16, activation = "relu", input_shape = c(10000)) %>%
  layer_dense(units = 16, activation = "relu") %>%
  layer_dense(units = 1, activation = "sigmoid")

Finally, you need to choose a loss function and an optimizer. Because you’re facing a binary classification problem and the output of your network is a probability (you end your network with a single-unit layer with a sigmoid activation), it’s best to use the binary_crossentropy loss. It isn’t the only viable choice: you could use, for instance, mean_squared_error. But crossentropy is usually the best choice when you’re dealing with models that output probabilities. Crossentropy is a quantity from the field of Information Theory that measures the distance between probability distributions or, in this case, between the ground-truth distribution and your predictions. For the optimizer we go for the allrounder rmsprop, since we in this case have no reason to believe otherwise.

model %>% compile(
  optimizer = "adam",
  loss = "binary_crossentropy",
  metrics = "accuracy"
)

So, we are pretty much done. Due to the big fiorst layer, we have a lot of parameter, but otherwise nothing extraordinary happening yet.

summary(model)

Model: "sequential_3"
______________________________________________________________________________________________________________
Layer (type)                                     Output Shape                                Param #          
==============================================================================================================
dense_2 (Dense)                                  (None, 16)                                  160016           
______________________________________________________________________________________________________________
dense_1 (Dense)                                  (None, 16)                                  272              
______________________________________________________________________________________________________________
dense (Dense)                                    (None, 1)                                   17               
==============================================================================================================
Total params: 160,305
Trainable params: 160,305
Non-trainable params: 0
______________________________________________________________________________________________________________

Lets run it!

history_ann <- model %>% fit(
  x_train,
  y_train,
  epochs = 10,
  batch_size = 512,
  validation_split = 0.25
)

Epoch 1/10

 1/37 [..............................] - ETA: 0s - loss: 0.7031 - accuracy: 0.4961
 3/37 [=>............................] - ETA: 0s - loss: 0.6969 - accuracy: 0.4818
 7/37 [====>.........................] - ETA: 0s - loss: 0.6890 - accuracy: 0.5505
10/37 [=======>......................] - ETA: 0s - loss: 0.6828 - accuracy: 0.5619
13/37 [=========>....................] - ETA: 0s - loss: 0.6755 - accuracy: 0.5739
16/37 [===========>..................] - ETA: 0s - loss: 0.6681 - accuracy: 0.5903
19/37 [==============>...............] - ETA: 0s - loss: 0.6589 - accuracy: 0.6101
22/37 [================>.............] - ETA: 0s - loss: 0.6485 - accuracy: 0.6295
25/37 [===================>..........] - ETA: 0s - loss: 0.6386 - accuracy: 0.6450
28/37 [=====================>........] - ETA: 0s - loss: 0.6293 - accuracy: 0.6608
31/37 [========================>.....] - ETA: 0s - loss: 0.6206 - accuracy: 0.6764
34/37 [==========================>...] - ETA: 0s - loss: 0.6107 - accuracy: 0.6899
37/37 [==============================] - 1s 21ms/step - loss: 0.6035 - accuracy: 0.6983

37/37 [==============================] - 2s 63ms/step - loss: 0.6035 - accuracy: 0.6983 - val_loss: 0.4827 - val_accuracy: 0.8339
Epoch 2/10

 1/37 [..............................] - ETA: 0s - loss: 0.4659 - accuracy: 0.8594
 5/37 [===>..........................] - ETA: 0s - loss: 0.4492 - accuracy: 0.8770
 9/37 [======>.......................] - ETA: 0s - loss: 0.4347 - accuracy: 0.8787
13/37 [=========>....................] - ETA: 0s - loss: 0.4249 - accuracy: 0.8758
17/37 [============>.................] - ETA: 0s - loss: 0.4143 - accuracy: 0.8795
21/37 [================>.............] - ETA: 0s - loss: 0.4020 - accuracy: 0.8820
24/37 [==================>...........] - ETA: 0s - loss: 0.3969 - accuracy: 0.8810
27/37 [====================>.........] - ETA: 0s - loss: 0.3890 - accuracy: 0.8825
32/37 [========================>.....] - ETA: 0s - loss: 0.3785 - accuracy: 0.8855
37/37 [==============================] - 1s 15ms/step - loss: 0.3687 - accuracy: 0.8877

37/37 [==============================] - 1s 27ms/step - loss: 0.3687 - accuracy: 0.8877 - val_loss: 0.3224 - val_accuracy: 0.8797
Epoch 3/10

 1/37 [..............................] - ETA: 0s - loss: 0.2877 - accuracy: 0.9004
 5/37 [===>..........................] - ETA: 0s - loss: 0.2581 - accuracy: 0.9223
 9/37 [======>.......................] - ETA: 0s - loss: 0.2473 - accuracy: 0.9273
13/37 [=========>....................] - ETA: 0s - loss: 0.2485 - accuracy: 0.9264
16/37 [===========>..................] - ETA: 0s - loss: 0.2467 - accuracy: 0.9255
19/37 [==============>...............] - ETA: 0s - loss: 0.2442 - accuracy: 0.9249
23/37 [=================>............] - ETA: 0s - loss: 0.2396 - accuracy: 0.9254
27/37 [====================>.........] - ETA: 0s - loss: 0.2374 - accuracy: 0.9256
30/37 [=======================>......] - ETA: 0s - loss: 0.2345 - accuracy: 0.9267
34/37 [==========================>...] - ETA: 0s - loss: 0.2320 - accuracy: 0.9265
37/37 [==============================] - 1s 15ms/step - loss: 0.2321 - accuracy: 0.9259

37/37 [==============================] - 1s 27ms/step - loss: 0.2321 - accuracy: 0.9259 - val_loss: 0.2739 - val_accuracy: 0.8898
Epoch 4/10

 1/37 [..............................] - ETA: 0s - loss: 0.1671 - accuracy: 0.9531
 5/37 [===>..........................] - ETA: 0s - loss: 0.1751 - accuracy: 0.9484
 9/37 [======>.......................] - ETA: 0s - loss: 0.1754 - accuracy: 0.9464
13/37 [=========>....................] - ETA: 0s - loss: 0.1724 - accuracy: 0.9483
17/37 [============>.................] - ETA: 0s - loss: 0.1724 - accuracy: 0.9497
20/37 [===============>..............] - ETA: 0s - loss: 0.1710 - accuracy: 0.9495
23/37 [=================>............] - ETA: 0s - loss: 0.1696 - accuracy: 0.9490
26/37 [====================>.........] - ETA: 0s - loss: 0.1680 - accuracy: 0.9488
28/37 [=====================>........] - ETA: 0s - loss: 0.1675 - accuracy: 0.9485
32/37 [========================>.....] - ETA: 0s - loss: 0.1669 - accuracy: 0.9482
36/37 [============================>.] - ETA: 0s - loss: 0.1675 - accuracy: 0.9472
37/37 [==============================] - 1s 15ms/step - loss: 0.1673 - accuracy: 0.9473

37/37 [==============================] - 1s 25ms/step - loss: 0.1673 - accuracy: 0.9473 - val_loss: 0.2730 - val_accuracy: 0.8898
Epoch 5/10

 1/37 [..............................] - ETA: 0s - loss: 0.1260 - accuracy: 0.9648
 5/37 [===>..........................] - ETA: 0s - loss: 0.1350 - accuracy: 0.9598
 9/37 [======>.......................] - ETA: 0s - loss: 0.1338 - accuracy: 0.9618
12/37 [========>.....................] - ETA: 0s - loss: 0.1314 - accuracy: 0.9629
16/37 [===========>..................] - ETA: 0s - loss: 0.1259 - accuracy: 0.9653
20/37 [===============>..............] - ETA: 0s - loss: 0.1258 - accuracy: 0.9654
24/37 [==================>...........] - ETA: 0s - loss: 0.1249 - accuracy: 0.9652
28/37 [=====================>........] - ETA: 0s - loss: 0.1263 - accuracy: 0.9644
33/37 [=========================>....] - ETA: 0s - loss: 0.1265 - accuracy: 0.9638
36/37 [============================>.] - ETA: 0s - loss: 0.1274 - accuracy: 0.9632
37/37 [==============================] - 1s 14ms/step - loss: 0.1279 - accuracy: 0.9627

37/37 [==============================] - 1s 24ms/step - loss: 0.1279 - accuracy: 0.9627 - val_loss: 0.2897 - val_accuracy: 0.8878
Epoch 6/10

 1/37 [..............................] - ETA: 0s - loss: 0.1041 - accuracy: 0.9707
 5/37 [===>..........................] - ETA: 0s - loss: 0.0985 - accuracy: 0.9766
 9/37 [======>.......................] - ETA: 0s - loss: 0.0994 - accuracy: 0.9746
14/37 [==========>...................] - ETA: 0s - loss: 0.0991 - accuracy: 0.9750
19/37 [==============>...............] - ETA: 0s - loss: 0.0980 - accuracy: 0.9754
24/37 [==================>...........] - ETA: 0s - loss: 0.0990 - accuracy: 0.9744
28/37 [=====================>........] - ETA: 0s - loss: 0.1001 - accuracy: 0.9735
32/37 [========================>.....] - ETA: 0s - loss: 0.0996 - accuracy: 0.9734
36/37 [============================>.] - ETA: 0s - loss: 0.0990 - accuracy: 0.9731
37/37 [==============================] - 0s 13ms/step - loss: 0.0987 - accuracy: 0.9732

37/37 [==============================] - 1s 23ms/step - loss: 0.0987 - accuracy: 0.9732 - val_loss: 0.3114 - val_accuracy: 0.8853
Epoch 7/10

 1/37 [..............................] - ETA: 0s - loss: 0.0590 - accuracy: 0.9941
 6/37 [===>..........................] - ETA: 0s - loss: 0.0769 - accuracy: 0.9805
11/37 [=======>......................] - ETA: 0s - loss: 0.0716 - accuracy: 0.9835
16/37 [===========>..................] - ETA: 0s - loss: 0.0716 - accuracy: 0.9841
20/37 [===============>..............] - ETA: 0s - loss: 0.0727 - accuracy: 0.9838
24/37 [==================>...........] - ETA: 0s - loss: 0.0739 - accuracy: 0.9833
28/37 [=====================>........] - ETA: 0s - loss: 0.0737 - accuracy: 0.9830
32/37 [========================>.....] - ETA: 0s - loss: 0.0738 - accuracy: 0.9825
36/37 [============================>.] - ETA: 0s - loss: 0.0752 - accuracy: 0.9820
37/37 [==============================] - 0s 13ms/step - loss: 0.0753 - accuracy: 0.9820

37/37 [==============================] - 1s 22ms/step - loss: 0.0753 - accuracy: 0.9820 - val_loss: 0.3387 - val_accuracy: 0.8806
Epoch 8/10

 1/37 [..............................] - ETA: 0s - loss: 0.0676 - accuracy: 0.9883
 5/37 [===>..........................] - ETA: 0s - loss: 0.0526 - accuracy: 0.9922
 9/37 [======>.......................] - ETA: 0s - loss: 0.0551 - accuracy: 0.9905
14/37 [==========>...................] - ETA: 0s - loss: 0.0589 - accuracy: 0.9874
19/37 [==============>...............] - ETA: 0s - loss: 0.0563 - accuracy: 0.9883
23/37 [=================>............] - ETA: 0s - loss: 0.0557 - accuracy: 0.9888
27/37 [====================>.........] - ETA: 0s - loss: 0.0560 - accuracy: 0.9889
31/37 [========================>.....] - ETA: 0s - loss: 0.0575 - accuracy: 0.9877
36/37 [============================>.] - ETA: 0s - loss: 0.0585 - accuracy: 0.9875
37/37 [==============================] - 0s 13ms/step - loss: 0.0585 - accuracy: 0.9875

37/37 [==============================] - 1s 21ms/step - loss: 0.0585 - accuracy: 0.9875 - val_loss: 0.3883 - val_accuracy: 0.8806
Epoch 9/10

 1/37 [..............................] - ETA: 0s - loss: 0.0410 - accuracy: 0.9941
 6/37 [===>..........................] - ETA: 0s - loss: 0.0463 - accuracy: 0.9919
10/37 [=======>......................] - ETA: 0s - loss: 0.0454 - accuracy: 0.9920
15/37 [===========>..................] - ETA: 0s - loss: 0.0432 - accuracy: 0.9928
19/37 [==============>...............] - ETA: 0s - loss: 0.0438 - accuracy: 0.9924
23/37 [=================>............] - ETA: 0s - loss: 0.0439 - accuracy: 0.9925
27/37 [====================>.........] - ETA: 0s - loss: 0.0439 - accuracy: 0.9925
31/37 [========================>.....] - ETA: 0s - loss: 0.0436 - accuracy: 0.9924
36/37 [============================>.] - ETA: 0s - loss: 0.0432 - accuracy: 0.9922
37/37 [==============================] - 0s 13ms/step - loss: 0.0434 - accuracy: 0.9923

37/37 [==============================] - 1s 21ms/step - loss: 0.0434 - accuracy: 0.9923 - val_loss: 0.4018 - val_accuracy: 0.8768
Epoch 10/10

 1/37 [..............................] - ETA: 0s - loss: 0.0268 - accuracy: 1.0000
 5/37 [===>..........................] - ETA: 0s - loss: 0.0321 - accuracy: 0.9965
10/37 [=======>......................] - ETA: 0s - loss: 0.0308 - accuracy: 0.9967
15/37 [===========>..................] - ETA: 0s - loss: 0.0320 - accuracy: 0.9965
19/37 [==============>...............] - ETA: 0s - loss: 0.0306 - accuracy: 0.9966
24/37 [==================>...........] - ETA: 0s - loss: 0.0308 - accuracy: 0.9964
29/37 [======================>.......] - ETA: 0s - loss: 0.0313 - accuracy: 0.9962
33/37 [=========================>....] - ETA: 0s - loss: 0.0315 - accuracy: 0.9961
37/37 [==============================] - 0s 12ms/step - loss: 0.0317 - accuracy: 0.9958

37/37 [==============================] - 1s 20ms/step - loss: 0.0317 - accuracy: 0.9958 - val_loss: 0.4379 - val_accuracy: 0.8762

And lets take a look how the accuracy and loss developed over the epocs.

plot(history_ann)

Ok, agaion we see that we are overfitting. While accuracy and loss as in most cases further increases during the epocs, we see after 3 epocs the metrics to decline on our validation data. At one point we have to deal with that, but not now.

Lets just sum up:

• We created a one-hot-encoding term-document matrix for the most 10.000 frequent terms used in Imdb reviews, and used that to predict if the review had a positive or negative sentiment.
• We did so by feeding this 10.000 dimensional vector as a 2d tensor of shape (sample, features) to a simple feed-forward network with pretty standard architecture
• We got an accuracy somewhere between 85-90% in the validation sample.

I would say, not too bad at all. However, represenmting a review as a 10.000 dimensional vector of one-hot encodings of word occurence appears pretty blunt. There must be something better, right?

A Recurrent Neural Network Approach to text data in Keras

So, now that we tried a pretty naive model as baseline, lets move on.

Preprocessing

In our first baseline model, we used a document-term matrix as inputs for training, with one-hot-encodings (= dummy variables) for the 10.000 most popular terms. This has a couple of disadvantages. Besides being a very large and sparse vector for every review, as a “bag-of-words”, it did not take the word-order (sequence) into account.

This time, we use a different approach, therefore also need a different input data-structure. We now use pad_sequences() to create a integer tensor of shape (samples, word_indices). However, reviews vary in lenght, which is a problem since Keras reqieres the inputs to have the same shape across the whole sample. Therefore, we use the maxlen = 500 argument, to restrict ourselves to the first 500 words in every review.

As a consequence, longer reviews will be cut at 500 words, and shorter reviews will be filled up with 0 values.

x_train <- pad_sequences(train_data, maxlen = 500)
x_test <- pad_sequences(test_data, maxlen = 500)

Lets take a look:

glimpse(x_train)

 int [1:25000, 1:500] 0 0 0 687 0 0 0 4 0 0 ...

We see that we indeed end up with word sequences. However, also notice that we have quite a bunch of 0s, representing unknown words.

Lets set up our model. as discussed, we will first use a layer_embedding to compress our initial one-hot-encoding vector of lenght 10.000 to a “meaning-vector” (=embedding) of the lower dimensionality of 32. We did not talk about that too much, but the next session will dig deeper into that. Just take it for now…

Then we add a layer_simple_rnn on top, and finally a layer_dense for the binary prediction of review sentiment.

model_rnn <- keras_model_sequential() %>%
  layer_embedding(input_dim = 10000, output_dim = 32) %>%
  layer_simple_rnn(units = 32, activation = "tanh") %>%
  layer_dense(units = 1, activation = "sigmoid")

summary(model_rnn)

Model: "sequential_4"
_________________________________________________________________________________________________
Layer (type)                               Output Shape                           Param #        
=================================================================================================
embedding_3 (Embedding)                    (None, None, 32)                       320000         
_________________________________________________________________________________________________
simple_rnn_4 (SimpleRNN)                   (None, 32)                             2080           
_________________________________________________________________________________________________
dense_3 (Dense)                            (None, 1)                              33             
=================================================================================================
Total params: 322,113
Trainable params: 322,113
Non-trainable params: 0
_________________________________________________________________________________________________

THe further parameters are quite conventional and by now well-known.

model_rnn %>% compile(
  optimizer = "adam",
  loss = "binary_crossentropy",
  metrics = "accuracy"
)

And we already run the model:

history_rnn <- model_rnn %>% fit(
  x_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.25
)

Epoch 1/10

  1/147 [..............................] - ETA: 0s - loss: 0.6993 - accuracy: 0.4922
  2/147 [..............................] - ETA: 11s - loss: 0.6987 - accuracy: 0.5039
  3/147 [..............................] - ETA: 15s - loss: 0.6990 - accuracy: 0.5156
  4/147 [..............................] - ETA: 18s - loss: 0.6989 - accuracy: 0.5254
  5/147 [>.............................] - ETA: 18s - loss: 0.7006 - accuracy: 0.5172
  6/147 [>.............................] - ETA: 19s - loss: 0.7042 - accuracy: 0.5026
  7/147 [>.............................] - ETA: 20s - loss: 0.7021 - accuracy: 0.5078
  8/147 [>.............................] - ETA: 20s - loss: 0.7022 - accuracy: 0.5020
  9/147 [>.............................] - ETA: 20s - loss: 0.7013 - accuracy: 0.5035
 10/147 [=>............................] - ETA: 20s - loss: 0.7019 - accuracy: 0.4984
 11/147 [=>............................] - ETA: 21s - loss: 0.7026 - accuracy: 0.4936
 12/147 [=>............................] - ETA: 21s - loss: 0.7021 - accuracy: 0.4928
 13/147 [=>............................] - ETA: 21s - loss: 0.7021 - accuracy: 0.4892
 14/147 [=>............................] - ETA: 21s - loss: 0.7000 - accuracy: 0.4950
 15/147 [==>...........................] - ETA: 21s - loss: 0.7000 - accuracy: 0.4958
 16/147 [==>...........................] - ETA: 21s - loss: 0.7000 - accuracy: 0.4976
 17/147 [==>...........................] - ETA: 21s - loss: 0.6995 - accuracy: 0.4986
 18/147 [==>...........................] - ETA: 21s - loss: 0.6991 - accuracy: 0.5013
 19/147 [==>...........................] - ETA: 21s - loss: 0.6997 - accuracy: 0.4967
 20/147 [===>..........................] - ETA: 21s - loss: 0.7003 - accuracy: 0.4934
 21/147 [===>..........................] - ETA: 21s - loss: 0.6999 - accuracy: 0.4955
 22/147 [===>..........................] - ETA: 21s - loss: 0.6992 - accuracy: 0.4947
 23/147 [===>..........................] - ETA: 21s - loss: 0.6990 - accuracy: 0.4949
 24/147 [===>..........................] - ETA: 21s - loss: 0.6989 - accuracy: 0.4954
 25/147 [====>.........................] - ETA: 21s - loss: 0.6989 - accuracy: 0.4950
 26/147 [====>.........................] - ETA: 21s - loss: 0.6987 - accuracy: 0.4961
 27/147 [====>.........................] - ETA: 21s - loss: 0.6982 - accuracy: 0.4986
 28/147 [====>.........................] - ETA: 21s - loss: 0.6983 - accuracy: 0.4983
 29/147 [====>.........................] - ETA: 21s - loss: 0.6988 - accuracy: 0.4968
 30/147 [=====>........................] - ETA: 21s - loss: 0.6985 - accuracy: 0.4979
 31/147 [=====>........................] - ETA: 21s - loss: 0.6980 - accuracy: 0.4997
 32/147 [=====>........................] - ETA: 21s - loss: 0.6979 - accuracy: 0.4993
 33/147 [=====>........................] - ETA: 20s - loss: 0.6976 - accuracy: 0.5017
 34/147 [=====>........................] - ETA: 20s - loss: 0.6973 - accuracy: 0.5028
 35/147 [======>.......................] - ETA: 20s - loss: 0.6970 - accuracy: 0.5042
 36/147 [======>.......................] - ETA: 20s - loss: 0.6968 - accuracy: 0.5039
 37/147 [======>.......................] - ETA: 20s - loss: 0.6964 - accuracy: 0.5063
 38/147 [======>.......................] - ETA: 20s - loss: 0.6963 - accuracy: 0.5060
 39/147 [======>.......................] - ETA: 20s - loss: 0.6964 - accuracy: 0.5056
 40/147 [=======>......................] - ETA: 20s - loss: 0.6963 - accuracy: 0.5049
 41/147 [=======>......................] - ETA: 19s - loss: 0.6961 - accuracy: 0.5063
 42/147 [=======>......................] - ETA: 19s - loss: 0.6958 - accuracy: 0.5071
 43/147 [=======>......................] - ETA: 19s - loss: 0.6958 - accuracy: 0.5067
 44/147 [=======>......................] - ETA: 19s - loss: 0.6957 - accuracy: 0.5075
 45/147 [========>.....................] - ETA: 19s - loss: 0.6956 - accuracy: 0.5083
 46/147 [========>.....................] - ETA: 19s - loss: 0.6954 - accuracy: 0.5095
 47/147 [========>.....................] - ETA: 18s - loss: 0.6954 - accuracy: 0.5085
 48/147 [========>.....................] - ETA: 18s - loss: 0.6952 - accuracy: 0.5093
 49/147 [=========>....................] - ETA: 18s - loss: 0.6952 - accuracy: 0.5094
 50/147 [=========>....................] - ETA: 18s - loss: 0.6949 - accuracy: 0.5106
 51/147 [=========>....................] - ETA: 18s - loss: 0.6948 - accuracy: 0.5112
 52/147 [=========>....................] - ETA: 18s - loss: 0.6947 - accuracy: 0.5114
 53/147 [=========>....................] - ETA: 18s - loss: 0.6944 - accuracy: 0.5127
 54/147 [==========>...................] - ETA: 17s - loss: 0.6942 - accuracy: 0.5130
 55/147 [==========>...................] - ETA: 17s - loss: 0.6938 - accuracy: 0.5153
 56/147 [==========>...................] - ETA: 17s - loss: 0.6938 - accuracy: 0.5149
 57/147 [==========>...................] - ETA: 17s - loss: 0.6938 - accuracy: 0.5143
 58/147 [==========>...................] - ETA: 17s - loss: 0.6938 - accuracy: 0.5145
 59/147 [===========>..................] - ETA: 17s - loss: 0.6938 - accuracy: 0.5156
 60/147 [===========>..................] - ETA: 16s - loss: 0.6936 - accuracy: 0.5160
 61/147 [===========>..................] - ETA: 16s - loss: 0.6935 - accuracy: 0.5160
 62/147 [===========>..................] - ETA: 16s - loss: 0.6934 - accuracy: 0.5163
 63/147 [===========>..................] - ETA: 16s - loss: 0.6934 - accuracy: 0.5157
 64/147 [============>.................] - ETA: 16s - loss: 0.6933 - accuracy: 0.5160
 65/147 [============>.................] - ETA: 16s - loss: 0.6929 - accuracy: 0.5179
 66/147 [============>.................] - ETA: 16s - loss: 0.6926 - accuracy: 0.5196
 67/147 [============>.................] - ETA: 16s - loss: 0.6926 - accuracy: 0.5198
 68/147 [============>.................] - ETA: 15s - loss: 0.6924 - accuracy: 0.5210
 69/147 [=============>................] - ETA: 15s - loss: 0.6923 - accuracy: 0.5216
 70/147 [=============>................] - ETA: 15s - loss: 0.6921 - accuracy: 0.5217
 71/147 [=============>................] - ETA: 15s - loss: 0.6920 - accuracy: 0.5218
 72/147 [=============>................] - ETA: 15s - loss: 0.6920 - accuracy: 0.5218
 73/147 [=============>................] - ETA: 15s - loss: 0.6919 - accuracy: 0.5222
 74/147 [==============>...............] - ETA: 15s - loss: 0.6917 - accuracy: 0.5227
 75/147 [==============>...............] - ETA: 14s - loss: 0.6916 - accuracy: 0.5233
 76/147 [==============>...............] - ETA: 14s - loss: 0.6914 - accuracy: 0.5247
 77/147 [==============>...............] - ETA: 14s - loss: 0.6911 - accuracy: 0.5264
 78/147 [==============>...............] - ETA: 14s - loss: 0.6910 - accuracy: 0.5271
 79/147 [===============>..............] - ETA: 14s - loss: 0.6908 - accuracy: 0.5281
 80/147 [===============>..............] - ETA: 13s - loss: 0.6907 - accuracy: 0.5287
 81/147 [===============>..............] - ETA: 13s - loss: 0.6906 - accuracy: 0.5291
 82/147 [===============>..............] - ETA: 13s - loss: 0.6905 - accuracy: 0.5291
 83/147 [===============>..............] - ETA: 13s - loss: 0.6905 - accuracy: 0.5291
 84/147 [================>.............] - ETA: 13s - loss: 0.6904 - accuracy: 0.5294
 85/147 [================>.............] - ETA: 12s - loss: 0.6903 - accuracy: 0.5298
 86/147 [================>.............] - ETA: 12s - loss: 0.6902 - accuracy: 0.5306
 87/147 [================>.............] - ETA: 12s - loss: 0.6901 - accuracy: 0.5312
 88/147 [================>.............] - ETA: 12s - loss: 0.6901 - accuracy: 0.5312
 89/147 [=================>............] - ETA: 12s - loss: 0.6900 - accuracy: 0.5312
 90/147 [=================>............] - ETA: 11s - loss: 0.6900 - accuracy: 0.5309
 91/147 [=================>............] - ETA: 11s - loss: 0.6899 - accuracy: 0.5315
 92/147 [=================>............] - ETA: 11s - loss: 0.6900 - accuracy: 0.5315
 93/147 [=================>............] - ETA: 11s - loss: 0.6899 - accuracy: 0.5320
 94/147 [==================>...........] - ETA: 11s - loss: 0.6898 - accuracy: 0.5319
 95/147 [==================>...........] - ETA: 10s - loss: 0.6896 - accuracy: 0.5326
 96/147 [==================>...........] - ETA: 10s - loss: 0.6896 - accuracy: 0.5321
 97/147 [==================>...........] - ETA: 10s - loss: 0.6895 - accuracy: 0.5323
 98/147 [===================>..........] - ETA: 10s - loss: 0.6893 - accuracy: 0.5329
 99/147 [===================>..........] - ETA: 10s - loss: 0.6893 - accuracy: 0.5335
100/147 [===================>..........] - ETA: 9s - loss: 0.6892 - accuracy: 0.5334 
101/147 [===================>..........] - ETA: 9s - loss: 0.6891 - accuracy: 0.5341
102/147 [===================>..........] - ETA: 9s - loss: 0.6890 - accuracy: 0.5353
103/147 [====================>.........] - ETA: 9s - loss: 0.6889 - accuracy: 0.5357
104/147 [====================>.........] - ETA: 8s - loss: 0.6888 - accuracy: 0.5362
105/147 [====================>.........] - ETA: 8s - loss: 0.6886 - accuracy: 0.5368
106/147 [====================>.........] - ETA: 8s - loss: 0.6886 - accuracy: 0.5374
107/147 [====================>.........] - ETA: 8s - loss: 0.6886 - accuracy: 0.5375
108/147 [=====================>........] - ETA: 8s - loss: 0.6885 - accuracy: 0.5382
109/147 [=====================>........] - ETA: 7s - loss: 0.6883 - accuracy: 0.5386
110/147 [=====================>........] - ETA: 7s - loss: 0.6882 - accuracy: 0.5386
111/147 [=====================>........] - ETA: 7s - loss: 0.6883 - accuracy: 0.5378
112/147 [=====================>........] - ETA: 7s - loss: 0.6883 - accuracy: 0.5382
113/147 [======================>.......] - ETA: 6s - loss: 0.6882 - accuracy: 0.5382
114/147 [======================>.......] - ETA: 6s - loss: 0.6880 - accuracy: 0.5391
115/147 [======================>.......] - ETA: 6s - loss: 0.6880 - accuracy: 0.5391
116/147 [======================>.......] - ETA: 6s - loss: 0.6880 - accuracy: 0.5386
117/147 [======================>.......] - ETA: 6s - loss: 0.6881 - accuracy: 0.5383
118/147 [=======================>......] - ETA: 5s - loss: 0.6878 - accuracy: 0.5391
119/147 [=======================>......] - ETA: 5s - loss: 0.6876 - accuracy: 0.5395
120/147 [=======================>......] - ETA: 5s - loss: 0.6876 - accuracy: 0.5389
121/147 [=======================>......] - ETA: 5s - loss: 0.6874 - accuracy: 0.5402
122/147 [=======================>......] - ETA: 5s - loss: 0.6872 - accuracy: 0.5409
123/147 [========================>.....] - ETA: 4s - loss: 0.6870 - accuracy: 0.5419
124/147 [========================>.....] - ETA: 4s - loss: 0.6869 - accuracy: 0.5425
125/147 [========================>.....] - ETA: 4s - loss: 0.6868 - accuracy: 0.5427
126/147 [========================>.....] - ETA: 4s - loss: 0.6867 - accuracy: 0.5433
127/147 [========================>.....] - ETA: 3s - loss: 0.6865 - accuracy: 0.5435
128/147 [=========================>....] - ETA: 3s - loss: 0.6865 - accuracy: 0.5435
129/147 [=========================>....] - ETA: 3s - loss: 0.6864 - accuracy: 0.5441
130/147 [=========================>....] - ETA: 3s - loss: 0.6863 - accuracy: 0.5444
131/147 [=========================>....] - ETA: 3s - loss: 0.6861 - accuracy: 0.5446
132/147 [=========================>....] - ETA: 2s - loss: 0.6860 - accuracy: 0.5452
133/147 [==========================>...] - ETA: 2s - loss: 0.6860 - accuracy: 0.5449
134/147 [==========================>...] - ETA: 2s - loss: 0.6859 - accuracy: 0.5450
135/147 [==========================>...] - ETA: 2s - loss: 0.6858 - accuracy: 0.5450
136/147 [==========================>...] - ETA: 2s - loss: 0.6857 - accuracy: 0.5455
137/147 [==========================>...] - ETA: 1s - loss: 0.6856 - accuracy: 0.5454
138/147 [===========================>..] - ETA: 1s - loss: 0.6855 - accuracy: 0.5459
139/147 [===========================>..] - ETA: 1s - loss: 0.6854 - accuracy: 0.5462
140/147 [===========================>..] - ETA: 1s - loss: 0.6853 - accuracy: 0.5464
141/147 [===========================>..] - ETA: 1s - loss: 0.6853 - accuracy: 0.5465
142/147 [===========================>..] - ETA: 0s - loss: 0.6852 - accuracy: 0.5470
143/147 [============================>.] - ETA: 0s - loss: 0.6851 - accuracy: 0.5471
144/147 [============================>.] - ETA: 0s - loss: 0.6851 - accuracy: 0.5470
145/147 [============================>.] - ETA: 0s - loss: 0.6851 - accuracy: 0.5472
146/147 [============================>.] - ETA: 0s - loss: 0.6850 - accuracy: 0.5472
147/147 [==============================] - 28s 192ms/step - loss: 0.6850 - accuracy: 0.5474

147/147 [==============================] - 31s 209ms/step - loss: 0.6850 - accuracy: 0.5474 - val_loss: 0.6713 - val_accuracy: 0.5750
Epoch 2/10

  1/147 [..............................] - ETA: 0s - loss: 0.6293 - accuracy: 0.7500
  2/147 [..............................] - ETA: 10s - loss: 0.6384 - accuracy: 0.7227
  3/147 [..............................] - ETA: 14s - loss: 0.6352 - accuracy: 0.7266
  4/147 [..............................] - ETA: 15s - loss: 0.6344 - accuracy: 0.7285
  5/147 [>.............................] - ETA: 17s - loss: 0.6338 - accuracy: 0.7391
  6/147 [>.............................] - ETA: 17s - loss: 0.6362 - accuracy: 0.7396
  7/147 [>.............................] - ETA: 20s - loss: 0.6370 - accuracy: 0.7355
  8/147 [>.............................] - ETA: 21s - loss: 0.6358 - accuracy: 0.7373
  9/147 [>.............................] - ETA: 21s - loss: 0.6339 - accuracy: 0.7431
 10/147 [=>............................] - ETA: 20s - loss: 0.6326 - accuracy: 0.7477
 11/147 [=>............................] - ETA: 20s - loss: 0.6307 - accuracy: 0.7500
 12/147 [=>............................] - ETA: 20s - loss: 0.6310 - accuracy: 0.7520
 13/147 [=>............................] - ETA: 20s - loss: 0.6296 - accuracy: 0.7554
 14/147 [=>............................] - ETA: 20s - loss: 0.6282 - accuracy: 0.7573
 15/147 [==>...........................] - ETA: 19s - loss: 0.6273 - accuracy: 0.7589
 16/147 [==>...........................] - ETA: 19s - loss: 0.6270 - accuracy: 0.7563
 17/147 [==>...........................] - ETA: 20s - loss: 0.6272 - accuracy: 0.7491
 18/147 [==>...........................] - ETA: 19s - loss: 0.6269 - accuracy: 0.7504
 19/147 [==>...........................] - ETA: 19s - loss: 0.6263 - accuracy: 0.7500
 20/147 [===>..........................] - ETA: 19s - loss: 0.6250 - accuracy: 0.7527
 21/147 [===>..........................] - ETA: 19s - loss: 0.6249 - accuracy: 0.7500
 22/147 [===>..........................] - ETA: 19s - loss: 0.6250 - accuracy: 0.7482
 23/147 [===>..........................] - ETA: 18s - loss: 0.6246 - accuracy: 0.7497
 24/147 [===>..........................] - ETA: 18s - loss: 0.6231 - accuracy: 0.7503
 25/147 [====>.........................] - ETA: 18s - loss: 0.6222 - accuracy: 0.7513
 26/147 [====>.........................] - ETA: 18s - loss: 0.6208 - accuracy: 0.7533
 27/147 [====>.........................] - ETA: 18s - loss: 0.6205 - accuracy: 0.7529
 28/147 [====>.........................] - ETA: 18s - loss: 0.6195 - accuracy: 0.7556
 29/147 [====>.........................] - ETA: 18s - loss: 0.6174 - accuracy: 0.7578
 30/147 [=====>........................] - ETA: 17s - loss: 0.6165 - accuracy: 0.7591
 31/147 [=====>........................] - ETA: 18s - loss: 0.6159 - accuracy: 0.7586
 32/147 [=====>........................] - ETA: 18s - loss: 0.6158 - accuracy: 0.7563
 33/147 [=====>........................] - ETA: 18s - loss: 0.6152 - accuracy: 0.7566
 34/147 [=====>........................] - ETA: 18s - loss: 0.6144 - accuracy: 0.7567
 35/147 [======>.......................] - ETA: 17s - loss: 0.6132 - accuracy: 0.7583
 36/147 [======>.......................] - ETA: 17s - loss: 0.6124 - accuracy: 0.7580
 37/147 [======>.......................] - ETA: 17s - loss: 0.6113 - accuracy: 0.7589
 38/147 [======>.......................] - ETA: 17s - loss: 0.6109 - accuracy: 0.7588
 39/147 [======>.......................] - ETA: 17s - loss: 0.6102 - accuracy: 0.7590
 40/147 [=======>......................] - ETA: 17s - loss: 0.6097 - accuracy: 0.7574
 41/147 [=======>......................] - ETA: 16s - loss: 0.6093 - accuracy: 0.7576
 42/147 [=======>......................] - ETA: 16s - loss: 0.6091 - accuracy: 0.7573
 43/147 [=======>......................] - ETA: 16s - loss: 0.6090 - accuracy: 0.7564
 44/147 [=======>......................] - ETA: 16s - loss: 0.6083 - accuracy: 0.7566
 45/147 [========>.....................] - ETA: 16s - loss: 0.6080 - accuracy: 0.7568
 46/147 [========>.....................] - ETA: 16s - loss: 0.6074 - accuracy: 0.7573
 47/147 [========>.....................] - ETA: 15s - loss: 0.6072 - accuracy: 0.7568
 48/147 [========>.....................] - ETA: 15s - loss: 0.6072 - accuracy: 0.7555
 49/147 [=========>....................] - ETA: 15s - loss: 0.6071 - accuracy: 0.7546
 50/147 [=========>....................] - ETA: 15s - loss: 0.6062 - accuracy: 0.7550
 51/147 [=========>....................] - ETA: 15s - loss: 0.6055 - accuracy: 0.7543
 52/147 [=========>....................] - ETA: 15s - loss: 0.6052 - accuracy: 0.7542
 53/147 [=========>....................] - ETA: 14s - loss: 0.6044 - accuracy: 0.7547
 54/147 [==========>...................] - ETA: 14s - loss: 0.6034 - accuracy: 0.7552
 55/147 [==========>...................] - ETA: 14s - loss: 0.6026 - accuracy: 0.7558
 56/147 [==========>...................] - ETA: 14s - loss: 0.6025 - accuracy: 0.7549
 57/147 [==========>...................] - ETA: 14s - loss: 0.6026 - accuracy: 0.7544
 58/147 [==========>...................] - ETA: 14s - loss: 0.6022 - accuracy: 0.7535
 59/147 [===========>..................] - ETA: 14s - loss: 0.6015 - accuracy: 0.7536
 60/147 [===========>..................] - ETA: 14s - loss: 0.6014 - accuracy: 0.7526
 61/147 [===========>..................] - ETA: 13s - loss: 0.6009 - accuracy: 0.7522
 62/147 [===========>..................] - ETA: 13s - loss: 0.6006 - accuracy: 0.7515
 63/147 [===========>..................] - ETA: 13s - loss: 0.5999 - accuracy: 0.7516
 64/147 [============>.................] - ETA: 13s - loss: 0.5994 - accuracy: 0.7515
 65/147 [============>.................] - ETA: 13s - loss: 0.5989 - accuracy: 0.7512
 66/147 [============>.................] - ETA: 13s - loss: 0.5988 - accuracy: 0.7505
 67/147 [============>.................] - ETA: 12s - loss: 0.5985 - accuracy: 0.7497
 68/147 [============>.................] - ETA: 12s - loss: 0.5976 - accuracy: 0.7501
 69/147 [=============>................] - ETA: 12s - loss: 0.5975 - accuracy: 0.7490
 70/147 [=============>................] - ETA: 12s - loss: 0.5972 - accuracy: 0.7487
 71/147 [=============>................] - ETA: 12s - loss: 0.5968 - accuracy: 0.7485
 72/147 [=============>................] - ETA: 12s - loss: 0.5967 - accuracy: 0.7473
 73/147 [=============>................] - ETA: 11s - loss: 0.5962 - accuracy: 0.7476
 74/147 [==============>...............] - ETA: 11s - loss: 0.5952 - accuracy: 0.7478
 75/147 [==============>...............] - ETA: 11s - loss: 0.5944 - accuracy: 0.7483
 76/147 [==============>...............] - ETA: 11s - loss: 0.5944 - accuracy: 0.7477
 77/147 [==============>...............] - ETA: 11s - loss: 0.5941 - accuracy: 0.7470
 78/147 [==============>...............] - ETA: 10s - loss: 0.5939 - accuracy: 0.7464
 79/147 [===============>..............] - ETA: 10s - loss: 0.5928 - accuracy: 0.7472
 80/147 [===============>..............] - ETA: 10s - loss: 0.5921 - accuracy: 0.7471
 81/147 [===============>..............] - ETA: 10s - loss: 0.5923 - accuracy: 0.7461
 82/147 [===============>..............] - ETA: 10s - loss: 0.5921 - accuracy: 0.7455
 83/147 [===============>..............] - ETA: 10s - loss: 0.5916 - accuracy: 0.7452
 84/147 [================>.............] - ETA: 10s - loss: 0.5916 - accuracy: 0.7449
 85/147 [================>.............] - ETA: 9s - loss: 0.5913 - accuracy: 0.7447 
 86/147 [================>.............] - ETA: 9s - loss: 0.5913 - accuracy: 0.7438
 87/147 [================>.............] - ETA: 9s - loss: 0.5910 - accuracy: 0.7436
 88/147 [================>.............] - ETA: 9s - loss: 0.5905 - accuracy: 0.7434
 89/147 [=================>............] - ETA: 9s - loss: 0.5907 - accuracy: 0.7428
 90/147 [=================>............] - ETA: 9s - loss: 0.5900 - accuracy: 0.7433
 91/147 [=================>............] - ETA: 8s - loss: 0.5900 - accuracy: 0.7431
 92/147 [=================>............] - ETA: 8s - loss: 0.5899 - accuracy: 0.7426
 93/147 [=================>............] - ETA: 8s - loss: 0.5892 - accuracy: 0.7427
 94/147 [==================>...........] - ETA: 8s - loss: 0.5889 - accuracy: 0.7423
 95/147 [==================>...........] - ETA: 8s - loss: 0.5889 - accuracy: 0.7417
 96/147 [==================>...........] - ETA: 8s - loss: 0.5880 - accuracy: 0.7420
 97/147 [==================>...........] - ETA: 7s - loss: 0.5884 - accuracy: 0.7411
 98/147 [===================>..........] - ETA: 7s - loss: 0.5879 - accuracy: 0.7410
 99/147 [===================>..........] - ETA: 7s - loss: 0.5885 - accuracy: 0.7400
100/147 [===================>..........] - ETA: 7s - loss: 0.5884 - accuracy: 0.7394
101/147 [===================>..........] - ETA: 7s - loss: 0.5883 - accuracy: 0.7389
102/147 [===================>..........] - ETA: 7s - loss: 0.5884 - accuracy: 0.7388
103/147 [====================>.........] - ETA: 7s - loss: 0.5881 - accuracy: 0.7389
104/147 [====================>.........] - ETA: 6s - loss: 0.5878 - accuracy: 0.7389
105/147 [====================>.........] - ETA: 6s - loss: 0.5876 - accuracy: 0.7389
106/147 [====================>.........] - ETA: 6s - loss: 0.5875 - accuracy: 0.7385
107/147 [====================>.........] - ETA: 6s - loss: 0.5875 - accuracy: 0.7377
108/147 [=====================>........] - ETA: 6s - loss: 0.5877 - accuracy: 0.7372
109/147 [=====================>........] - ETA: 6s - loss: 0.5875 - accuracy: 0.7367
110/147 [=====================>........] - ETA: 5s - loss: 0.5875 - accuracy: 0.7361
111/147 [=====================>........] - ETA: 5s - loss: 0.5874 - accuracy: 0.7355
112/147 [=====================>........] - ETA: 5s - loss: 0.5876 - accuracy: 0.7347
113/147 [======================>.......] - ETA: 5s - loss: 0.5874 - accuracy: 0.7344
114/147 [======================>.......] - ETA: 5s - loss: 0.5875 - accuracy: 0.7342
115/147 [======================>.......] - ETA: 5s - loss: 0.5870 - accuracy: 0.7340
116/147 [======================>.......] - ETA: 5s - loss: 0.5872 - accuracy: 0.7330
117/147 [======================>.......] - ETA: 4s - loss: 0.5871 - accuracy: 0.7329
118/147 [=======================>......] - ETA: 4s - loss: 0.5867 - accuracy: 0.7330
119/147 [=======================>......] - ETA: 4s - loss: 0.5865 - accuracy: 0.7326
120/147 [=======================>......] - ETA: 4s - loss: 0.5864 - accuracy: 0.7320
121/147 [=======================>......] - ETA: 4s - loss: 0.5861 - accuracy: 0.7321
122/147 [=======================>......] - ETA: 4s - loss: 0.5861 - accuracy: 0.7317
123/147 [========================>.....] - ETA: 3s - loss: 0.5856 - accuracy: 0.7322
124/147 [========================>.....] - ETA: 3s - loss: 0.5852 - accuracy: 0.7322
125/147 [========================>.....] - ETA: 3s - loss: 0.5851 - accuracy: 0.7319
126/147 [========================>.....] - ETA: 3s - loss: 0.5849 - accuracy: 0.7320
127/147 [========================>.....] - ETA: 3s - loss: 0.5853 - accuracy: 0.7315
128/147 [=========================>....] - ETA: 3s - loss: 0.5852 - accuracy: 0.7316
129/147 [=========================>....] - ETA: 2s - loss: 0.5853 - accuracy: 0.7312
130/147 [=========================>....] - ETA: 2s - loss: 0.5854 - accuracy: 0.7308
131/147 [=========================>....] - ETA: 2s - loss: 0.5854 - accuracy: 0.7300
132/147 [=========================>....] - ETA: 2s - loss: 0.5852 - accuracy: 0.7299
133/147 [==========================>...] - ETA: 2s - loss: 0.5853 - accuracy: 0.7295
134/147 [==========================>...] - ETA: 2s - loss: 0.5855 - accuracy: 0.7292
135/147 [==========================>...] - ETA: 1s - loss: 0.5854 - accuracy: 0.7289
136/147 [==========================>...] - ETA: 1s - loss: 0.5855 - accuracy: 0.7282
137/147 [==========================>...] - ETA: 1s - loss: 0.5855 - accuracy: 0.7282
138/147 [===========================>..] - ETA: 1s - loss: 0.5853 - accuracy: 0.7280
139/147 [===========================>..] - ETA: 1s - loss: 0.5851 - accuracy: 0.7280
140/147 [===========================>..] - ETA: 1s - loss: 0.5853 - accuracy: 0.7274
141/147 [===========================>..] - ETA: 0s - loss: 0.5850 - accuracy: 0.7273
142/147 [===========================>..] - ETA: 0s - loss: 0.5846 - accuracy: 0.7274
143/147 [============================>.] - ETA: 0s - loss: 0.5845 - accuracy: 0.7271
144/147 [============================>.] - ETA: 0s - loss: 0.5843 - accuracy: 0.7271
145/147 [============================>.] - ETA: 0s - loss: 0.5841 - accuracy: 0.7271
146/147 [============================>.] - ETA: 0s - loss: 0.5838 - accuracy: 0.7272
147/147 [==============================] - 24s 162ms/step - loss: 0.5836 - accuracy: 0.7271

147/147 [==============================] - 26s 176ms/step - loss: 0.5836 - accuracy: 0.7271 - val_loss: 0.6386 - val_accuracy: 0.6221
Epoch 3/10

  1/147 [..............................] - ETA: 0s - loss: 0.4413 - accuracy: 0.8750
  2/147 [..............................] - ETA: 8s - loss: 0.4173 - accuracy: 0.9023
  3/147 [..............................] - ETA: 11s - loss: 0.4221 - accuracy: 0.8932
  4/147 [..............................] - ETA: 13s - loss: 0.4287 - accuracy: 0.8848
  5/147 [>.............................] - ETA: 14s - loss: 0.4309 - accuracy: 0.8766
  6/147 [>.............................] - ETA: 15s - loss: 0.4360 - accuracy: 0.8711
  7/147 [>.............................] - ETA: 15s - loss: 0.4376 - accuracy: 0.8717
  8/147 [>.............................] - ETA: 15s - loss: 0.4357 - accuracy: 0.8711
  9/147 [>.............................] - ETA: 15s - loss: 0.4353 - accuracy: 0.8715
 10/147 [=>............................] - ETA: 15s - loss: 0.4321 - accuracy: 0.8734
 11/147 [=>............................] - ETA: 15s - loss: 0.4297 - accuracy: 0.8771
 12/147 [=>............................] - ETA: 15s - loss: 0.4266 - accuracy: 0.8802
 13/147 [=>............................] - ETA: 16s - loss: 0.4235 - accuracy: 0.8804
 14/147 [=>............................] - ETA: 16s - loss: 0.4236 - accuracy: 0.8778
 15/147 [==>...........................] - ETA: 15s - loss: 0.4217 - accuracy: 0.8797
 16/147 [==>...........................] - ETA: 15s - loss: 0.4210 - accuracy: 0.8794
 17/147 [==>...........................] - ETA: 15s - loss: 0.4228 - accuracy: 0.8778
 18/147 [==>...........................] - ETA: 15s - loss: 0.4215 - accuracy: 0.8789
 19/147 [==>...........................] - ETA: 15s - loss: 0.4202 - accuracy: 0.8787
 20/147 [===>..........................] - ETA: 15s - loss: 0.4208 - accuracy: 0.8773
 21/147 [===>..........................] - ETA: 15s - loss: 0.4192 - accuracy: 0.8772
 22/147 [===>..........................] - ETA: 15s - loss: 0.4185 - accuracy: 0.8764
 23/147 [===>..........................] - ETA: 15s - loss: 0.4177 - accuracy: 0.8747
 24/147 [===>..........................] - ETA: 15s - loss: 0.4159 - accuracy: 0.8747
 25/147 [====>.........................] - ETA: 15s - loss: 0.4148 - accuracy: 0.8741
 26/147 [====>.........................] - ETA: 14s - loss: 0.4134 - accuracy: 0.8753
 27/147 [====>.........................] - ETA: 14s - loss: 0.4131 - accuracy: 0.8744
 28/147 [====>.........................] - ETA: 15s - loss: 0.4115 - accuracy: 0.8742
 29/147 [====>.........................] - ETA: 15s - loss: 0.4100 - accuracy: 0.8753
 30/147 [=====>........................] - ETA: 14s - loss: 0.4085 - accuracy: 0.8747
 31/147 [=====>........................] - ETA: 14s - loss: 0.4077 - accuracy: 0.8737
 32/147 [=====>........................] - ETA: 14s - loss: 0.4072 - accuracy: 0.8721
 33/147 [=====>........................] - ETA: 14s - loss: 0.4060 - accuracy: 0.8726
 34/147 [=====>........................] - ETA: 14s - loss: 0.4059 - accuracy: 0.8727
 35/147 [======>.......................] - ETA: 14s - loss: 0.4039 - accuracy: 0.8734
 36/147 [======>.......................] - ETA: 14s - loss: 0.4026 - accuracy: 0.8726
 37/147 [======>.......................] - ETA: 14s - loss: 0.4013 - accuracy: 0.8727
 38/147 [======>.......................] - ETA: 14s - loss: 0.4002 - accuracy: 0.8725
 39/147 [======>.......................] - ETA: 14s - loss: 0.4003 - accuracy: 0.8718
 40/147 [=======>......................] - ETA: 14s - loss: 0.4005 - accuracy: 0.8703
 41/147 [=======>......................] - ETA: 14s - loss: 0.3999 - accuracy: 0.8699
 42/147 [=======>......................] - ETA: 14s - loss: 0.3989 - accuracy: 0.8702
 43/147 [=======>......................] - ETA: 14s - loss: 0.3984 - accuracy: 0.8699
 44/147 [=======>......................] - ETA: 14s - loss: 0.3974 - accuracy: 0.8699
 45/147 [========>.....................] - ETA: 14s - loss: 0.3967 - accuracy: 0.8701
 46/147 [========>.....................] - ETA: 14s - loss: 0.3957 - accuracy: 0.8706
 47/147 [========>.....................] - ETA: 13s - loss: 0.3956 - accuracy: 0.8705
 48/147 [========>.....................] - ETA: 13s - loss: 0.3947 - accuracy: 0.8708
 49/147 [=========>....................] - ETA: 13s - loss: 0.3941 - accuracy: 0.8705
 50/147 [=========>....................] - ETA: 13s - loss: 0.3934 - accuracy: 0.8708
 51/147 [=========>....................] - ETA: 13s - loss: 0.3928 - accuracy: 0.8707
 52/147 [=========>....................] - ETA: 13s - loss: 0.3922 - accuracy: 0.8706
 53/147 [=========>....................] - ETA: 12s - loss: 0.3926 - accuracy: 0.8700
 54/147 [==========>...................] - ETA: 12s - loss: 0.3915 - accuracy: 0.8707
 55/147 [==========>...................] - ETA: 12s - loss: 0.3908 - accuracy: 0.8706
 56/147 [==========>...................] - ETA: 12s - loss: 0.3897 - accuracy: 0.8710
 57/147 [==========>...................] - ETA: 12s - loss: 0.3891 - accuracy: 0.8708
 58/147 [==========>...................] - ETA: 12s - loss: 0.3880 - accuracy: 0.8715
 59/147 [===========>..................] - ETA: 12s - loss: 0.3873 - accuracy: 0.8712
 60/147 [===========>..................] - ETA: 12s - loss: 0.3878 - accuracy: 0.8698
 61/147 [===========>..................] - ETA: 12s - loss: 0.3870 - accuracy: 0.8700
 62/147 [===========>..................] - ETA: 12s - loss: 0.3862 - accuracy: 0.8703
 63/147 [===========>..................] - ETA: 12s - loss: 0.3856 - accuracy: 0.8705
 64/147 [============>.................] - ETA: 11s - loss: 0.3852 - accuracy: 0.8700
 65/147 [============>.................] - ETA: 11s - loss: 0.3850 - accuracy: 0.8696
 66/147 [============>.................] - ETA: 11s - loss: 0.3847 - accuracy: 0.8693
 67/147 [============>.................] - ETA: 11s - loss: 0.3840 - accuracy: 0.8700
 68/147 [============>.................] - ETA: 11s - loss: 0.3836 - accuracy: 0.8701
 69/147 [=============>................] - ETA: 11s - loss: 0.3837 - accuracy: 0.8697
 70/147 [=============>................] - ETA: 11s - loss: 0.3821 - accuracy: 0.8705
 71/147 [=============>................] - ETA: 11s - loss: 0.3811 - accuracy: 0.8709
 72/147 [=============>................] - ETA: 10s - loss: 0.3801 - accuracy: 0.8712
 73/147 [=============>................] - ETA: 10s - loss: 0.3800 - accuracy: 0.8711
 74/147 [==============>...............] - ETA: 10s - loss: 0.3789 - accuracy: 0.8717
 75/147 [==============>...............] - ETA: 10s - loss: 0.3781 - accuracy: 0.8724
 76/147 [==============>...............] - ETA: 10s - loss: 0.3778 - accuracy: 0.8719
 77/147 [==============>...............] - ETA: 10s - loss: 0.3775 - accuracy: 0.8714
 78/147 [==============>...............] - ETA: 10s - loss: 0.3778 - accuracy: 0.8710
 79/147 [===============>..............] - ETA: 9s - loss: 0.3771 - accuracy: 0.8711 
 80/147 [===============>..............] - ETA: 9s - loss: 0.3767 - accuracy: 0.8706
 81/147 [===============>..............] - ETA: 9s - loss: 0.3764 - accuracy: 0.8709
 82/147 [===============>..............] - ETA: 9s - loss: 0.3758 - accuracy: 0.8712
 83/147 [===============>..............] - ETA: 9s - loss: 0.3756 - accuracy: 0.8707
 84/147 [================>.............] - ETA: 9s - loss: 0.3759 - accuracy: 0.8703
 85/147 [================>.............] - ETA: 9s - loss: 0.3750 - accuracy: 0.8708
 86/147 [================>.............] - ETA: 9s - loss: 0.3741 - accuracy: 0.8709
 87/147 [================>.............] - ETA: 8s - loss: 0.3737 - accuracy: 0.8710
 88/147 [================>.............] - ETA: 8s - loss: 0.3734 - accuracy: 0.8712
 89/147 [=================>............] - ETA: 8s - loss: 0.3735 - accuracy: 0.8708
 90/147 [=================>............] - ETA: 8s - loss: 0.3731 - accuracy: 0.8707
 91/147 [=================>............] - ETA: 8s - loss: 0.3733 - accuracy: 0.8703
 92/147 [=================>............] - ETA: 8s - loss: 0.3729 - accuracy: 0.8704
 93/147 [=================>............] - ETA: 7s - loss: 0.3722 - accuracy: 0.8706
 94/147 [==================>...........] - ETA: 7s - loss: 0.3725 - accuracy: 0.8701
 95/147 [==================>...........] - ETA: 7s - loss: 0.3719 - accuracy: 0.8703
 96/147 [==================>...........] - ETA: 7s - loss: 0.3715 - accuracy: 0.8704
 97/147 [==================>...........] - ETA: 7s - loss: 0.3708 - accuracy: 0.8709
 98/147 [===================>..........] - ETA: 7s - loss: 0.3705 - accuracy: 0.8707
 99/147 [===================>..........] - ETA: 7s - loss: 0.3700 - accuracy: 0.8709
100/147 [===================>..........] - ETA: 6s - loss: 0.3697 - accuracy: 0.8706
101/147 [===================>..........] - ETA: 6s - loss: 0.3693 - accuracy: 0.8707
102/147 [===================>..........] - ETA: 6s - loss: 0.3691 - accuracy: 0.8706
103/147 [====================>.........] - ETA: 6s - loss: 0.3688 - accuracy: 0.8710
104/147 [====================>.........] - ETA: 6s - loss: 0.3686 - accuracy: 0.8708
105/147 [====================>.........] - ETA: 6s - loss: 0.3683 - accuracy: 0.8709
106/147 [====================>.........] - ETA: 6s - loss: 0.3681 - accuracy: 0.8706
107/147 [====================>.........] - ETA: 5s - loss: 0.3676 - accuracy: 0.8703
108/147 [=====================>........] - ETA: 5s - loss: 0.3675 - accuracy: 0.8700
109/147 [=====================>........] - ETA: 5s - loss: 0.3668 - accuracy: 0.8700
110/147 [=====================>........] - ETA: 5s - loss: 0.3667 - accuracy: 0.8698
111/147 [=====================>........] - ETA: 5s - loss: 0.3661 - accuracy: 0.8701
112/147 [=====================>........] - ETA: 5s - loss: 0.3660 - accuracy: 0.8700
113/147 [======================>.......] - ETA: 5s - loss: 0.3660 - accuracy: 0.8697
114/147 [======================>.......] - ETA: 4s - loss: 0.3652 - accuracy: 0.8701
115/147 [======================>.......] - ETA: 4s - loss: 0.3644 - accuracy: 0.8704
116/147 [======================>.......] - ETA: 4s - loss: 0.3650 - accuracy: 0.8697
117/147 [======================>.......] - ETA: 4s - loss: 0.3647 - accuracy: 0.8695
118/147 [=======================>......] - ETA: 4s - loss: 0.3647 - accuracy: 0.8694
119/147 [=======================>......] - ETA: 4s - loss: 0.3644 - accuracy: 0.8695
120/147 [=======================>......] - ETA: 4s - loss: 0.3643 - accuracy: 0.8693
121/147 [=======================>......] - ETA: 3s - loss: 0.3647 - accuracy: 0.8691
122/147 [=======================>......] - ETA: 3s - loss: 0.3647 - accuracy: 0.8690
123/147 [========================>.....] - ETA: 3s - loss: 0.3648 - accuracy: 0.8685
124/147 [========================>.....] - ETA: 3s - loss: 0.3648 - accuracy: 0.8683
125/147 [========================>.....] - ETA: 3s - loss: 0.3648 - accuracy: 0.8680
126/147 [========================>.....] - ETA: 3s - loss: 0.3647 - accuracy: 0.8677
127/147 [========================>.....] - ETA: 3s - loss: 0.3645 - accuracy: 0.8676
128/147 [=========================>....] - ETA: 2s - loss: 0.3645 - accuracy: 0.8679
129/147 [=========================>....] - ETA: 2s - loss: 0.3650 - accuracy: 0.8676
130/147 [=========================>....] - ETA: 2s - loss: 0.3651 - accuracy: 0.8674
131/147 [=========================>....] - ETA: 2s - loss: 0.3653 - accuracy: 0.8674
132/147 [=========================>....] - ETA: 2s - loss: 0.3647 - accuracy: 0.8678
133/147 [==========================>...] - ETA: 2s - loss: 0.3648 - accuracy: 0.8675
134/147 [==========================>...] - ETA: 1s - loss: 0.3647 - accuracy: 0.8672
135/147 [==========================>...] - ETA: 1s - loss: 0.3647 - accuracy: 0.8671
136/147 [==========================>...] - ETA: 1s - loss: 0.3646 - accuracy: 0.8671
137/147 [==========================>...] - ETA: 1s - loss: 0.3648 - accuracy: 0.8667
138/147 [===========================>..] - ETA: 1s - loss: 0.3647 - accuracy: 0.8667
139/147 [===========================>..] - ETA: 1s - loss: 0.3651 - accuracy: 0.8660
140/147 [===========================>..] - ETA: 1s - loss: 0.3653 - accuracy: 0.8655
141/147 [===========================>..] - ETA: 0s - loss: 0.3646 - accuracy: 0.8656
142/147 [===========================>..] - ETA: 0s - loss: 0.3645 - accuracy: 0.8655
143/147 [============================>.] - ETA: 0s - loss: 0.3642 - accuracy: 0.8657
144/147 [============================>.] - ETA: 0s - loss: 0.3638 - accuracy: 0.8657
145/147 [============================>.] - ETA: 0s - loss: 0.3636 - accuracy: 0.8657
146/147 [============================>.] - ETA: 0s - loss: 0.3640 - accuracy: 0.8653
147/147 [==============================] - 22s 153ms/step - loss: 0.3639 - accuracy: 0.8652

147/147 [==============================] - 24s 165ms/step - loss: 0.3639 - accuracy: 0.8652 - val_loss: 0.7341 - val_accuracy: 0.6195
Epoch 4/10

  1/147 [..............................] - ETA: 0s - loss: 0.1819 - accuracy: 0.9609
  2/147 [..............................] - ETA: 13s - loss: 0.1877 - accuracy: 0.9648
  3/147 [..............................] - ETA: 16s - loss: 0.1934 - accuracy: 0.9583
  4/147 [..............................] - ETA: 17s - loss: 0.1838 - accuracy: 0.9551
  5/147 [>.............................] - ETA: 17s - loss: 0.1883 - accuracy: 0.9547
  6/147 [>.............................] - ETA: 18s - loss: 0.1913 - accuracy: 0.9518
  7/147 [>.............................] - ETA: 18s - loss: 0.1861 - accuracy: 0.9520
  8/147 [>.............................] - ETA: 17s - loss: 0.1862 - accuracy: 0.9561
  9/147 [>.............................] - ETA: 17s - loss: 0.1837 - accuracy: 0.9575
 10/147 [=>............................] - ETA: 17s - loss: 0.1837 - accuracy: 0.9578
 11/147 [=>............................] - ETA: 17s - loss: 0.1819 - accuracy: 0.9588
 12/147 [=>............................] - ETA: 17s - loss: 0.1814 - accuracy: 0.9583
 13/147 [=>............................] - ETA: 17s - loss: 0.1789 - accuracy: 0.9591
 14/147 [=>............................] - ETA: 17s - loss: 0.1807 - accuracy: 0.9565
 15/147 [==>...........................] - ETA: 17s - loss: 0.1800 - accuracy: 0.9573
 16/147 [==>...........................] - ETA: 17s - loss: 0.1773 - accuracy: 0.9585
 17/147 [==>...........................] - ETA: 17s - loss: 0.1753 - accuracy: 0.9596
 18/147 [==>...........................] - ETA: 17s - loss: 0.1741 - accuracy: 0.9601
 19/147 [==>...........................] - ETA: 17s - loss: 0.1741 - accuracy: 0.9597
 20/147 [===>..........................] - ETA: 16s - loss: 0.1735 - accuracy: 0.9609
 21/147 [===>..........................] - ETA: 16s - loss: 0.1715 - accuracy: 0.9613
 22/147 [===>..........................] - ETA: 16s - loss: 0.1705 - accuracy: 0.9620
 23/147 [===>..........................] - ETA: 16s - loss: 0.1706 - accuracy: 0.9620
 24/147 [===>..........................] - ETA: 16s - loss: 0.1706 - accuracy: 0.9609
 25/147 [====>.........................] - ETA: 16s - loss: 0.1712 - accuracy: 0.9613
 26/147 [====>.........................] - ETA: 16s - loss: 0.1703 - accuracy: 0.9612
 27/147 [====>.........................] - ETA: 15s - loss: 0.1681 - accuracy: 0.9621
 28/147 [====>.........................] - ETA: 15s - loss: 0.1676 - accuracy: 0.9618
 29/147 [====>.........................] - ETA: 15s - loss: 0.1669 - accuracy: 0.9617
 30/147 [=====>........................] - ETA: 15s - loss: 0.1665 - accuracy: 0.9615
 31/147 [=====>........................] - ETA: 15s - loss: 0.1659 - accuracy: 0.9617
 32/147 [=====>........................] - ETA: 15s - loss: 0.1649 - accuracy: 0.9619
 33/147 [=====>........................] - ETA: 15s - loss: 0.1638 - accuracy: 0.9624
 34/147 [=====>........................] - ETA: 15s - loss: 0.1634 - accuracy: 0.9628
 35/147 [======>.......................] - ETA: 15s - loss: 0.1641 - accuracy: 0.9621
 36/147 [======>.......................] - ETA: 15s - loss: 0.1644 - accuracy: 0.9616
 37/147 [======>.......................] - ETA: 15s - loss: 0.1648 - accuracy: 0.9611
 38/147 [======>.......................] - ETA: 15s - loss: 0.1638 - accuracy: 0.9616
 39/147 [======>.......................] - ETA: 14s - loss: 0.1636 - accuracy: 0.9613
 40/147 [=======>......................] - ETA: 14s - loss: 0.1632 - accuracy: 0.9611
 41/147 [=======>......................] - ETA: 14s - loss: 0.1627 - accuracy: 0.9613
 42/147 [=======>......................] - ETA: 14s - loss: 0.1625 - accuracy: 0.9619
 43/147 [=======>......................] - ETA: 14s - loss: 0.1618 - accuracy: 0.9624
 44/147 [=======>......................] - ETA: 14s - loss: 0.1613 - accuracy: 0.9624
 45/147 [========>.....................] - ETA: 14s - loss: 0.1621 - accuracy: 0.9616
 46/147 [========>.....................] - ETA: 14s - loss: 0.1625 - accuracy: 0.9613
 47/147 [========>.....................] - ETA: 14s - loss: 0.1615 - accuracy: 0.9618
 48/147 [========>.....................] - ETA: 13s - loss: 0.1609 - accuracy: 0.9619
 49/147 [=========>....................] - ETA: 13s - loss: 0.1609 - accuracy: 0.9617
 50/147 [=========>....................] - ETA: 13s - loss: 0.1608 - accuracy: 0.9619
 51/147 [=========>....................] - ETA: 13s - loss: 0.1601 - accuracy: 0.9622
 52/147 [=========>....................] - ETA: 13s - loss: 0.1597 - accuracy: 0.9620
 53/147 [=========>....................] - ETA: 13s - loss: 0.1597 - accuracy: 0.9618
 54/147 [==========>...................] - ETA: 12s - loss: 0.1589 - accuracy: 0.9620
 55/147 [==========>...................] - ETA: 12s - loss: 0.1593 - accuracy: 0.9614
 56/147 [==========>...................] - ETA: 12s - loss: 0.1586 - accuracy: 0.9618
 57/147 [==========>...................] - ETA: 12s - loss: 0.1592 - accuracy: 0.9618
 58/147 [==========>...................] - ETA: 12s - loss: 0.1590 - accuracy: 0.9615
 59/147 [===========>..................] - ETA: 12s - loss: 0.1584 - accuracy: 0.9615
 60/147 [===========>..................] - ETA: 12s - loss: 0.1578 - accuracy: 0.9618
 61/147 [===========>..................] - ETA: 12s - loss: 0.1571 - accuracy: 0.9621
 62/147 [===========>..................] - ETA: 11s - loss: 0.1569 - accuracy: 0.9618
 63/147 [===========>..................] - ETA: 11s - loss: 0.1561 - accuracy: 0.9623
 64/147 [============>.................] - ETA: 11s - loss: 0.1555 - accuracy: 0.9622
 65/147 [============>.................] - ETA: 11s - loss: 0.1553 - accuracy: 0.9621
 66/147 [============>.................] - ETA: 11s - loss: 0.1553 - accuracy: 0.9622
 67/147 [============>.................] - ETA: 11s - loss: 0.1556 - accuracy: 0.9622
 68/147 [============>.................] - ETA: 11s - loss: 0.1559 - accuracy: 0.9617
 69/147 [=============>................] - ETA: 10s - loss: 0.1557 - accuracy: 0.9614
 70/147 [=============>................] - ETA: 10s - loss: 0.1561 - accuracy: 0.9608
 71/147 [=============>................] - ETA: 10s - loss: 0.1554 - accuracy: 0.9613
 72/147 [=============>................] - ETA: 10s - loss: 0.1550 - accuracy: 0.9613
 73/147 [=============>................] - ETA: 10s - loss: 0.1549 - accuracy: 0.9610
 74/147 [==============>...............] - ETA: 10s - loss: 0.1547 - accuracy: 0.9609
 75/147 [==============>...............] - ETA: 10s - loss: 0.1544 - accuracy: 0.9608
 76/147 [==============>...............] - ETA: 10s - loss: 0.1541 - accuracy: 0.9608
 77/147 [==============>...............] - ETA: 10s - loss: 0.1538 - accuracy: 0.9607
 78/147 [==============>...............] - ETA: 9s - loss: 0.1536 - accuracy: 0.9608 
 79/147 [===============>..............] - ETA: 9s - loss: 0.1536 - accuracy: 0.9609
 80/147 [===============>..............] - ETA: 9s - loss: 0.1537 - accuracy: 0.9608
 81/147 [===============>..............] - ETA: 9s - loss: 0.1538 - accuracy: 0.9606
 82/147 [===============>..............] - ETA: 9s - loss: 0.1533 - accuracy: 0.9607
 83/147 [===============>..............] - ETA: 9s - loss: 0.1529 - accuracy: 0.9607
 84/147 [================>.............] - ETA: 9s - loss: 0.1521 - accuracy: 0.9608
 85/147 [================>.............] - ETA: 9s - loss: 0.1514 - accuracy: 0.9612
 86/147 [================>.............] - ETA: 9s - loss: 0.1508 - accuracy: 0.9614
 87/147 [================>.............] - ETA: 8s - loss: 0.1506 - accuracy: 0.9616
 88/147 [================>.............] - ETA: 8s - loss: 0.1507 - accuracy: 0.9615
 89/147 [=================>............] - ETA: 8s - loss: 0.1501 - accuracy: 0.9616
 90/147 [=================>............] - ETA: 8s - loss: 0.1497 - accuracy: 0.9616
 91/147 [=================>............] - ETA: 8s - loss: 0.1491 - accuracy: 0.9618
 92/147 [=================>............] - ETA: 8s - loss: 0.1486 - accuracy: 0.9618
 93/147 [=================>............] - ETA: 8s - loss: 0.1485 - accuracy: 0.9617
 94/147 [==================>...........] - ETA: 7s - loss: 0.1489 - accuracy: 0.9614
 95/147 [==================>...........] - ETA: 7s - loss: 0.1488 - accuracy: 0.9613
 96/147 [==================>...........] - ETA: 7s - loss: 0.1488 - accuracy: 0.9613
 97/147 [==================>...........] - ETA: 7s - loss: 0.1483 - accuracy: 0.9616
 98/147 [===================>..........] - ETA: 7s - loss: 0.1481 - accuracy: 0.9614
 99/147 [===================>..........] - ETA: 7s - loss: 0.1478 - accuracy: 0.9616
100/147 [===================>..........] - ETA: 7s - loss: 0.1477 - accuracy: 0.9616
101/147 [===================>..........] - ETA: 6s - loss: 0.1474 - accuracy: 0.9616
102/147 [===================>..........] - ETA: 6s - loss: 0.1475 - accuracy: 0.9614
103/147 [====================>.........] - ETA: 6s - loss: 0.1474 - accuracy: 0.9614
104/147 [====================>.........] - ETA: 6s - loss: 0.1471 - accuracy: 0.9614
105/147 [====================>.........] - ETA: 6s - loss: 0.1469 - accuracy: 0.9615
106/147 [====================>.........] - ETA: 6s - loss: 0.1464 - accuracy: 0.9617
107/147 [====================>.........] - ETA: 5s - loss: 0.1460 - accuracy: 0.9618
108/147 [=====================>........] - ETA: 5s - loss: 0.1457 - accuracy: 0.9620
109/147 [=====================>........] - ETA: 5s - loss: 0.1456 - accuracy: 0.9619
110/147 [=====================>........] - ETA: 5s - loss: 0.1452 - accuracy: 0.9618
111/147 [=====================>........] - ETA: 5s - loss: 0.1450 - accuracy: 0.9619
112/147 [=====================>........] - ETA: 5s - loss: 0.1446 - accuracy: 0.9621
113/147 [======================>.......] - ETA: 5s - loss: 0.1445 - accuracy: 0.9621
114/147 [======================>.......] - ETA: 4s - loss: 0.1442 - accuracy: 0.9623
115/147 [======================>.......] - ETA: 4s - loss: 0.1441 - accuracy: 0.9622
116/147 [======================>.......] - ETA: 4s - loss: 0.1442 - accuracy: 0.9621
117/147 [======================>.......] - ETA: 4s - loss: 0.1443 - accuracy: 0.9620
118/147 [=======================>......] - ETA: 4s - loss: 0.1445 - accuracy: 0.9619
119/147 [=======================>......] - ETA: 4s - loss: 0.1448 - accuracy: 0.9617
120/147 [=======================>......] - ETA: 4s - loss: 0.1446 - accuracy: 0.9618
121/147 [=======================>......] - ETA: 3s - loss: 0.1446 - accuracy: 0.9617
122/147 [=======================>......] - ETA: 3s - loss: 0.1448 - accuracy: 0.9614
123/147 [========================>.....] - ETA: 3s - loss: 0.1447 - accuracy: 0.9613
124/147 [========================>.....] - ETA: 3s - loss: 0.1443 - accuracy: 0.9614
125/147 [========================>.....] - ETA: 3s - loss: 0.1442 - accuracy: 0.9614
126/147 [========================>.....] - ETA: 3s - loss: 0.1439 - accuracy: 0.9615
127/147 [========================>.....] - ETA: 3s - loss: 0.1443 - accuracy: 0.9613
128/147 [=========================>....] - ETA: 2s - loss: 0.1442 - accuracy: 0.9614
129/147 [=========================>....] - ETA: 2s - loss: 0.1441 - accuracy: 0.9614
130/147 [=========================>....] - ETA: 2s - loss: 0.1438 - accuracy: 0.9614
131/147 [=========================>....] - ETA: 2s - loss: 0.1440 - accuracy: 0.9614
132/147 [=========================>....] - ETA: 2s - loss: 0.1442 - accuracy: 0.9611
133/147 [==========================>...] - ETA: 2s - loss: 0.1442 - accuracy: 0.9609
134/147 [==========================>...] - ETA: 1s - loss: 0.1445 - accuracy: 0.9605
135/147 [==========================>...] - ETA: 1s - loss: 0.1442 - accuracy: 0.9606
136/147 [==========================>...] - ETA: 1s - loss: 0.1441 - accuracy: 0.9607
137/147 [==========================>...] - ETA: 1s - loss: 0.1443 - accuracy: 0.9604
138/147 [===========================>..] - ETA: 1s - loss: 0.1441 - accuracy: 0.9605
139/147 [===========================>..] - ETA: 1s - loss: 0.1440 - accuracy: 0.9604
140/147 [===========================>..] - ETA: 1s - loss: 0.1440 - accuracy: 0.9604
141/147 [===========================>..] - ETA: 0s - loss: 0.1441 - accuracy: 0.9603
142/147 [===========================>..] - ETA: 0s - loss: 0.1440 - accuracy: 0.9601
143/147 [============================>.] - ETA: 0s - loss: 0.1439 - accuracy: 0.9602
144/147 [============================>.] - ETA: 0s - loss: 0.1437 - accuracy: 0.9600
145/147 [============================>.] - ETA: 0s - loss: 0.1438 - accuracy: 0.9599
146/147 [============================>.] - ETA: 0s - loss: 0.1440 - accuracy: 0.9597
147/147 [==============================] - 22s 152ms/step - loss: 0.1438 - accuracy: 0.9597

147/147 [==============================] - 24s 166ms/step - loss: 0.1438 - accuracy: 0.9597 - val_loss: 0.9010 - val_accuracy: 0.6213
Epoch 5/10

  1/147 [..............................] - ETA: 0s - loss: 0.0371 - accuracy: 1.0000
  2/147 [..............................] - ETA: 12s - loss: 0.0417 - accuracy: 0.9961
  3/147 [..............................] - ETA: 15s - loss: 0.0431 - accuracy: 0.9974
  4/147 [..............................] - ETA: 17s - loss: 0.0430 - accuracy: 0.9980
  5/147 [>.............................] - ETA: 18s - loss: 0.0425 - accuracy: 0.9984
  6/147 [>.............................] - ETA: 19s - loss: 0.0420 - accuracy: 0.9987
  7/147 [>.............................] - ETA: 20s - loss: 0.0410 - accuracy: 0.9989
  8/147 [>.............................] - ETA: 19s - loss: 0.0419 - accuracy: 0.9990
  9/147 [>.............................] - ETA: 19s - loss: 0.0415 - accuracy: 0.9991
 10/147 [=>............................] - ETA: 19s - loss: 0.0421 - accuracy: 0.9984
 11/147 [=>............................] - ETA: 19s - loss: 0.0429 - accuracy: 0.9972
 12/147 [=>............................] - ETA: 19s - loss: 0.0433 - accuracy: 0.9961
 13/147 [=>............................] - ETA: 19s - loss: 0.0431 - accuracy: 0.9964
 14/147 [=>............................] - ETA: 19s - loss: 0.0437 - accuracy: 0.9955
 15/147 [==>...........................] - ETA: 19s - loss: 0.0442 - accuracy: 0.9953
 16/147 [==>...........................] - ETA: 18s - loss: 0.0439 - accuracy: 0.9956
 17/147 [==>...........................] - ETA: 18s - loss: 0.0434 - accuracy: 0.9959
 18/147 [==>...........................] - ETA: 18s - loss: 0.0430 - accuracy: 0.9961
 19/147 [==>...........................] - ETA: 18s - loss: 0.0426 - accuracy: 0.9963
 20/147 [===>..........................] - ETA: 17s - loss: 0.0428 - accuracy: 0.9961
 21/147 [===>..........................] - ETA: 17s - loss: 0.0426 - accuracy: 0.9959
 22/147 [===>..........................] - ETA: 17s - loss: 0.0431 - accuracy: 0.9954
 23/147 [===>..........................] - ETA: 17s - loss: 0.0427 - accuracy: 0.9956
 24/147 [===>..........................] - ETA: 17s - loss: 0.0420 - accuracy: 0.9958
 25/147 [====>.........................] - ETA: 17s - loss: 0.0414 - accuracy: 0.9959
 26/147 [====>.........................] - ETA: 17s - loss: 0.0412 - accuracy: 0.9961
 27/147 [====>.........................] - ETA: 16s - loss: 0.0410 - accuracy: 0.9962
 28/147 [====>.........................] - ETA: 16s - loss: 0.0414 - accuracy: 0.9958
 29/147 [====>.........................] - ETA: 16s - loss: 0.0413 - accuracy: 0.9960
 30/147 [=====>........................] - ETA: 16s - loss: 0.0419 - accuracy: 0.9956
 31/147 [=====>........................] - ETA: 16s - loss: 0.0417 - accuracy: 0.9957
 32/147 [=====>........................] - ETA: 16s - loss: 0.0412 - accuracy: 0.9958
 33/147 [=====>........................] - ETA: 15s - loss: 0.0409 - accuracy: 0.9957
 34/147 [=====>........................] - ETA: 15s - loss: 0.0411 - accuracy: 0.9956
 35/147 [======>.......................] - ETA: 15s - loss: 0.0407 - accuracy: 0.9958
 36/147 [======>.......................] - ETA: 15s - loss: 0.0407 - accuracy: 0.9954
 37/147 [======>.......................] - ETA: 15s - loss: 0.0410 - accuracy: 0.9954
 38/147 [======>.......................] - ETA: 15s - loss: 0.0411 - accuracy: 0.9953
 39/147 [======>.......................] - ETA: 15s - loss: 0.0407 - accuracy: 0.9954
 40/147 [=======>......................] - ETA: 14s - loss: 0.0406 - accuracy: 0.9955
 41/147 [=======>......................] - ETA: 14s - loss: 0.0404 - accuracy: 0.9956
 42/147 [=======>......................] - ETA: 14s - loss: 0.0404 - accuracy: 0.9957
 43/147 [=======>......................] - ETA: 14s - loss: 0.0401 - accuracy: 0.9958
 44/147 [=======>......................] - ETA: 14s - loss: 0.0400 - accuracy: 0.9957
 45/147 [========>.....................] - ETA: 14s - loss: 0.0400 - accuracy: 0.9957
 46/147 [========>.....................] - ETA: 14s - loss: 0.0396 - accuracy: 0.9958
 47/147 [========>.....................] - ETA: 14s - loss: 0.0395 - accuracy: 0.9958
 48/147 [========>.....................] - ETA: 13s - loss: 0.0393 - accuracy: 0.9958
 49/147 [=========>....................] - ETA: 13s - loss: 0.0393 - accuracy: 0.9957
 50/147 [=========>....................] - ETA: 13s - loss: 0.0392 - accuracy: 0.9958
 51/147 [=========>....................] - ETA: 13s - loss: 0.0394 - accuracy: 0.9956
 52/147 [=========>....................] - ETA: 13s - loss: 0.0394 - accuracy: 0.9955
 53/147 [=========>....................] - ETA: 13s - loss: 0.0392 - accuracy: 0.9956
 54/147 [==========>...................] - ETA: 13s - loss: 0.0394 - accuracy: 0.9955
 55/147 [==========>...................] - ETA: 13s - loss: 0.0394 - accuracy: 0.9955
 56/147 [==========>...................] - ETA: 12s - loss: 0.0392 - accuracy: 0.9955
 57/147 [==========>...................] - ETA: 12s - loss: 0.0389 - accuracy: 0.9956
 58/147 [==========>...................] - ETA: 12s - loss: 0.0389 - accuracy: 0.9957
 59/147 [===========>..................] - ETA: 12s - loss: 0.0392 - accuracy: 0.9956
 60/147 [===========>..................] - ETA: 12s - loss: 0.0392 - accuracy: 0.9956
 61/147 [===========>..................] - ETA: 12s - loss: 0.0392 - accuracy: 0.9956
 62/147 [===========>..................] - ETA: 12s - loss: 0.0390 - accuracy: 0.9957
 63/147 [===========>..................] - ETA: 12s - loss: 0.0388 - accuracy: 0.9957
 64/147 [============>.................] - ETA: 11s - loss: 0.0386 - accuracy: 0.9957
 65/147 [============>.................] - ETA: 11s - loss: 0.0389 - accuracy: 0.9956
 66/147 [============>.................] - ETA: 11s - loss: 0.0389 - accuracy: 0.9955
 67/147 [============>.................] - ETA: 11s - loss: 0.0388 - accuracy: 0.9955
 68/147 [============>.................] - ETA: 11s - loss: 0.0390 - accuracy: 0.9954
 69/147 [=============>................] - ETA: 11s - loss: 0.0389 - accuracy: 0.9955
 70/147 [=============>................] - ETA: 11s - loss: 0.0388 - accuracy: 0.9955
 71/147 [=============>................] - ETA: 11s - loss: 0.0387 - accuracy: 0.9955
 72/147 [=============>................] - ETA: 11s - loss: 0.0386 - accuracy: 0.9954
 73/147 [=============>................] - ETA: 11s - loss: 0.0386 - accuracy: 0.9955
 74/147 [==============>...............] - ETA: 10s - loss: 0.0388 - accuracy: 0.9954
 75/147 [==============>...............] - ETA: 10s - loss: 0.0386 - accuracy: 0.9954
 76/147 [==============>...............] - ETA: 10s - loss: 0.0385 - accuracy: 0.9955
 77/147 [==============>...............] - ETA: 10s - loss: 0.0384 - accuracy: 0.9955
 78/147 [==============>...............] - ETA: 10s - loss: 0.0382 - accuracy: 0.9956
 79/147 [===============>..............] - ETA: 10s - loss: 0.0381 - accuracy: 0.9956
 80/147 [===============>..............] - ETA: 10s - loss: 0.0381 - accuracy: 0.9956
 81/147 [===============>..............] - ETA: 9s - loss: 0.0379 - accuracy: 0.9957 
 82/147 [===============>..............] - ETA: 9s - loss: 0.0379 - accuracy: 0.9957
 83/147 [===============>..............] - ETA: 9s - loss: 0.0379 - accuracy: 0.9957
 84/147 [================>.............] - ETA: 9s - loss: 0.0378 - accuracy: 0.9956
 85/147 [================>.............] - ETA: 9s - loss: 0.0378 - accuracy: 0.9956
 86/147 [================>.............] - ETA: 9s - loss: 0.0382 - accuracy: 0.9955
 87/147 [================>.............] - ETA: 9s - loss: 0.0381 - accuracy: 0.9955
 88/147 [================>.............] - ETA: 8s - loss: 0.0380 - accuracy: 0.9956
 89/147 [=================>............] - ETA: 8s - loss: 0.0380 - accuracy: 0.9955
 90/147 [=================>............] - ETA: 8s - loss: 0.0378 - accuracy: 0.9956
 91/147 [=================>............] - ETA: 8s - loss: 0.0378 - accuracy: 0.9955
 92/147 [=================>............] - ETA: 8s - loss: 0.0378 - accuracy: 0.9955
 93/147 [=================>............] - ETA: 8s - loss: 0.0377 - accuracy: 0.9955
 94/147 [==================>...........] - ETA: 8s - loss: 0.0376 - accuracy: 0.9956
 95/147 [==================>...........] - ETA: 8s - loss: 0.0374 - accuracy: 0.9956
 96/147 [==================>...........] - ETA: 7s - loss: 0.0374 - accuracy: 0.9957
 97/147 [==================>...........] - ETA: 7s - loss: 0.0372 - accuracy: 0.9957
 98/147 [===================>..........] - ETA: 7s - loss: 0.0373 - accuracy: 0.9956
 99/147 [===================>..........] - ETA: 7s - loss: 0.0372 - accuracy: 0.9957
100/147 [===================>..........] - ETA: 7s - loss: 0.0370 - accuracy: 0.9957
101/147 [===================>..........] - ETA: 7s - loss: 0.0370 - accuracy: 0.9957
102/147 [===================>..........] - ETA: 6s - loss: 0.0369 - accuracy: 0.9958
103/147 [====================>.........] - ETA: 6s - loss: 0.0368 - accuracy: 0.9958
104/147 [====================>.........] - ETA: 6s - loss: 0.0367 - accuracy: 0.9959
105/147 [====================>.........] - ETA: 6s - loss: 0.0366 - accuracy: 0.9959
106/147 [====================>.........] - ETA: 6s - loss: 0.0365 - accuracy: 0.9959
107/147 [====================>.........] - ETA: 6s - loss: 0.0365 - accuracy: 0.9960
108/147 [=====================>........] - ETA: 6s - loss: 0.0364 - accuracy: 0.9959
109/147 [=====================>........] - ETA: 5s - loss: 0.0363 - accuracy: 0.9960
110/147 [=====================>........] - ETA: 5s - loss: 0.0363 - accuracy: 0.9960
111/147 [=====================>........] - ETA: 5s - loss: 0.0362 - accuracy: 0.9960
112/147 [=====================>........] - ETA: 5s - loss: 0.0360 - accuracy: 0.9960
113/147 [======================>.......] - ETA: 5s - loss: 0.0359 - accuracy: 0.9960
114/147 [======================>.......] - ETA: 5s - loss: 0.0360 - accuracy: 0.9959
115/147 [======================>.......] - ETA: 4s - loss: 0.0359 - accuracy: 0.9959
116/147 [======================>.......] - ETA: 4s - loss: 0.0359 - accuracy: 0.9958
117/147 [======================>.......] - ETA: 4s - loss: 0.0361 - accuracy: 0.9957
118/147 [=======================>......] - ETA: 4s - loss: 0.0360 - accuracy: 0.9957
119/147 [=======================>......] - ETA: 4s - loss: 0.0359 - accuracy: 0.9957
120/147 [=======================>......] - ETA: 4s - loss: 0.0358 - accuracy: 0.9958
121/147 [=======================>......] - ETA: 4s - loss: 0.0361 - accuracy: 0.9957
122/147 [=======================>......] - ETA: 3s - loss: 0.0361 - accuracy: 0.9956
123/147 [========================>.....] - ETA: 3s - loss: 0.0361 - accuracy: 0.9956
124/147 [========================>.....] - ETA: 3s - loss: 0.0360 - accuracy: 0.9957
125/147 [========================>.....] - ETA: 3s - loss: 0.0360 - accuracy: 0.9956
126/147 [========================>.....] - ETA: 3s - loss: 0.0359 - accuracy: 0.9957
127/147 [========================>.....] - ETA: 3s - loss: 0.0359 - accuracy: 0.9957
128/147 [=========================>....] - ETA: 2s - loss: 0.0357 - accuracy: 0.9957
129/147 [=========================>....] - ETA: 2s - loss: 0.0360 - accuracy: 0.9957
130/147 [=========================>....] - ETA: 2s - loss: 0.0359 - accuracy: 0.9957
131/147 [=========================>....] - ETA: 2s - loss: 0.0358 - accuracy: 0.9958
132/147 [=========================>....] - ETA: 2s - loss: 0.0357 - accuracy: 0.9958
133/147 [==========================>...] - ETA: 2s - loss: 0.0356 - accuracy: 0.9958
134/147 [==========================>...] - ETA: 2s - loss: 0.0356 - accuracy: 0.9959
135/147 [==========================>...] - ETA: 1s - loss: 0.0355 - accuracy: 0.9959
136/147 [==========================>...] - ETA: 1s - loss: 0.0354 - accuracy: 0.9959
137/147 [==========================>...] - ETA: 1s - loss: 0.0354 - accuracy: 0.9959
138/147 [===========================>..] - ETA: 1s - loss: 0.0354 - accuracy: 0.9959
139/147 [===========================>..] - ETA: 1s - loss: 0.0353 - accuracy: 0.9959
140/147 [===========================>..] - ETA: 1s - loss: 0.0352 - accuracy: 0.9959
141/147 [===========================>..] - ETA: 0s - loss: 0.0353 - accuracy: 0.9958
142/147 [===========================>..] - ETA: 0s - loss: 0.0353 - accuracy: 0.9957
143/147 [============================>.] - ETA: 0s - loss: 0.0353 - accuracy: 0.9957
144/147 [============================>.] - ETA: 0s - loss: 0.0352 - accuracy: 0.9957
145/147 [============================>.] - ETA: 0s - loss: 0.0352 - accuracy: 0.9957
146/147 [============================>.] - ETA: 0s - loss: 0.0352 - accuracy: 0.9957
147/147 [==============================] - 23s 155ms/step - loss: 0.0352 - accuracy: 0.9957

147/147 [==============================] - 25s 169ms/step - loss: 0.0352 - accuracy: 0.9957 - val_loss: 1.0697 - val_accuracy: 0.6170
Epoch 6/10

  1/147 [..............................] - ETA: 0s - loss: 0.0098 - accuracy: 1.0000
  2/147 [..............................] - ETA: 9s - loss: 0.0101 - accuracy: 1.0000
  3/147 [..............................] - ETA: 13s - loss: 0.0106 - accuracy: 1.0000
  4/147 [..............................] - ETA: 15s - loss: 0.0107 - accuracy: 1.0000
  5/147 [>.............................] - ETA: 16s - loss: 0.0108 - accuracy: 1.0000
  6/147 [>.............................] - ETA: 17s - loss: 0.0114 - accuracy: 1.0000
  7/147 [>.............................] - ETA: 17s - loss: 0.0114 - accuracy: 1.0000
  8/147 [>.............................] - ETA: 18s - loss: 0.0110 - accuracy: 1.0000
  9/147 [>.............................] - ETA: 18s - loss: 0.0109 - accuracy: 1.0000
 10/147 [=>............................] - ETA: 18s - loss: 0.0109 - accuracy: 1.0000
 11/147 [=>............................] - ETA: 18s - loss: 0.0110 - accuracy: 1.0000
 12/147 [=>............................] - ETA: 19s - loss: 0.0109 - accuracy: 1.0000
 13/147 [=>............................] - ETA: 18s - loss: 0.0108 - accuracy: 1.0000
 14/147 [=>............................] - ETA: 18s - loss: 0.0109 - accuracy: 1.0000
 15/147 [==>...........................] - ETA: 19s - loss: 0.0107 - accuracy: 1.0000
 16/147 [==>...........................] - ETA: 19s - loss: 0.0107 - accuracy: 1.0000
 17/147 [==>...........................] - ETA: 19s - loss: 0.0110 - accuracy: 1.0000
 18/147 [==>...........................] - ETA: 19s - loss: 0.0112 - accuracy: 1.0000
 19/147 [==>...........................] - ETA: 19s - loss: 0.0110 - accuracy: 1.0000
 20/147 [===>..........................] - ETA: 19s - loss: 0.0109 - accuracy: 1.0000
 21/147 [===>..........................] - ETA: 19s - loss: 0.0108 - accuracy: 1.0000
 22/147 [===>..........................] - ETA: 18s - loss: 0.0108 - accuracy: 1.0000
 23/147 [===>..........................] - ETA: 18s - loss: 0.0111 - accuracy: 1.0000
 24/147 [===>..........................] - ETA: 18s - loss: 0.0110 - accuracy: 1.0000
 25/147 [====>.........................] - ETA: 18s - loss: 0.0109 - accuracy: 1.0000
 26/147 [====>.........................] - ETA: 18s - loss: 0.0107 - accuracy: 1.0000
 27/147 [====>.........................] - ETA: 18s - loss: 0.0107 - accuracy: 1.0000
 28/147 [====>.........................] - ETA: 18s - loss: 0.0106 - accuracy: 1.0000
 29/147 [====>.........................] - ETA: 18s - loss: 0.0105 - accuracy: 1.0000
 30/147 [=====>........................] - ETA: 18s - loss: 0.0104 - accuracy: 1.0000
 31/147 [=====>........................] - ETA: 18s - loss: 0.0104 - accuracy: 1.0000
 32/147 [=====>........................] - ETA: 18s - loss: 0.0105 - accuracy: 1.0000
 33/147 [=====>........................] - ETA: 18s - loss: 0.0104 - accuracy: 1.0000
 34/147 [=====>........................] - ETA: 17s - loss: 0.0104 - accuracy: 1.0000
 35/147 [======>.......................] - ETA: 17s - loss: 0.0103 - accuracy: 1.0000
 36/147 [======>.......................] - ETA: 17s - loss: 0.0103 - accuracy: 1.0000
 37/147 [======>.......................] - ETA: 17s - loss: 0.0103 - accuracy: 1.0000
 38/147 [======>.......................] - ETA: 17s - loss: 0.0102 - accuracy: 1.0000
 39/147 [======>.......................] - ETA: 17s - loss: 0.0101 - accuracy: 1.0000
 40/147 [=======>......................] - ETA: 17s - loss: 0.0101 - accuracy: 1.0000
 41/147 [=======>......................] - ETA: 16s - loss: 0.0101 - accuracy: 1.0000
 42/147 [=======>......................] - ETA: 16s - loss: 0.0100 - accuracy: 1.0000
 43/147 [=======>......................] - ETA: 16s - loss: 0.0100 - accuracy: 1.0000
 44/147 [=======>......................] - ETA: 16s - loss: 0.0099 - accuracy: 1.0000
 45/147 [========>.....................] - ETA: 16s - loss: 0.0099 - accuracy: 1.0000
 46/147 [========>.....................] - ETA: 16s - loss: 0.0099 - accuracy: 1.0000
 47/147 [========>.....................] - ETA: 16s - loss: 0.0099 - accuracy: 1.0000
 48/147 [========>.....................] - ETA: 16s - loss: 0.0098 - accuracy: 1.0000
 49/147 [=========>....................] - ETA: 15s - loss: 0.0098 - accuracy: 1.0000
 50/147 [=========>....................] - ETA: 15s - loss: 0.0098 - accuracy: 1.0000
 51/147 [=========>....................] - ETA: 15s - loss: 0.0098 - accuracy: 1.0000
 52/147 [=========>....................] - ETA: 15s - loss: 0.0098 - accuracy: 1.0000
 53/147 [=========>....................] - ETA: 15s - loss: 0.0097 - accuracy: 1.0000
 54/147 [==========>...................] - ETA: 15s - loss: 0.0097 - accuracy: 1.0000
 55/147 [==========>...................] - ETA: 14s - loss: 0.0097 - accuracy: 1.0000
 56/147 [==========>...................] - ETA: 14s - loss: 0.0097 - accuracy: 1.0000
 57/147 [==========>...................] - ETA: 14s - loss: 0.0097 - accuracy: 1.0000
 58/147 [==========>...................] - ETA: 14s - loss: 0.0097 - accuracy: 1.0000
 59/147 [===========>..................] - ETA: 14s - loss: 0.0097 - accuracy: 1.0000
 60/147 [===========>..................] - ETA: 14s - loss: 0.0097 - accuracy: 1.0000
 61/147 [===========>..................] - ETA: 13s - loss: 0.0097 - accuracy: 1.0000
 62/147 [===========>..................] - ETA: 13s - loss: 0.0097 - accuracy: 1.0000
 63/147 [===========>..................] - ETA: 13s - loss: 0.0097 - accuracy: 1.0000
 64/147 [============>.................] - ETA: 13s - loss: 0.0097 - accuracy: 1.0000
 65/147 [============>.................] - ETA: 13s - loss: 0.0097 - accuracy: 1.0000
 66/147 [============>.................] - ETA: 13s - loss: 0.0097 - accuracy: 1.0000
 67/147 [============>.................] - ETA: 12s - loss: 0.0096 - accuracy: 1.0000
 68/147 [============>.................] - ETA: 12s - loss: 0.0096 - accuracy: 1.0000
 69/147 [=============>................] - ETA: 12s - loss: 0.0095 - accuracy: 1.0000
 70/147 [=============>................] - ETA: 12s - loss: 0.0095 - accuracy: 1.0000
 71/147 [=============>................] - ETA: 12s - loss: 0.0095 - accuracy: 1.0000
 72/147 [=============>................] - ETA: 12s - loss: 0.0095 - accuracy: 1.0000
 73/147 [=============>................] - ETA: 11s - loss: 0.0095 - accuracy: 1.0000
 74/147 [==============>...............] - ETA: 11s - loss: 0.0095 - accuracy: 1.0000
 75/147 [==============>...............] - ETA: 11s - loss: 0.0095 - accuracy: 1.0000
 76/147 [==============>...............] - ETA: 11s - loss: 0.0095 - accuracy: 1.0000
 77/147 [==============>...............] - ETA: 11s - loss: 0.0095 - accuracy: 1.0000
 78/147 [==============>...............] - ETA: 11s - loss: 0.0095 - accuracy: 1.0000
 79/147 [===============>..............] - ETA: 10s - loss: 0.0094 - accuracy: 1.0000
 80/147 [===============>..............] - ETA: 10s - loss: 0.0094 - accuracy: 1.0000
 81/147 [===============>..............] - ETA: 10s - loss: 0.0094 - accuracy: 1.0000
 82/147 [===============>..............] - ETA: 10s - loss: 0.0094 - accuracy: 1.0000
 83/147 [===============>..............] - ETA: 10s - loss: 0.0094 - accuracy: 1.0000
 84/147 [================>.............] - ETA: 10s - loss: 0.0094 - accuracy: 1.0000
 85/147 [================>.............] - ETA: 10s - loss: 0.0093 - accuracy: 1.0000
 86/147 [================>.............] - ETA: 9s - loss: 0.0093 - accuracy: 1.0000 
 87/147 [================>.............] - ETA: 9s - loss: 0.0093 - accuracy: 1.0000
 88/147 [================>.............] - ETA: 9s - loss: 0.0093 - accuracy: 1.0000
 89/147 [=================>............] - ETA: 9s - loss: 0.0092 - accuracy: 1.0000
 90/147 [=================>............] - ETA: 9s - loss: 0.0092 - accuracy: 1.0000
 91/147 [=================>............] - ETA: 9s - loss: 0.0092 - accuracy: 1.0000
 92/147 [=================>............] - ETA: 8s - loss: 0.0092 - accuracy: 1.0000
 93/147 [=================>............] - ETA: 8s - loss: 0.0092 - accuracy: 1.0000
 94/147 [==================>...........] - ETA: 8s - loss: 0.0092 - accuracy: 1.0000
 95/147 [==================>...........] - ETA: 8s - loss: 0.0091 - accuracy: 1.0000
 96/147 [==================>...........] - ETA: 8s - loss: 0.0091 - accuracy: 1.0000
 97/147 [==================>...........] - ETA: 8s - loss: 0.0091 - accuracy: 1.0000
 98/147 [===================>..........] - ETA: 7s - loss: 0.0091 - accuracy: 1.0000
 99/147 [===================>..........] - ETA: 7s - loss: 0.0091 - accuracy: 1.0000
100/147 [===================>..........] - ETA: 7s - loss: 0.0091 - accuracy: 1.0000
101/147 [===================>..........] - ETA: 7s - loss: 0.0090 - accuracy: 1.0000
102/147 [===================>..........] - ETA: 7s - loss: 0.0090 - accuracy: 1.0000
103/147 [====================>.........] - ETA: 7s - loss: 0.0090 - accuracy: 1.0000
104/147 [====================>.........] - ETA: 6s - loss: 0.0090 - accuracy: 1.0000
105/147 [====================>.........] - ETA: 6s - loss: 0.0090 - accuracy: 1.0000
106/147 [====================>.........] - ETA: 6s - loss: 0.0090 - accuracy: 1.0000
107/147 [====================>.........] - ETA: 6s - loss: 0.0090 - accuracy: 1.0000
108/147 [=====================>........] - ETA: 6s - loss: 0.0090 - accuracy: 1.0000
109/147 [=====================>........] - ETA: 6s - loss: 0.0090 - accuracy: 1.0000
110/147 [=====================>........] - ETA: 5s - loss: 0.0090 - accuracy: 1.0000
111/147 [=====================>........] - ETA: 5s - loss: 0.0090 - accuracy: 1.0000
112/147 [=====================>........] - ETA: 5s - loss: 0.0090 - accuracy: 1.0000
113/147 [======================>.......] - ETA: 5s - loss: 0.0089 - accuracy: 1.0000
114/147 [======================>.......] - ETA: 5s - loss: 0.0089 - accuracy: 1.0000
115/147 [======================>.......] - ETA: 5s - loss: 0.0089 - accuracy: 1.0000
116/147 [======================>.......] - ETA: 5s - loss: 0.0089 - accuracy: 1.0000
117/147 [======================>.......] - ETA: 4s - loss: 0.0089 - accuracy: 1.0000
118/147 [=======================>......] - ETA: 4s - loss: 0.0089 - accuracy: 1.0000
119/147 [=======================>......] - ETA: 4s - loss: 0.0089 - accuracy: 1.0000
120/147 [=======================>......] - ETA: 4s - loss: 0.0088 - accuracy: 1.0000
121/147 [=======================>......] - ETA: 4s - loss: 0.0088 - accuracy: 1.0000
122/147 [=======================>......] - ETA: 4s - loss: 0.0088 - accuracy: 1.0000
123/147 [========================>.....] - ETA: 3s - loss: 0.0088 - accuracy: 1.0000
124/147 [========================>.....] - ETA: 3s - loss: 0.0088 - accuracy: 1.0000
125/147 [========================>.....] - ETA: 3s - loss: 0.0088 - accuracy: 1.0000
126/147 [========================>.....] - ETA: 3s - loss: 0.0088 - accuracy: 1.0000
127/147 [========================>.....] - ETA: 3s - loss: 0.0088 - accuracy: 1.0000
128/147 [=========================>....] - ETA: 3s - loss: 0.0088 - accuracy: 1.0000
129/147 [=========================>....] - ETA: 2s - loss: 0.0088 - accuracy: 1.0000
130/147 [=========================>....] - ETA: 2s - loss: 0.0087 - accuracy: 1.0000
131/147 [=========================>....] - ETA: 2s - loss: 0.0087 - accuracy: 1.0000
132/147 [=========================>....] - ETA: 2s - loss: 0.0087 - accuracy: 1.0000
133/147 [==========================>...] - ETA: 2s - loss: 0.0087 - accuracy: 1.0000
134/147 [==========================>...] - ETA: 2s - loss: 0.0087 - accuracy: 1.0000
135/147 [==========================>...] - ETA: 1s - loss: 0.0087 - accuracy: 1.0000
136/147 [==========================>...] - ETA: 1s - loss: 0.0087 - accuracy: 1.0000
137/147 [==========================>...] - ETA: 1s - loss: 0.0087 - accuracy: 1.0000
138/147 [===========================>..] - ETA: 1s - loss: 0.0087 - accuracy: 1.0000
139/147 [===========================>..] - ETA: 1s - loss: 0.0086 - accuracy: 1.0000
140/147 [===========================>..] - ETA: 1s - loss: 0.0086 - accuracy: 1.0000
141/147 [===========================>..] - ETA: 0s - loss: 0.0086 - accuracy: 1.0000
142/147 [===========================>..] - ETA: 0s - loss: 0.0086 - accuracy: 1.0000
143/147 [============================>.] - ETA: 0s - loss: 0.0086 - accuracy: 1.0000
144/147 [============================>.] - ETA: 0s - loss: 0.0086 - accuracy: 1.0000
145/147 [============================>.] - ETA: 0s - loss: 0.0086 - accuracy: 1.0000
146/147 [============================>.] - ETA: 0s - loss: 0.0086 - accuracy: 1.0000
147/147 [==============================] - 24s 160ms/step - loss: 0.0086 - accuracy: 1.0000

147/147 [==============================] - 26s 174ms/step - loss: 0.0086 - accuracy: 1.0000 - val_loss: 1.1863 - val_accuracy: 0.6126
Epoch 7/10

  1/147 [..............................] - ETA: 0s - loss: 0.0039 - accuracy: 1.0000
  2/147 [..............................] - ETA: 9s - loss: 0.0043 - accuracy: 1.0000
  3/147 [..............................] - ETA: 12s - loss: 0.0046 - accuracy: 1.0000
  4/147 [..............................] - ETA: 14s - loss: 0.0044 - accuracy: 1.0000
  5/147 [>.............................] - ETA: 15s - loss: 0.0044 - accuracy: 1.0000
  6/147 [>.............................] - ETA: 15s - loss: 0.0044 - accuracy: 1.0000
  7/147 [>.............................] - ETA: 15s - loss: 0.0043 - accuracy: 1.0000
  8/147 [>.............................] - ETA: 16s - loss: 0.0043 - accuracy: 1.0000
  9/147 [>.............................] - ETA: 16s - loss: 0.0043 - accuracy: 1.0000
 10/147 [=>............................] - ETA: 16s - loss: 0.0043 - accuracy: 1.0000
 11/147 [=>............................] - ETA: 15s - loss: 0.0043 - accuracy: 1.0000
 12/147 [=>............................] - ETA: 15s - loss: 0.0042 - accuracy: 1.0000
 13/147 [=>............................] - ETA: 15s - loss: 0.0041 - accuracy: 1.0000
 14/147 [=>............................] - ETA: 15s - loss: 0.0041 - accuracy: 1.0000
 15/147 [==>...........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 16/147 [==>...........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 17/147 [==>...........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 18/147 [==>...........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 19/147 [==>...........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 20/147 [===>..........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 21/147 [===>..........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 22/147 [===>..........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 23/147 [===>..........................] - ETA: 15s - loss: 0.0041 - accuracy: 1.0000
 24/147 [===>..........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 25/147 [====>.........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 26/147 [====>.........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 27/147 [====>.........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 28/147 [====>.........................] - ETA: 15s - loss: 0.0039 - accuracy: 1.0000
 29/147 [====>.........................] - ETA: 15s - loss: 0.0039 - accuracy: 1.0000
 30/147 [=====>........................] - ETA: 15s - loss: 0.0039 - accuracy: 1.0000
 31/147 [=====>........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 32/147 [=====>........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 33/147 [=====>........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 34/147 [=====>........................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 35/147 [======>.......................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 36/147 [======>.......................] - ETA: 15s - loss: 0.0040 - accuracy: 1.0000
 37/147 [======>.......................] - ETA: 14s - loss: 0.0040 - accuracy: 1.0000
 38/147 [======>.......................] - ETA: 14s - loss: 0.0040 - accuracy: 1.0000
 39/147 [======>.......................] - ETA: 14s - loss: 0.0040 - accuracy: 1.0000
 40/147 [=======>......................] - ETA: 15s - loss: 0.0039 - accuracy: 1.0000
 41/147 [=======>......................] - ETA: 15s - loss: 0.0039 - accuracy: 1.0000
 42/147 [=======>......................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 43/147 [=======>......................] - ETA: 14s - loss: 0.0040 - accuracy: 1.0000
 44/147 [=======>......................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 45/147 [========>.....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 46/147 [========>.....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 47/147 [========>.....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 48/147 [========>.....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 49/147 [=========>....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 50/147 [=========>....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 51/147 [=========>....................] - ETA: 14s - loss: 0.0039 - accuracy: 1.0000
 52/147 [=========>....................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 53/147 [=========>....................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 54/147 [==========>...................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 55/147 [==========>...................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 56/147 [==========>...................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 57/147 [==========>...................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 58/147 [==========>...................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 59/147 [===========>..................] - ETA: 13s - loss: 0.0039 - accuracy: 1.0000
 60/147 [===========>..................] - ETA: 12s - loss: 0.0039 - accuracy: 1.0000
 61/147 [===========>..................] - ETA: 12s - loss: 0.0038 - accuracy: 1.0000
 62/147 [===========>..................] - ETA: 12s - loss: 0.0038 - accuracy: 1.0000
 63/147 [===========>..................] - ETA: 12s - loss: 0.0038 - accuracy: 1.0000
 64/147 [============>.................] - ETA: 12s - loss: 0.0038 - accuracy: 1.0000
 65/147 [============>.................] - ETA: 12s - loss: 0.0038 - accuracy: 1.0000
 66/147 [============>.................] - ETA: 12s - loss: 0.0038 - accuracy: 1.0000
 67/147 [============>.................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 68/147 [============>.................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 69/147 [=============>................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 70/147 [=============>................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 71/147 [=============>................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 72/147 [=============>................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 73/147 [=============>................] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 74/147 [==============>...............] - ETA: 11s - loss: 0.0038 - accuracy: 1.0000
 75/147 [==============>...............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 76/147 [==============>...............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 77/147 [==============>...............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 78/147 [==============>...............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 79/147 [===============>..............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 80/147 [===============>..............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 81/147 [===============>..............] - ETA: 10s - loss: 0.0038 - accuracy: 1.0000
 82/147 [===============>..............] - ETA: 9s - loss: 0.0038 - accuracy: 1.0000 
 83/147 [===============>..............] - ETA: 9s - loss: 0.0038 - accuracy: 1.0000
 84/147 [================>.............] - ETA: 9s - loss: 0.0037 - accuracy: 1.0000
 85/147 [================>.............] - ETA: 9s - loss: 0.0037 - accuracy: 1.0000
 86/147 [================>.............] - ETA: 9s - loss: 0.0037 - accuracy: 1.0000
 87/147 [================>.............] - ETA: 9s - loss: 0.0037 - accuracy: 1.0000
 88/147 [================>.............] - ETA: 9s - loss: 0.0037 - accuracy: 1.0000
 89/147 [=================>............] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 90/147 [=================>............] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 91/147 [=================>............] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 92/147 [=================>............] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 93/147 [=================>............] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 94/147 [==================>...........] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 95/147 [==================>...........] - ETA: 8s - loss: 0.0037 - accuracy: 1.0000
 96/147 [==================>...........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
 97/147 [==================>...........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
 98/147 [===================>..........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
 99/147 [===================>..........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
100/147 [===================>..........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
101/147 [===================>..........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
102/147 [===================>..........] - ETA: 7s - loss: 0.0037 - accuracy: 1.0000
103/147 [====================>.........] - ETA: 6s - loss: 0.0037 - accuracy: 1.0000
104/147 [====================>.........] - ETA: 6s - loss: 0.0037 - accuracy: 1.0000
105/147 [====================>.........] - ETA: 6s - loss: 0.0037 - accuracy: 1.0000
106/147 [====================>.........] - ETA: 6s - loss: 0.0037 - accuracy: 1.0000
107/147 [====================>.........] - ETA: 6s - loss: 0.0037 - accuracy: 1.0000
108/147 [=====================>........] - ETA: 6s - loss: 0.0037 - accuracy: 1.0000
109/147 [=====================>........] - ETA: 5s - loss: 0.0037 - accuracy: 1.0000
110/147 [=====================>........] - ETA: 5s - loss: 0.0037 - accuracy: 1.0000
111/147 [=====================>........] - ETA: 5s - loss: 0.0037 - accuracy: 1.0000
112/147 [=====================>........] - ETA: 5s - loss: 0.0037 - accuracy: 1.0000
113/147 [======================>.......] - ETA: 5s - loss: 0.0036 - accuracy: 1.0000
114/147 [======================>.......] - ETA: 5s - loss: 0.0036 - accuracy: 1.0000
115/147 [======================>.......] - ETA: 5s - loss: 0.0036 - accuracy: 1.0000
116/147 [======================>.......] - ETA: 4s - loss: 0.0036 - accuracy: 1.0000
117/147 [======================>.......] - ETA: 4s - loss: 0.0036 - accuracy: 1.0000
118/147 [=======================>......] - ETA: 4s - loss: 0.0036 - accuracy: 1.0000
119/147 [=======================>......] - ETA: 4s - loss: 0.0036 - accuracy: 1.0000
120/147 [=======================>......] - ETA: 4s - loss: 0.0036 - accuracy: 1.0000
121/147 [=======================>......] - ETA: 4s - loss: 0.0036 - accuracy: 1.0000
122/147 [=======================>......] - ETA: 3s - loss: 0.0036 - accuracy: 1.0000
123/147 [========================>.....] - ETA: 3s - loss: 0.0036 - accuracy: 1.0000
124/147 [========================>.....] - ETA: 3s - loss: 0.0036 - accuracy: 1.0000
125/147 [========================>.....] - ETA: 3s - loss: 0.0036 - accuracy: 1.0000
126/147 [========================>.....] - ETA: 3s - loss: 0.0036 - accuracy: 1.0000
127/147 [========================>.....] - ETA: 3s - loss: 0.0036 - accuracy: 1.0000
128/147 [=========================>....] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
129/147 [=========================>....] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
130/147 [=========================>....] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
131/147 [=========================>....] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
132/147 [=========================>....] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
133/147 [==========================>...] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
134/147 [==========================>...] - ETA: 2s - loss: 0.0036 - accuracy: 1.0000
135/147 [==========================>...] - ETA: 1s - loss: 0.0036 - accuracy: 1.0000
136/147 [==========================>...] - ETA: 1s - loss: 0.0036 - accuracy: 1.0000
137/147 [==========================>...] - ETA: 1s - loss: 0.0036 - accuracy: 1.0000
138/147 [===========================>..] - ETA: 1s - loss: 0.0036 - accuracy: 1.0000
139/147 [===========================>..] - ETA: 1s - loss: 0.0036 - accuracy: 1.0000
140/147 [===========================>..] - ETA: 1s - loss: 0.0036 - accuracy: 1.0000
141/147 [===========================>..] - ETA: 0s - loss: 0.0036 - accuracy: 1.0000
142/147 [===========================>..] - ETA: 0s - loss: 0.0036 - accuracy: 1.0000
143/147 [============================>.] - ETA: 0s - loss: 0.0036 - accuracy: 1.0000
144/147 [============================>.] - ETA: 0s - loss: 0.0036 - accuracy: 1.0000
145/147 [============================>.] - ETA: 0s - loss: 0.0036 - accuracy: 1.0000
146/147 [============================>.] - ETA: 0s - loss: 0.0036 - accuracy: 1.0000
147/147 [==============================] - 23s 158ms/step - loss: 0.0036 - accuracy: 1.0000

147/147 [==============================] - 25s 171ms/step - loss: 0.0036 - accuracy: 1.0000 - val_loss: 1.2577 - val_accuracy: 0.6131
Epoch 8/10

  1/147 [..............................] - ETA: 0s - loss: 0.0022 - accuracy: 1.0000
  2/147 [..............................] - ETA: 13s - loss: 0.0024 - accuracy: 1.0000
  3/147 [..............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
  4/147 [..............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
  5/147 [>.............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
  6/147 [>.............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
  7/147 [>.............................] - ETA: 17s - loss: 0.0024 - accuracy: 1.0000
  8/147 [>.............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
  9/147 [>.............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
 10/147 [=>............................] - ETA: 17s - loss: 0.0024 - accuracy: 1.0000
 11/147 [=>............................] - ETA: 17s - loss: 0.0024 - accuracy: 1.0000
 12/147 [=>............................] - ETA: 17s - loss: 0.0023 - accuracy: 1.0000
 13/147 [=>............................] - ETA: 16s - loss: 0.0023 - accuracy: 1.0000
 14/147 [=>............................] - ETA: 16s - loss: 0.0023 - accuracy: 1.0000
 15/147 [==>...........................] - ETA: 16s - loss: 0.0023 - accuracy: 1.0000
 16/147 [==>...........................] - ETA: 16s - loss: 0.0023 - accuracy: 1.0000
 17/147 [==>...........................] - ETA: 16s - loss: 0.0024 - accuracy: 1.0000
 18/147 [==>...........................] - ETA: 16s - loss: 0.0024 - accuracy: 1.0000
 19/147 [==>...........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 20/147 [===>..........................] - ETA: 15s - loss: 0.0024 - accuracy: 1.0000
 21/147 [===>..........................] - ETA: 15s - loss: 0.0024 - accuracy: 1.0000
 22/147 [===>..........................] - ETA: 15s - loss: 0.0024 - accuracy: 1.0000
 23/147 [===>..........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 24/147 [===>..........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 25/147 [====>.........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 26/147 [====>.........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 27/147 [====>.........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 28/147 [====>.........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 29/147 [====>.........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 30/147 [=====>........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 31/147 [=====>........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 32/147 [=====>........................] - ETA: 15s - loss: 0.0023 - accuracy: 1.0000
 33/147 [=====>........................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 34/147 [=====>........................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 35/147 [======>.......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 36/147 [======>.......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 37/147 [======>.......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 38/147 [======>.......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 39/147 [======>.......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 40/147 [=======>......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 41/147 [=======>......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 42/147 [=======>......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 43/147 [=======>......................] - ETA: 14s - loss: 0.0023 - accuracy: 1.0000
 44/147 [=======>......................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 45/147 [========>.....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 46/147 [========>.....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 47/147 [========>.....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 48/147 [========>.....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 49/147 [=========>....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 50/147 [=========>....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 51/147 [=========>....................] - ETA: 13s - loss: 0.0023 - accuracy: 1.0000
 52/147 [=========>....................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 53/147 [=========>....................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 54/147 [==========>...................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 55/147 [==========>...................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 56/147 [==========>...................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 57/147 [==========>...................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 58/147 [==========>...................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 59/147 [===========>..................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 60/147 [===========>..................] - ETA: 12s - loss: 0.0023 - accuracy: 1.0000
 61/147 [===========>..................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 62/147 [===========>..................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 63/147 [===========>..................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 64/147 [============>.................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 65/147 [============>.................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 66/147 [============>.................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 67/147 [============>.................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 68/147 [============>.................] - ETA: 11s - loss: 0.0023 - accuracy: 1.0000
 69/147 [=============>................] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 70/147 [=============>................] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 71/147 [=============>................] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 72/147 [=============>................] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 73/147 [=============>................] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 74/147 [==============>...............] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 75/147 [==============>...............] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 76/147 [==============>...............] - ETA: 10s - loss: 0.0023 - accuracy: 1.0000
 77/147 [==============>...............] - ETA: 9s - loss: 0.0023 - accuracy: 1.0000 
 78/147 [==============>...............] - ETA: 9s - loss: 0.0023 - accuracy: 1.0000
 79/147 [===============>..............] - ETA: 9s - loss: 0.0022 - accuracy: 1.0000
 80/147 [===============>..............] - ETA: 9s - loss: 0.0022 - accuracy: 1.0000
 81/147 [===============>..............] - ETA: 9s - loss: 0.0022 - accuracy: 1.0000
 82/147 [===============>..............] - ETA: 9s - loss: 0.0022 - accuracy: 1.0000
 83/147 [===============>..............] - ETA: 9s - loss: 0.0022 - accuracy: 1.0000
 84/147 [================>.............] - ETA: 9s - loss: 0.0022 - accuracy: 1.0000
 85/147 [================>.............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 86/147 [================>.............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 87/147 [================>.............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 88/147 [================>.............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 89/147 [=================>............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 90/147 [=================>............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 91/147 [=================>............] - ETA: 8s - loss: 0.0022 - accuracy: 1.0000
 92/147 [=================>............] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 93/147 [=================>............] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 94/147 [==================>...........] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 95/147 [==================>...........] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 96/147 [==================>...........] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 97/147 [==================>...........] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 98/147 [===================>..........] - ETA: 7s - loss: 0.0022 - accuracy: 1.0000
 99/147 [===================>..........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
100/147 [===================>..........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
101/147 [===================>..........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
102/147 [===================>..........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
103/147 [====================>.........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
104/147 [====================>.........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
105/147 [====================>.........] - ETA: 6s - loss: 0.0022 - accuracy: 1.0000
106/147 [====================>.........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
107/147 [====================>.........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
108/147 [=====================>........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
109/147 [=====================>........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
110/147 [=====================>........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
111/147 [=====================>........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
112/147 [=====================>........] - ETA: 5s - loss: 0.0022 - accuracy: 1.0000
113/147 [======================>.......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
114/147 [======================>.......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
115/147 [======================>.......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
116/147 [======================>.......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
117/147 [======================>.......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
118/147 [=======================>......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
119/147 [=======================>......] - ETA: 4s - loss: 0.0022 - accuracy: 1.0000
120/147 [=======================>......] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
121/147 [=======================>......] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
122/147 [=======================>......] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
123/147 [========================>.....] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
124/147 [========================>.....] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
125/147 [========================>.....] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
126/147 [========================>.....] - ETA: 3s - loss: 0.0022 - accuracy: 1.0000
127/147 [========================>.....] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
128/147 [=========================>....] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
129/147 [=========================>....] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
130/147 [=========================>....] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
131/147 [=========================>....] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
132/147 [=========================>....] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
133/147 [==========================>...] - ETA: 2s - loss: 0.0022 - accuracy: 1.0000
134/147 [==========================>...] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
135/147 [==========================>...] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
136/147 [==========================>...] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
137/147 [==========================>...] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
138/147 [===========================>..] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
139/147 [===========================>..] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
140/147 [===========================>..] - ETA: 1s - loss: 0.0022 - accuracy: 1.0000
141/147 [===========================>..] - ETA: 0s - loss: 0.0021 - accuracy: 1.0000
142/147 [===========================>..] - ETA: 0s - loss: 0.0021 - accuracy: 1.0000
143/147 [============================>.] - ETA: 0s - loss: 0.0021 - accuracy: 1.0000
144/147 [============================>.] - ETA: 0s - loss: 0.0021 - accuracy: 1.0000
145/147 [============================>.] - ETA: 0s - loss: 0.0021 - accuracy: 1.0000
146/147 [============================>.] - ETA: 0s - loss: 0.0021 - accuracy: 1.0000
147/147 [==============================] - 22s 150ms/step - loss: 0.0021 - accuracy: 1.0000

147/147 [==============================] - 24s 163ms/step - loss: 0.0021 - accuracy: 1.0000 - val_loss: 1.3165 - val_accuracy: 0.6075
Epoch 9/10

  1/147 [..............................] - ETA: 0s - loss: 0.0016 - accuracy: 1.0000
  2/147 [..............................] - ETA: 10s - loss: 0.0016 - accuracy: 1.0000
  3/147 [..............................] - ETA: 13s - loss: 0.0018 - accuracy: 1.0000
  4/147 [..............................] - ETA: 15s - loss: 0.0017 - accuracy: 1.0000
  5/147 [>.............................] - ETA: 16s - loss: 0.0017 - accuracy: 1.0000
  6/147 [>.............................] - ETA: 17s - loss: 0.0017 - accuracy: 1.0000
  7/147 [>.............................] - ETA: 17s - loss: 0.0017 - accuracy: 1.0000
  8/147 [>.............................] - ETA: 17s - loss: 0.0017 - accuracy: 1.0000
  9/147 [>.............................] - ETA: 17s - loss: 0.0017 - accuracy: 1.0000
 10/147 [=>............................] - ETA: 17s - loss: 0.0017 - accuracy: 1.0000
 11/147 [=>............................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 12/147 [=>............................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 13/147 [=>............................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 14/147 [=>............................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 15/147 [==>...........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 16/147 [==>...........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 17/147 [==>...........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 18/147 [==>...........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 19/147 [==>...........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 20/147 [===>..........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 21/147 [===>..........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 22/147 [===>..........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 23/147 [===>..........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 24/147 [===>..........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 25/147 [====>.........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 26/147 [====>.........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 27/147 [====>.........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 28/147 [====>.........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 29/147 [====>.........................] - ETA: 17s - loss: 0.0016 - accuracy: 1.0000
 30/147 [=====>........................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 31/147 [=====>........................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 32/147 [=====>........................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 33/147 [=====>........................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 34/147 [=====>........................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 35/147 [======>.......................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 36/147 [======>.......................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 37/147 [======>.......................] - ETA: 16s - loss: 0.0016 - accuracy: 1.0000
 38/147 [======>.......................] - ETA: 15s - loss: 0.0016 - accuracy: 1.0000
 39/147 [======>.......................] - ETA: 15s - loss: 0.0015 - accuracy: 1.0000
 40/147 [=======>......................] - ETA: 15s - loss: 0.0015 - accuracy: 1.0000
 41/147 [=======>......................] - ETA: 15s - loss: 0.0015 - accuracy: 1.0000
 42/147 [=======>......................] - ETA: 15s - loss: 0.0015 - accuracy: 1.0000
 43/147 [=======>......................] - ETA: 15s - loss: 0.0015 - accuracy: 1.0000
 44/147 [=======>......................] - ETA: 15s - loss: 0.0015 - accuracy: 1.0000
 45/147 [========>.....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 46/147 [========>.....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 47/147 [========>.....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 48/147 [========>.....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 49/147 [=========>....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 50/147 [=========>....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 51/147 [=========>....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 52/147 [=========>....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 53/147 [=========>....................] - ETA: 14s - loss: 0.0015 - accuracy: 1.0000
 54/147 [==========>...................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 55/147 [==========>...................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 56/147 [==========>...................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 57/147 [==========>...................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 58/147 [==========>...................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 59/147 [===========>..................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 60/147 [===========>..................] - ETA: 13s - loss: 0.0015 - accuracy: 1.0000
 61/147 [===========>..................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 62/147 [===========>..................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 63/147 [===========>..................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 64/147 [============>.................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 65/147 [============>.................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 66/147 [============>.................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 67/147 [============>.................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 68/147 [============>.................] - ETA: 12s - loss: 0.0015 - accuracy: 1.0000
 69/147 [=============>................] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 70/147 [=============>................] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 71/147 [=============>................] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 72/147 [=============>................] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 73/147 [=============>................] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 74/147 [==============>...............] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 75/147 [==============>...............] - ETA: 11s - loss: 0.0015 - accuracy: 1.0000
 76/147 [==============>...............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 77/147 [==============>...............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 78/147 [==============>...............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 79/147 [===============>..............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 80/147 [===============>..............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 81/147 [===============>..............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 82/147 [===============>..............] - ETA: 10s - loss: 0.0015 - accuracy: 1.0000
 83/147 [===============>..............] - ETA: 9s - loss: 0.0015 - accuracy: 1.0000 
 84/147 [================>.............] - ETA: 9s - loss: 0.0015 - accuracy: 1.0000
 85/147 [================>.............] - ETA: 9s - loss: 0.0015 - accuracy: 1.0000
 86/147 [================>.............] - ETA: 9s - loss: 0.0015 - accuracy: 1.0000
 87/147 [================>.............] - ETA: 9s - loss: 0.0015 - accuracy: 1.0000
 88/147 [================>.............] - ETA: 9s - loss: 0.0015 - accuracy: 1.0000
 89/147 [=================>............] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 90/147 [=================>............] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 91/147 [=================>............] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 92/147 [=================>............] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 93/147 [=================>............] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 94/147 [==================>...........] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 95/147 [==================>...........] - ETA: 8s - loss: 0.0015 - accuracy: 1.0000
 96/147 [==================>...........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
 97/147 [==================>...........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
 98/147 [===================>..........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
 99/147 [===================>..........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
100/147 [===================>..........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
101/147 [===================>..........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
102/147 [===================>..........] - ETA: 7s - loss: 0.0015 - accuracy: 1.0000
103/147 [====================>.........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
104/147 [====================>.........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
105/147 [====================>.........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
106/147 [====================>.........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
107/147 [====================>.........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
108/147 [=====================>........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
109/147 [=====================>........] - ETA: 6s - loss: 0.0015 - accuracy: 1.0000
110/147 [=====================>........] - ETA: 5s - loss: 0.0015 - accuracy: 1.0000
111/147 [=====================>........] - ETA: 5s - loss: 0.0015 - accuracy: 1.0000
112/147 [=====================>........] - ETA: 5s - loss: 0.0015 - accuracy: 1.0000
113/147 [======================>.......] - ETA: 5s - loss: 0.0015 - accuracy: 1.0000
114/147 [======================>.......] - ETA: 5s - loss: 0.0015 - accuracy: 1.0000
115/147 [======================>.......] - ETA: 5s - loss: 0.0015 - accuracy: 1.0000
116/147 [======================>.......] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
117/147 [======================>.......] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
118/147 [=======================>......] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
119/147 [=======================>......] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
120/147 [=======================>......] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
121/147 [=======================>......] - ETA: 4s - loss: 0.0015 - accuracy: 1.0000
122/147 [=======================>......] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
123/147 [========================>.....] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
124/147 [========================>.....] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
125/147 [========================>.....] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
126/147 [========================>.....] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
127/147 [========================>.....] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
128/147 [=========================>....] - ETA: 3s - loss: 0.0015 - accuracy: 1.0000
129/147 [=========================>....] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
130/147 [=========================>....] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
131/147 [=========================>....] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
132/147 [=========================>....] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
133/147 [==========================>...] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
134/147 [==========================>...] - ETA: 2s - loss: 0.0015 - accuracy: 1.0000
135/147 [==========================>...] - ETA: 1s - loss: 0.0015 - accuracy: 1.0000
136/147 [==========================>...] - ETA: 1s - loss: 0.0015 - accuracy: 1.0000
137/147 [==========================>...] - ETA: 1s - loss: 0.0015 - accuracy: 1.0000
138/147 [===========================>..] - ETA: 1s - loss: 0.0015 - accuracy: 1.0000
139/147 [===========================>..] - ETA: 1s - loss: 0.0015 - accuracy: 1.0000
140/147 [===========================>..] - ETA: 1s - loss: 0.0015 - accuracy: 1.0000
141/147 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 1.0000
142/147 [===========================>..] - ETA: 0s - loss: 0.0015 - accuracy: 1.0000
143/147 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 1.0000
144/147 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 1.0000
145/147 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 1.0000
146/147 [============================>.] - ETA: 0s - loss: 0.0015 - accuracy: 1.0000
147/147 [==============================] - 24s 160ms/step - loss: 0.0015 - accuracy: 1.0000

147/147 [==============================] - 26s 174ms/step - loss: 0.0015 - accuracy: 1.0000 - val_loss: 1.3674 - val_accuracy: 0.6075
Epoch 10/10

  1/147 [..............................] - ETA: 0s - loss: 0.0010 - accuracy: 1.0000
  2/147 [..............................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
  3/147 [..............................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
  4/147 [..............................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
  5/147 [>.............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
  6/147 [>.............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
  7/147 [>.............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
  8/147 [>.............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
  9/147 [>.............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
 10/147 [=>............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
 11/147 [=>............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
 12/147 [=>............................] - ETA: 16s - loss: 0.0011 - accuracy: 1.0000
 13/147 [=>............................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 14/147 [=>............................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 15/147 [==>...........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 16/147 [==>...........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 17/147 [==>...........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 18/147 [==>...........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 19/147 [==>...........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 20/147 [===>..........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 21/147 [===>..........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 22/147 [===>..........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 23/147 [===>..........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 24/147 [===>..........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 25/147 [====>.........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 26/147 [====>.........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 27/147 [====>.........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 28/147 [====>.........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 29/147 [====>.........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 30/147 [=====>........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 31/147 [=====>........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 32/147 [=====>........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 33/147 [=====>........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 34/147 [=====>........................] - ETA: 15s - loss: 0.0011 - accuracy: 1.0000
 35/147 [======>.......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 36/147 [======>.......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 37/147 [======>.......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 38/147 [======>.......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 39/147 [======>.......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 40/147 [=======>......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 41/147 [=======>......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 42/147 [=======>......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 43/147 [=======>......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 44/147 [=======>......................] - ETA: 14s - loss: 0.0011 - accuracy: 1.0000
 45/147 [========>.....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 46/147 [========>.....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 47/147 [========>.....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 48/147 [========>.....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 49/147 [=========>....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 50/147 [=========>....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 51/147 [=========>....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 52/147 [=========>....................] - ETA: 13s - loss: 0.0011 - accuracy: 1.0000
 53/147 [=========>....................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 54/147 [==========>...................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 55/147 [==========>...................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 56/147 [==========>...................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 57/147 [==========>...................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 58/147 [==========>...................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 59/147 [===========>..................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 60/147 [===========>..................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 61/147 [===========>..................] - ETA: 12s - loss: 0.0011 - accuracy: 1.0000
 62/147 [===========>..................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 63/147 [===========>..................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 64/147 [============>.................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 65/147 [============>.................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 66/147 [============>.................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 67/147 [============>.................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 68/147 [============>.................] - ETA: 11s - loss: 0.0011 - accuracy: 1.0000
 69/147 [=============>................] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 70/147 [=============>................] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 71/147 [=============>................] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 72/147 [=============>................] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 73/147 [=============>................] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 74/147 [==============>...............] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 75/147 [==============>...............] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 76/147 [==============>...............] - ETA: 10s - loss: 0.0011 - accuracy: 1.0000
 77/147 [==============>...............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000 
 78/147 [==============>...............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000
 79/147 [===============>..............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000
 80/147 [===============>..............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000
 81/147 [===============>..............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000
 82/147 [===============>..............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000
 83/147 [===============>..............] - ETA: 9s - loss: 0.0011 - accuracy: 1.0000
 84/147 [================>.............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 85/147 [================>.............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 86/147 [================>.............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 87/147 [================>.............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 88/147 [================>.............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 89/147 [=================>............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 90/147 [=================>............] - ETA: 8s - loss: 0.0011 - accuracy: 1.0000
 91/147 [=================>............] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 92/147 [=================>............] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 93/147 [=================>............] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 94/147 [==================>...........] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 95/147 [==================>...........] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 96/147 [==================>...........] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 97/147 [==================>...........] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 98/147 [===================>..........] - ETA: 7s - loss: 0.0011 - accuracy: 1.0000
 99/147 [===================>..........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
100/147 [===================>..........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
101/147 [===================>..........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
102/147 [===================>..........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
103/147 [====================>.........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
104/147 [====================>.........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
105/147 [====================>.........] - ETA: 6s - loss: 0.0011 - accuracy: 1.0000
106/147 [====================>.........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
107/147 [====================>.........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
108/147 [=====================>........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
109/147 [=====================>........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
110/147 [=====================>........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
111/147 [=====================>........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
112/147 [=====================>........] - ETA: 5s - loss: 0.0011 - accuracy: 1.0000
113/147 [======================>.......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
114/147 [======================>.......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
115/147 [======================>.......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
116/147 [======================>.......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
117/147 [======================>.......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
118/147 [=======================>......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
119/147 [=======================>......] - ETA: 4s - loss: 0.0011 - accuracy: 1.0000
120/147 [=======================>......] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
121/147 [=======================>......] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
122/147 [=======================>......] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
123/147 [========================>.....] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
124/147 [========================>.....] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
125/147 [========================>.....] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
126/147 [========================>.....] - ETA: 3s - loss: 0.0011 - accuracy: 1.0000
127/147 [========================>.....] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
128/147 [=========================>....] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
129/147 [=========================>....] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
130/147 [=========================>....] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
131/147 [=========================>....] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
132/147 [=========================>....] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
133/147 [==========================>...] - ETA: 2s - loss: 0.0011 - accuracy: 1.0000
134/147 [==========================>...] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
135/147 [==========================>...] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
136/147 [==========================>...] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
137/147 [==========================>...] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
138/147 [===========================>..] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
139/147 [===========================>..] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
140/147 [===========================>..] - ETA: 1s - loss: 0.0011 - accuracy: 1.0000
141/147 [===========================>..] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
142/147 [===========================>..] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
143/147 [============================>.] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
144/147 [============================>.] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
145/147 [============================>.] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
146/147 [============================>.] - ETA: 0s - loss: 0.0011 - accuracy: 1.0000
147/147 [==============================] - 22s 147ms/step - loss: 0.0011 - accuracy: 1.0000

147/147 [==============================] - 24s 160ms/step - loss: 0.0011 - accuracy: 1.0000 - val_loss: 1.4042 - val_accuracy: 0.6062

Yes, I know, this one takes a while…

history_rnn %>% plot()

In adittion, the model performs quite dissapointing. After a couple of epochs, it starts overfitting, as often. However, the highest validation accuracy we get during the epochs is slightly higher than 85%, which is no improvement compared to our baseling. Part of the problem is that your inputs only consider the first 500 words, rather than full sequences—hence, the RNN has access to less information than the earlier baseline model.

The insight of the problem is that layer_simple_rnn isn’t good at processing long sequences, such as 500+ words review text. The reason is that words often derive their meaning not only from the one word before, but a longer sequence, eg. the whole sentence in which they occur. For such settings, other types of recurrent layers perform much better. Let’s go on exploring.

LSTM Application

model_lstm <- keras_model_sequential() %>%
  layer_embedding(input_dim = 10000, output_dim = 32) %>%
  layer_lstm(units = 500, dropout = 0.25, recurrent_dropout = 0.25, return_sequences = FALSE) %>%
  layer_dense(units = 1, activation = "sigmoid")

model_lstm %>% compile(
  optimizer = "adam",
  loss = "binary_crossentropy",
  metrics = "accuracy"
)

summary(model_lstm)

Model: "sequential_5"
_________________________________________________________________________________________________
Layer (type)                               Output Shape                           Param #        
=================================================================================================
embedding_4 (Embedding)                    (None, None, 32)                       320000         
_________________________________________________________________________________________________
lstm (LSTM)                                (None, 500)                            1066000        
_________________________________________________________________________________________________
dense_4 (Dense)                            (None, 1)                              501            
=================================================================================================
Total params: 1,386,501
Trainable params: 1,386,501
Non-trainable params: 0
_________________________________________________________________________________________________

Lets run it!

history_lstm <- model_lstm %>% fit(
  x_train,
  y_train,
  epochs = 5,
  batch_size = 512,
  validation_split = 0.25
  )

And lets take a look how the accuracy and loss developed over the epocs.

plot(history_lstm)

Well… that takes forever. In this case, we also not exprience any particulr performance benefits… sadly we waited in vain this time… :( To be fair, we see the accuracy still improving, so if we might wait a it…. who knows…

In case you have the GPU version of keras installed, you can actually get some real benefits if you use the layer_cudnn_gru() instead of the layer_simple_rnn(), and the layer_cudnn_lstm() instead of the layer_lstm. Both are optimized for CUDA parallel processing on the GPU, and speed up the whole story quite a bit. Both can also be used on colab, since there is the GPU available.

Endnotes

References

[^1] Yoshua Bengio, Patrice Simard, and Paolo Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” IEEE Transactions on Neural Networks 5, no. 2 (1994).

[^2] Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation 9, no. 8 (1997).

More info

You can find more info about:

• keras here

Session info

sessionInfo()