I have a question about how Keras handles validation. I have a pre-trained model (ResNet34 with a U-NET architecture) and want to train on a custom dataset for binary segmentation. I created a validation set of 60 images and when I tried to set an EarlyStopping with the val_loss, I had weird results. I find them odd because I'm using the same directories for training and validation since the model wasn't learning and it returned full black or full white images. My main question is, is this behavior logical even with regularization terms and drop-out layer? It just looks wrong. When I trained the model without validation it seemed to work and learn. I had an accuracy of ~85%. I'm using a library for pre-trained models called segmentation_models, which uses the Keras framework. So, when using the train set and validation shouldn't the scores be close if not the same?
The Results:
Epoch 1/200 60/60 [==============================] - 32s 473ms/step - loss: 0.3068 - iou_score: 0.7164 - val_loss: 2.5515 - val_iou_score: 0.1365 Epoch 2/200 60/60 [==============================] - 28s 465ms/step - loss: 0.1274 - iou_score: 0.8831 - val_loss: 1.1177 - val_iou_score: 0.3766 Epoch 3/200 60/60 [==============================] - 27s 444ms/step - loss: 0.0852 - iou_score: 0.9224 - val_loss: 1.6045 - val_iou_score: 0.3112 Epoch 4/200 60/60 [==============================] - 27s 454ms/step - loss: 0.0603 - iou_score: 0.9445 - val_loss: 0.9970 - val_iou_score: 0.4547 Epoch 5/200 60/60 [==============================] - 27s 459ms/step - loss: 0.0493 - iou_score: 0.9545 - val_loss: 0.8126 - val_iou_score: 0.3551 Epoch 6/200 60/60 [==============================] - 27s 447ms/step - loss: 0.0336 - iou_score: 0.9679 - val_loss: 2.2715 - val_iou_score: 0.6446 Epoch 7/200 60/60 [==============================] - 27s 445ms/step - loss: 0.0289 - iou_score: 0.9730 - val_loss: 1.9526 - val_iou_score: 0.6421
import numpy as np
from segmentation_models import Unet, get_preprocessing
import segmentation_models.losses
from segmentation_models.metrics import iou_score
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping
Define the directory where models will be saved
SAVE_DIR = "hf_0_saved_models"
os.makedirs(SAVE_DIR, exist_ok=True)
List of all model backbones to loop through
BACKBONES = [
'resnet34'
#'seresnet18', 'efficientnetb2', 'resnext50'
#'vgg16' , 'vgg19', 'resnet18', 'resnet34', 'resnet50', 'resnet101', 'resnet152',
#'seresnet18', 'seresnet34', 'seresnet50', 'seresnet101', 'seresnet152',
#'resnext50', 'resnext101', 'seresnext50', 'seresnext101', 'senet154',
#'densenet121', 'densenet169', 'densenet201', 'inceptionv3', 'inceptionresnetv2',
#'mobilenet', 'mobilenetv2', 'efficientnetb0', 'efficientnetb1', 'efficientnetb2',
#'efficientnetb3', 'efficientnetb4', 'efficientnetb5', 'efficientnetb6', 'efficientnetb7'
]
x_train, y_train = load_data('./train/image', './train/mask')
x_val, y_val = load_data('./train/image', './train/mask')
Loop over each backbone and train the model
for BACKBONE in BACKBONES:
print(f"Training model with backbone: {BACKBONE}")
# Preprocess input according to the backbone
preprocess_input = get_preprocessing(BACKBONE)
x_train = preprocess_input(x_train)
x_val = preprocess_input(x_val)
# Define the model
model = Unet(BACKBONE, encoder_weights='imagenet', classes=1, activation='sigmoid')
loss = segmentation_models.losses.BinaryCELoss()
# Compile the model with Adam optimizer, loss, and IOU score metric
model.compile(optimizer='Adam', loss=loss, metrics=[iou_score])
# Define a callback to save the model after training
model_save_path = os.path.join(SAVE_DIR, f"{BACKBONE}_unet_model.h5")
checkpoint = ModelCheckpoint(model_save_path, save_best_only=True)
early_stopping = EarlyStopping(
monitor='val_loss',
patience=10,
restore_best_weights=True
)
# Train the model
history = model.fit(
x=x_train,
y=y_train,
batch_size=4,
epochs=200,
callbacks=[checkpoint, early_stopping],
validation_data=(x_val, y_val)
)
print(f"Model with backbone {BACKBONE} has been trained and saved at {model_save_path}.")
```
