Forensic Analysis of Convolutional Neural Networks (CNNs)

10 min readDec 10, 2024

Deep neural networks are infamously known as the “black box” due to their opaque decision-making. However this is quite the opposite when it comes to convolution networks.

This text presents a forensic analysis of CNNs to understand their decision-making mechanisms. The methods discussed are listed below, these methods also aid in debugging CNNs.

Environment Setup
Visualising intermediate convnet outputs
Visualising heat-maps of class activation in an image

All examples are build around the Keras library, after installing keras one can execute the code given, with no any other installation.

Environment Setup

Before diving into the code and topics, let’s set up our notebook as follows:

import math
import os
import random
from io import BytesIO

import albumentations as A
import cv2
import keras
import matplotlib.pyplot as plt
import numpy as np
import requests
import tensorflow as tf
from PIL import Image
from sklearn.model_selection import train_test_split

####################################################################

def system_config(SEED_VALUE):
    
    random.seed(SEED_VALUE)
    np.random.seed(SEED_VALUE)
    tf.random.set_seed(SEED_VALUE)

    # Get list of GPUs.
    gpu_devices = tf.config.list_physical_devices('GPU')
    print(gpu_devices)

    if len(gpu_devices) > 0:
        print('Using GPU')
        os.environ["CUDA_VISIBLE_DEVICES"] = '0'
        os.environ['TF_CUDNN_DETERMINISTIC'] = '1'

        # If there are any gpu devices, use first gpu.
        tf.config.experimental.set_visible_devices(gpu_devices[0], 'GPU')

        # Grow the memory usage as it is needed by the process.
        tf.config.experimental.set_memory_growth(gpu_devices[0], True)

        # Enable using cudNN.
        os.environ['TF_USE_CUDNN'] = "true"
    else:
        print('Using CPU')


SEED_VALUE = 42
system_config(SEED_VALUE=SEED_VALUE)


####################################################################

def load_and_process_image(image_path_or_url, target_size=(224, 224)):
    """
    Load an image from a local directory or URL, resize it, and return as a NumPy array.
    
    Parameters:
        image_path_or_url (str): Local file path or URL of the image.
        target_size (tuple): The target size to resize the image to (width, height).
        
    Returns:
        numpy.ndarray: Resized image as a NumPy array.
    """
    # If the input is a URL, download the image
    if image_path_or_url.startswith('http') or image_path_or_url.startswith('https'):
        # Download image from URL
        response = requests.get(image_path_or_url)
        img = Image.open(BytesIO(response.content))
    elif os.path.isfile(image_path_or_url):
        # Load image from local path
        img = keras.utils.load_img(image_path_or_url)
    else:
        raise ValueError("Provided path is not valid (neither a valid URL nor local file).")
    
    img = img.resize(target_size)
     
    # Convert the image to a NumPy array
    return keras.utils.img_to_array(img).astype(np.uint8)

####################################################################

def draw_activation_layer_output(layer):
    """
    Visualize activation maps for each batch in a given layer output,
    arranging them into 2 rows and dynamically calculated columns.
    
    Parameters:
        layer (numpy.ndarray): The activation layer output with shape (batch_size, height, width, channels).
    """
    for batch_output in range(layer.shape[0]):
        num_filters = layer.shape[-1]  # Number of filters (channels)
        num_rows = 2  # Always 2 rows
        num_cols = math.ceil(num_filters / num_rows)  # Calculate columns based on filters
        
        # Create a figure with 2 rows and calculated columns
        fig, axes = plt.subplots(num_rows, num_cols, figsize=(num_cols * 2, 4))
        axes = axes.flatten()  # Flatten to easily access each subplot
        
        # Iterate over each filter and display it
        for i in range(num_filters):
            axes[i].imshow(layer[batch_output, ..., i], cmap='viridis')  # Show the i-th channel
            axes[i].set_title(f"Batch {batch_output} Filter {i}")
            axes[i].axis('off')  # Remove axis for a cleaner look
        
        # Hide any remaining axes (if filters < num_rows * num_cols)
        for j in range(num_filters, len(axes)):
            axes[j].axis('off')
        
        plt.tight_layout()  # Adjust layout for better spacing
        plt.show()

Visualising intermediate convnet outputs

TL;DR; The main idea of this experiment is to understand how a conv-net see our image data and associates that to its predicted class.

Visualizing intermediate activations involves displaying the outputs of various convolution and pooling layers in a model for a given input. These outputs, often referred to as activations (the result of the activation function), provide insight into how the input is broken down by the different filters learned by the network.

Let’s download 2 images from net.

bike = load_and_process_image("https://p.vitalmtb.com/photos/users/21424/setup_checks/27503/photos/32249/original_DSC_1028a.jpg?1417526613")
plt.imshow(bike)
plt.show()

car = load_and_process_image("https://st.automobilemag.com/uploads/sites/11/2017/12/Shelby-GT-500CR-front-three-quarter-01.jpg")
plt.imshow(car)
plt.show()

These two images will be used as guinea pigs. Now lets get a model that we now trained on car and bike images (note that this is not necessary, since conv-layers are just feature extraction layers, it would extract features for any given image)

In the following code, we load MobileNetV2 from the Keras library, which is pretrained on the ‘ImageNet’ dataset by default. We then extract all layers that are either Conv2D, SeparableConv2D, or MaxPool2D, as these layers generate representations (images) based on the input image.

model = keras.applications.MobileNetV2(input_shape=(224,224,3),include_top=True)
print(model.summary())
layer_outputs = []
layer_names = []

for layer in model.layers:
    if isinstance(layer,(keras.layers.Conv2D, keras.layers.SeparableConv2D, keras.layers.MaxPool2D)):
        layer_names.append(layer.name)
        layer_outputs.append(layer.output)

┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Layer (type)        ┃ Output Shape      ┃    Param # ┃ Connected to      ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ input_layer_3       │ (None, 224, 224,  │          0 │ -                 │
│ (InputLayer)        │ 3)                │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ Conv1 (Conv2D)      │ (None, 112, 112,  │        864 │ input_layer_3[0]… │
│                     │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ bn_Conv1            │ (None, 112, 112,  │        128 │ Conv1[0][0]       │
│ (BatchNormalizatio… │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ Conv1_relu (ReLU)   │ (None, 112, 112,  │          0 │ bn_Conv1[0][0]    │
│                     │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ expanded_conv_dept… │ (None, 112, 112,  │        288 │ Conv1_relu[0][0]  │
│ (DepthwiseConv2D)   │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ expanded_conv_dept… │ (None, 112, 112,  │        128 │ expanded_conv_de… │
│ (BatchNormalizatio… │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ expanded_conv_dept… │ (None, 112, 112,  │          0 │ expanded_conv_de… │
│ (ReLU)              │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ expanded_conv_proj… │ (None, 112, 112,  │        512 │ expanded_conv_de… │
│ (Conv2D)            │ 16)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ expanded_conv_proj… │ (None, 112, 112,  │         64 │ expanded_conv_pr… │
│ (BatchNormalizatio… │ 16)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ block_1_expand      │ (None, 112, 112,  │      1,536 │ expanded_conv_pr… │
│ (Conv2D)            │ 96)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
.....
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ Conv_1 (Conv2D)     │ (None, 7, 7,      │    409,600 │ block_16_project… │
│                     │ 1280)             │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ Conv_1_bn           │ (None, 7, 7,      │      5,120 │ Conv_1[0][0]      │
│ (BatchNormalizatio… │ 1280)             │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ out_relu (ReLU)     │ (None, 7, 7,      │          0 │ Conv_1_bn[0][0]   │
│                     │ 1280)             │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ global_average_poo… │ (None, 1280)      │          0 │ out_relu[0][0]    │
│ (GlobalAveragePool… │                   │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ predictions (Dense) │ (None, 1000)      │  1,281,000 │ global_average_p… │
└─────────────────────┴───────────────────┴────────────┴───────────────────┘

 Total params: 3,538,984 (13.50 MB)
 Trainable params: 3,504,872 (13.37 MB)
 Non-trainable params: 34,112 (133.25 KB)

Let’s check what’s inside the layer_names

print(len(layer_names))
for lname in layer_names:
    print(lname)

########## OUTPUTS ######
35
Conv1
expanded_conv_project
block_1_expand
block_1_project
block_2_expand
block_2_project
block_3_expand
block_3_project
block_4_expand
block_4_project
block_5_expand
block_5_project
block_6_expand
block_6_project
block_7_expand
block_7_project
block_8_expand
block_8_project
block_9_expand
block_9_project
block_10_expand
block_10_project
block_11_expand
block_11_project
block_12_expand
block_12_project
block_13_expand
block_13_project
block_14_expand
block_14_project
block_15_expand
block_15_project
block_16_expand
block_16_project
Conv_1

As can be seen, we saved all conv2d, separableConv2d and maxPool2D layer’s names and outputs. By using the layers collected, we now create a new model where the input is the input of the MobileNetV2 whereas the output is all the layers that we collected.

activation_model = keras.Model(inputs=model.input, outputs=layer_outputs)

This model is named as “intermediate model” in the literature, for people who may wonder; this model internally has all the execution graph defined in the mobilenetv2. when a forward-pass is executed via (predict or evaluate) it sends data by using “model.input” and returns all results defined in “layer_outputs”

Now it’s time to make forward pass; following code batches the images and send the preprocessing pipeline for mobilenetv2 then calls predict method.

batch = np.stack([bike,car],axis=0) 
preprocessed_batch = keras.applications.mobilenet_v2.preprocess_input(batch)

activation_predictions = activation_model.predict(preprocessed_batch)
print(len(activation_predictions),activation_predictions[0].shape)

[1m1/1[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m4s[0m 4s/step
35 (2, 112, 112, 32)

Here you can see that we have 35 outputs, as we wanted to fetch, and 2 batch images. for the first layer our model outputs 112x112x32 images as can be seen. this is the first conv layer.

├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ Conv1 (Conv2D)      │ (None, 112, 112,  │        864 │ input_layer_3[0]… │
│                     │ 32)               │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤

Now let’s print 5 of the layers and the final convolutional layer and discuss how our model see our

for i in range(5):
    print(layer_names[i])
    draw_activation_layer_output(activation_predictions[i])

Since output class has 1280 feature maps, we’ll only draw 24 of them per class

├─────────────────────┼───────────────────┼────────────┼───────────────────┤
│ Conv_1 (Conv2D)     │ (None, 7, 7,      │    409,600 │ block_16_project… │
│                     │ 1280)             │            │                   │
├─────────────────────┼───────────────────┼────────────┼───────────────────┤

last_layer = activation_predictions[layer_names.index('Conv_1')]
print(last_layer.shape)

draw_activation_layer_output(last_layer[...,:24])

The Viridis colormap in matplotlib has a continuous range of values between:

0 (the minimum value, represented by the darkest purple)
1 (the maximum value, represented by the brightest yellow)

so you can assume that the dark parts are not seen by the model whereas yellow-ish parts are seen.

Take Away;

The first Conv_1 layer processes various regions of the image.
The second expanded_conv_project layer and the third block_1_expand layer function as distinct types of edge detectors.
As the network goes deeper, the activations become more abstract and harder to interpret visually.
In the deeper layers, more filters become inactive, suggesting that the patterns they are designed to detect do not appear in the input image.
Deeper representations contain progressively less detail about the image’s visual content and increasingly more information about its class, as evident in the output of the final Conv_1 layer.

Visualising heat-maps of class activation in an image

As you can see in the previous example Conv_1 layer, outputs some feature maps to be used in the dense layer part for class prediction however we still don’t know what parts of the image contributes the most to the output class. This bring us to this another useful method.

one can read the theory online, here I’ll just provide the code and the results.

elaphant = load_and_process_image("https://i.pinimg.com/originals/4b/ed/93/4bed9338b2e7b5ccde712551949ff97a.jpg")
plt.imshow(elaphant)
plt.show()
elaphant =  np.expand_dims(elaphant,axis=0)
print(elaphant.shape) 


truck = load_and_process_image("http://images8.alphacoders.com/538/538890.jpg")
plt.imshow(truck)
plt.show()
truck =  np.expand_dims(truck,axis=0)
print(truck.shape) 

elaphant = keras.applications.mobilenet_v2.preprocess_input(elaphant)
truck = keras.applications.mobilenet_v2.preprocess_input(truck)


batch = np.concatenate([preprocessed_batch,elaphant,truck],axis=0)
print(batch.shape)

predictions = model.predict(batch) 
print(predictions.shape)
print(np.max(elaphant),np.min(elaphant))

For this part I’ll download 2 more images to our current batch namely an elephant and a truck

(1, 224, 224, 3)
(4, 224, 224, 3)
[1m1/1[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m4s[0m 4s/step(4, 1000)1.0 -1.0

Here’s the overall top5

decoded_predictions = keras.applications.mobilenet.decode_predictions(predictions, top=5)  # Top 1 prediction
# Print the result
for pred in decoded_predictions:
    print(pred)

[('n03792782', 'mountain_bike', 0.85207945), ('n04482393', 'tricycle', 0.021734292), ('n04509417', 'unicycle', 0.007280677), ('n02835271', 'bicycle-built-for-two', 0.0055738064), ('n03127747', 'crash_helmet', 0.0029420464)]
[('n03100240', 'convertible', 0.6660847), ('n04285008', 'sports_car', 0.1382453), ('n02974003', 'car_wheel', 0.05493289), ('n04037443', 'racer', 0.026139919), ('n02814533', 'beach_wagon', 0.013757579)]
[('n01871265', 'tusker', 0.45991415), ('n02504013', 'Indian_elephant', 0.2266773), ('n02504458', 'African_elephant', 0.09998358), ('n02109047', 'Great_Dane', 0.0033731572), ('n03709823', 'mailbag', 0.0019660052)]
[('n04467665', 'trailer_truck', 0.99190044), ('n04149813', 'scoreboard', 0.0025432864), ('n03776460', 'mobile_home', 0.0023533762), ('n04065272', 'recreational_vehicle', 0.0003180726), ('n03796401', 'moving_van', 0.00024829843)]

# Grad-CAM function to generate the heatmap
def grad_cam(model, img_array, predicted_class):
    last_conv_layer = model.get_layer('Conv_1')  # Modify this if needed for your model's last conv layer
    grad_model = tf.keras.models.Model([model.inputs], [last_conv_layer.output, model.output])

    with tf.GradientTape() as tape:
        last_conv_layer_output, predictions = grad_model(img_array)
        loss = predictions[:, predicted_class]

    grads = tape.gradient(loss, last_conv_layer_output)
    
    # Compute pooled gradients
    pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))
    
    # Create heatmap
    last_conv_layer_output = last_conv_layer_output[0]
    heatmap = tf.reduce_mean(last_conv_layer_output * pooled_grads, axis=-1)
    heatmap = tf.maximum(heatmap, 0)
    heatmap = heatmap / tf.reduce_max(heatmap)
    
    return 1-heatmap.numpy()

# Overlay heatmap on the image
def overlay_heatmap(heatmap, img_array):
    # Resize the heatmap to match the image size
    heatmap_resized = cv2.resize(heatmap, (img_array.shape[1], img_array.shape[0]))
    
    # Apply the heatmap to the resized version (convert to 0-255 range)
    heatmap_rgb = cv2.applyColorMap(np.uint8(255 * heatmap_resized), cv2.COLORMAP_JET)
    
    # Convert the original image to 0-255 range (correcting for preprocessing in the range [-1, 1])
    img_array_rgb = np.uint8((img_array + 1) * 127.5)  # Convert from [-1,1] to [0, 255]
    
    # Blend the heatmap and the original image with higher heatmap weight
    overlay = cv2.addWeighted(heatmap_rgb, 0.6, img_array_rgb, 0.4, 0)  # Increase heatmap weight
    
    return overlay

# Loop through the batch of images and apply Grad-CAM
for i, img_array in enumerate(batch):
    # Generate the heatmap for the predicted class
    heatmap = grad_cam(model, np.expand_dims(img_array, axis=0), np.argmax(predictions[i]))
    
    # Overlay the heatmap on the image
    overlay_img = overlay_heatmap(heatmap, img_array)

    # Normalize and display the original image correctly
    img_array_rgb = np.uint8((img_array + 1) * 127.5)  # Convert from [-1, 1] to [0, 255]

    # Display the original image and the overlay
    plt.figure(figsize=(10, 5))
    
    plt.subplot(1, 2, 1)
    plt.imshow(np.clip(img_array_rgb, 0, 255))  # Ensure the values are within [0, 255] range
    plt.title(f"Original Image {i + 1}")
    plt.axis('off')

    plt.subplot(1, 2, 2)
    plt.imshow(overlay_img)
    plt.title(f"Overlay {i + 1}")
    plt.axis('off')

    plt.show()