CNN Robustness to Camera Occlusion

Project Overview

A critical aspect of autonomous system safety is ensuring the perception stack is robust to real-world sensor degradation. This project applies a SOTIF (Safety Of The Intended Functionality)-style methodology to systematically evaluate a CNN-based traffic sign classifier against camera occlusions like rain and dust.

To enable this research, I first developed camera-occlusion, a standalone, open-source Python library for generating realistic, physically-inspired sensor noise. Using this library, I built a config-driven experimental framework to train and test models under various conditions, automatically generating a detailed analysis of performance degradation, model introspection reports, and robustness metrics. This demonstrates a practical, end-to-end approach to identifying and quantifying insufficiencies in ML model performance as required by modern safety standards.

Key Features

• Foundational Tooling: Developed and open-sourced the camera-occlusion library to create realistic data augmentations.
• Config-Driven Pipeline: The entire experimental workflow—from model architecture to training conditions—is controlled via a central YAML file for reproducibility.
• Systematic Evaluation: Trains multiple models on different conditions (clean, light rain, etc.) and evaluates every model against every test condition to build a complete performance matrix.
• Automated Analysis: Automatically generates a detailed markdown report with performance heatmaps, robustness metrics, and statistical significance tests.
• Model Introspection: Includes command-line tools to visualize learned filters and feature map activations, allowing for deep inspection of model behavior under clean vs. noisy conditions.

Results & Impact

The analysis revealed clear patterns in model robustness. For example, training a model on images with simulated light rain not only improved its performance on heavy rain but also generalized better to dust conditions than the model trained on clean data. This provides concrete evidence that targeted, realistic data augmentation is a highly effective strategy for improving the SOTIF of perception systems. The tooling and methodology serve as a blueprint for conducting similar robustness audits on other computer vision models.