ChannelDropBack: Forward-Consistent Stochastic Regularization for Deep Networks

Evgeny Hershkovitch Neiterman, Gil Ben-Artzi

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    Abstract

    Incorporating stochasticity into the training process of deep convolutional networks is a widely used technique to reduce overfitting and improve regularization. Existing techniques often require modifying the architecture of the network by adding specialized layers, are effective only to specific network topologies or types of layers - linear or convolutional, and result in a trained model that is different from the deployed one. We present ChannelDropBack, a simple stochastic regularization approach that introduces randomness only into the backward information flow, leaving the forward pass intact. ChannelDropBack randomly selects a subset of channels within the network during the backpropagation step and applies weight updates only to them. As a consequence, it allows for seamless integration into the training process of any model and layers without the need to change its architecture, making it applicable to various network topologies, and the exact same network is deployed during training and inference. Experimental evaluations validate the effectiveness of our approach, demonstrating improved accuracy on popular datasets and models, including ImageNet and ViT. Code is available at https://github.com/neiterman21/ChannelDropBack.git.

    Original languageEnglish
    Title of host publicationPattern Recognition - 27th International Conference, ICPR 2024, Proceedings
    EditorsApostolos Antonacopoulos, Subhasis Chaudhuri, Rama Chellappa, Cheng-Lin Liu, Saumik Bhattacharya, Umapada Pal
    PublisherSpringer Science and Business Media Deutschland GmbH
    Pages390-400
    Number of pages11
    ISBN (Print)9783031783821
    DOIs
    StatePublished - 2025
    Event27th International Conference on Pattern Recognition, ICPR 2024 - Kolkata, India
    Duration: 1 Dec 20245 Dec 2024

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume15324 LNCS
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Conference

    Conference27th International Conference on Pattern Recognition, ICPR 2024
    Country/TerritoryIndia
    CityKolkata
    Period1/12/245/12/24

    Keywords

    • Deep Learning
    • Regularization
    • Stochastic Training

    Fingerprint

    Dive into the research topics of 'ChannelDropBack: Forward-Consistent Stochastic Regularization for Deep Networks'. Together they form a unique fingerprint.

    Cite this