Refined ReLU and Smooth Refined ReLU: A Bounded-Negative Activation Family Bridging ReLU Simplicity and GELU Smoothness

Abstract

The activation functions at the core determine the neural network learning through the lines of non-linearity and gradient propagation through the layers. Rectified Linear Unit (ReLU) brought a revolution to deep learning with its simplistic computation and sparse activations, but its design has a sharp directional gradient, which is zero, causing dead neurons and gradient flow being saturated. Gaussian Error Linear Units (GELU) solve these shortcomings by providing smooth probabilistic transitions at high computational costs through transcendental functions. The Refined ReLU (R-ReLU) and Smooth Refined ReLU (SR-ReLU) activation families strike this efficiency-smoothness trade-off balance with bounded-negative regions whose parameters can be tuned. R-ReLU uses a piecewise linear representation with an adjustable negative slope and threshold, whereas SR-ReLU applies cubic interpolation, which implies continuous first derivatives. Both of its forms have the computational simplicity of ReLU coupled with approximations of the gradient properties of GELU, but rely on simple, polynomially-expressible operations. Experimental results using benchmark datasets show comparable performance in terms of accuracy improvement at low computational costs compared to standard ReLU. The bounded-negative paradigm minimizes the probability of dead neurons and preserves hardware-friendly implementation properties, which can be deployed in resource-constrained environments. Neural tangent-based theoretical analysis shows that SR-ReLU is simulated by the same kernel as smooth activations and has the same convergence behaviour in analytically simple cases. These activations have a deterministic nature and quantization robustness that place them in an advantageous position to be used in edge computing applications, where energy consumption and inference latency are important limitations.