Which of the following is the advantage of Leaky ReLU compared to ReLU?

Question

Which of the following is the advantage of Leaky ReLU compared to ReLU?

BlackTom AI · Accepted Answer

Reframe the question and each option in terms of Leaky ReLU versus standard ReLU to understand their practical differences.
Option 1: 'Its output is always positive' — This is inaccurate. Leaky ReLU allows a small, nonzero slope for negative inputs, so outputs can still be negative for sufficiently negative inputs. Therefore, the claim that outputs are always positive is incorrect.
Option 2: 'There’s no saturation on the positive end' — Both Leaky ReLU and standard ReLU are linear and unbounded for positive inputs, so neither saturates as the input grows. This makes the statement true for both activations, not a distinctive advantage of Leaky ReLU over ReLU. Since the question asks for the advantage of Leaky ReLU compared to ReLU, this option does not capture a unique benefit.
Option 3: 'There’s no “dead” neuron when computing gradients' — This statement highlights a key practical advantage: ReLU can produce exactly zero gradients for negative inputs, potentially causing some neurons to “die” during training if they stay in the zero region. Leaky ReLU, by allowing a small slope for negative inputs, preserves some gradient flow even when the input is negative, reducing the risk of permanently inactive neurons. This is a well-known advantage attributed to Leaky ReLU.
Option 4: 'It is cheap to compute' — In practice, both Leaky ReLU and ReLU are very cheap to compute (involving a comparison or a simple conditional and a multiplication for Leaky). The computational cost is similar, so this is not a distinctive advantage of Leaky ReLU over ReLU.
In summary, evaluating each option shows that Option 3 uniquely captures the substantive advantage of Leaky ReLU over standard ReLU, namely maintaining gradient flow for negative inputs to mitigate dead neurons. The other statements either misrepresent the activation's outputs, overstate the uniqueness of a property shared with ReLU, or describe a non-unique or negligible difference in computation cost.

CS-7643-O01, OAN, OSZ Quiz #2: Optimization of Deep Neural Networks (Lesson 3) and Convolution and Pooling Layers (Lesson 5)

Which of the following is the advantage of Leaky ReLU compared to ReLU?

查看解析

登录即可查看完整答案

类似问题

What of the following are commonly used activation functions for neural networks? I Exponential function. II Rectified Linear Unit (ReLU) function. III Sigmoid function. IV Logarithm function.

哪个激活函数保持恒定的导数？ Which activation function maintains constant derivatives?

Which of these plots represents a ReLU activation function?

Question at position 22 What is the purpose of the activation function in an MLP?It determines the loss function used in the MLP. It controls the learning rate of the MLP algorithm.It provides the non-linearity of the MLP network. It determines the number of hidden layers in the MLP.

Question at position 7 What activation functions are commonly used in hidden layers of neural networks? (Select all possible answers)SigmoidReLUSoftmaxTanhLinearUnit Step

What does the ReLU activation function do?

更多留学生实用工具

考试浏览器助手

风格化写作助手

论文查重助手

文献引用助手

课堂转译助手

课堂笔记助手

Quiz搜索助手

学校历年真题

智能刷题助手

智能匹配练习

希望你的学习变得更简单