Relu functions

Despite their practical success, Relu activations are theoretically ill-defined. This is due to the existence of numerous trivial local optima where the gradient is zero, and we are optimizing them with approximate local optimizers.

Optimization

#Optimization is the core of #machinelearning. All GPU acceleration, cloud systems, and transfer learning in ML are for better optimization. There are many attempts to make it faster. However, there are no widely adopted second-order methods in the field despite some momentum methods depicting them. I believe there is still potential in this area. Having a robust 10% better algorithm could have billion of dollars of impact on the industry. Of course, it is very hard to achieve.