Meet Mish — New State of the Art AI Activation Function. The successor to ReLU?

Less Wright
6 min readAug 27, 2019

A new paper by Diganta Misra titled “Mish: A Self Regularized Non-Monotonic Neural Activation Function” introduces the AI world to a new deep learning activation function that shows improvements over both Swish (+.494%) and ReLU (+ 1.671%) on final accuracy.

Our small FastAI team used Mish in place of ReLU as part of our efforts to beat the previous accuracy scores on the FastAI global leaderboard. Combining Ranger optimizer, Mish activation, Flat + Cosine anneal and a self attention layer, we…

--

--

Less Wright

PyTorch, Deep Learning, Object detection, Stock Index investing and long term compounding.