Jarvis' Blog (总有美丽的风景让人流连) 总有美丽的风景让人流连

Useful Figures in Deep Learning

2021-06-04
2022-03-17
Jarvis
Post

提示

Activation Functions

  • 激活函数

图 1. Activation functions.

Softmax 的温度系数

图 2. Temperature of Softmax.

Exponential Moving Average (EMA/ema)

图 3. Exponential Moving Average with different momentum values.

Learning Rate Decay Policy

  • Polynomial
1
2
3
def polylr(it, init_lr, max_iter, eta_min=0, gamma=0.9):
    factor = eta_min + (init_lr - eta_min) * (1 - (it - 1) / (max_iter - 1)) ** gamma
    return factor

图 4. Polynomial learning rate

  • Cosine
1
2
3
def cosinelr(it, init_lr, T_max, eta_min=0):
    factor = eta_min + 0.5 * (init_lr - eta_min) * (1 + np.cos(((it - 1) % T_max) / (T_max - 1) * np.pi))
    return factor

图 5. Cosine learning rate


Content