How do you use cosine in simpletuner??

[Deathawaits4]

For some reason the learning rate in simpletuner drops to such low values in the first 400 steps (i have 7000 steps) that its basically non existant anymore. Any ideas? I also don't understand the warmup steps, because it'll start with full learning rate, drop down and then up again and only then will behave normally. Shouldn't the lr warmup start with a low value and gradually increase to the target lr?

[bghira]

are you using polynomial? you would probably want constant_with_warmup then.

[Deathawaits4]

Hei, no im using cosine. When setting lr_warmup to 0, it forces me to set a value or remove it entirely from the config and it might use the default value then. Usually cosine shouldn't have such an extrem curve that it drops to basically non existant (1e-6) lr after 500 steps of 7000 :/

The behaviour of lr_warmup_steps is just very strange to me, atleast in other trainers it behaves like this: Starts with low lr until it reaches desired lr and will then continue with the default behaviour of the set scheduler. In simpletuner its like this: Starts of with the set lr, drops to basically 0 and then goes back up to the default lr and then uses the scheduler lr. But this behaviour does not seem optimal at all.

39/7840 [07:32<24:46:02, 11.43s/it, lr=0.000591, step_loss=0.116
234/7840 [50:24<24:25:12, 11.56s/it, lr=0.00033, step_loss=0.161
468/7840 [1:41:48<23:42:38, 11.58s/it, lr=6.44e-6, step_loss=0.185
550/7840 [2:00:48<36:20:30, 17.95s/it, lr=1.51e-5, step_loss=0.173

i don't think this is desired, no?

[bghira]

oh. cosine is a cyclic scheduler. it approximates a cosine wave. it is not the typical ML definition of cosine schedule where it uses a cosine decay with hard restarts.

similarly, the Sine scheduler is also cyclic and approximates a sine wave.

[bghira]

with these schedulers, lr_warmup_steps is actually defining their interval between cycles.

[Deathawaits4]

Oh, i understand. I think it would be a little less confusing to add an extra option for cycles, but no that i know it's alright.