noether.core.schedules.linear_warmup_cosine_decay¶

Classes¶

LinearWarmupCosineDecaySchedule

A cosine annealing scheduler with linear increasing warmup phase."

Module Contents¶

class noether.core.schedules.linear_warmup_cosine_decay.LinearWarmupCosineDecaySchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A cosine annealing scheduler with linear increasing warmup phase.”

Example

schedule_config:
    kind: noether.core.schedules.LinearWarmupCosineDecaySchedule
    warmup_percent: 0.05
    end_value: 1.0e-6
    max_value: ${model.optim.lr} # or just manually set the max value

Takes either warmup_steps or warmup_percent as argument to determine the length of the warmup phase.

Parameters:: config (noether.core.schemas.schedules.LinearWarmupCosineDecayScheduleConfig) – Configuration for the linear warmup cosine decay schedule. See LinearWarmupCosineDecayScheduleConfig for details.

schedule: noether.core.schedules.base.ScheduleBase¶

warmup_steps¶

warmup_percent¶