noether.core.schedules.linear_warmup_cosine_decay

Classes

LinearWarmupCosineDecayScheduleConfig
LinearWarmupCosineDecaySchedule	A cosine annealing scheduler with linear increasing warmup phase."

Module Contents

class noether.core.schedules.linear_warmup_cosine_decay.LinearWarmupCosineDecayScheduleConfig(/, **data)

Bases: noether.core.schedules.schemas.ScheduleBaseConfig

Parameters:

data (Any)

max_value: float = None

The maximum value of the scheduler from which to start the cosine decay phase. This should be equal to the learning rate defined in the optimizer. I.e., max value is learning rate

kind: Literal['noether.core.schedules.LinearWarmupCosineDecaySchedule'] = 'noether.core.schedules.LinearWarmupCosineDecaySchedule'

The fully qualified class name of the scheduler.

warmup_steps: int | None = None

The number of steps to linearly increase the value from start to max.

warmup_percent: float | None = None

The percentage of steps to linearly increase the value from start to max.

validate_warmup()

Ensures that exactly one of ‘warmup_steps’ or ‘warmup_percent’ is specified.

Return type:

LinearWarmupCosineDecayScheduleConfig

class noether.core.schedules.linear_warmup_cosine_decay.LinearWarmupCosineDecaySchedule(config)

Bases: noether.core.schedules.base.ScheduleBase

A cosine annealing scheduler with linear increasing warmup phase.”

Example

schedule_config:
    kind: noether.core.schedules.LinearWarmupCosineDecaySchedule
    warmup_percent: 0.05
    end_value: 1.0e-6
    max_value: ${model.optim.lr} # or just manually set the max value

Takes either warmup_steps or warmup_percent as argument to determine the length of the warmup phase.

Parameters:

config (LinearWarmupCosineDecayScheduleConfig) – Configuration for the linear warmup cosine decay schedule. See LinearWarmupCosineDecayScheduleConfig for details.

schedule: noether.core.schedules.base.ScheduleBase

warmup_steps

warmup_percent