noether.core.schedules¶

Submodules¶

Classes¶

`DecreasingProgressSchedule`	Base class for schedules that monotonically decrease in value over time.
`IncreasingProgressSchedule`	Base class for schedules that monotonically increase in value over time.
`ProgressSchedule`	Base class for schedules that progress monotonically in value over time.
`ScheduleBase`	Base class for schedules.
`SequentialPercentSchedule`	A scheduler that switches between multiple schedules based on the percentage of steps completed.
`SequentialPercentScheduleConfig`
`SequentialStepSchedule`	A scheduler that switches between multiple schedules based on the step number.
`SequentialStepScheduleConfig`
`ConstantSchedule`	Constant value schedule that returns the same value for all steps.
`ConstantScheduleConfig`
`CosineDecreasingSchedule`	Cosine annealing scheduler with decreasing values.
`CosineIncreasingSchedule`	Cosine annealing scheduler with increasing values.
`CustomSchedule`	Custom schedule that simply returns the values provided in the constructor.
`CustomScheduleConfig`
`LinearDecreasingSchedule`	A scheduler that decreases linearly from the maximum to minimum value over the total number of steps.
`LinearIncreasingSchedule`	A scheduler that increases linearly from the minimum to maximum value over the total number of steps.
`LinearWarmupCosineDecaySchedule`	A cosine annealing scheduler with linear increasing warmup phase."
`PolynomialDecreasingSchedule`	A scheduler that decreases polynomially from the maximum to minimum value over the total number of steps.
`StepDecreasingSchedule`	A scheduler that decreases exponentially from the maximum to minimum value over the total number of steps.
`StepFixedSchedule`	A scheduler that progresses at fixed steps and increases or decreases by some factor at these steps.
`StepIntervalSchedule`	A scheduler that progresses at fixed intervals and increases or decreases by some factor at these intervals.

Functions¶

`cosine`(step, total_steps)	cosine schedule from [0 to 1]
`linear`(step, total_steps)	linearly increasing from [0 to 1]
`polynomial`(step, total_steps, power)	polynomial schedule from [0 to 1]

Package Contents¶

class noether.core.schedules.DecreasingProgressSchedule(config)¶

Bases: ProgressSchedule

Base class for schedules that monotonically decrease in value over time.

Initialize the scheduler.

Parameters:: config (noether.core.schemas.schedules.DecreasingProgressScheduleConfig)

class noether.core.schedules.IncreasingProgressSchedule(config)¶

Bases: ProgressSchedule

Base class for schedules that monotonically increase in value over time.

Initialize the scheduler.

Parameters:: config (noether.core.schemas.schedules.IncreasingProgressScheduleConfig)

class noether.core.schedules.ProgressSchedule(config, start_value, delta)¶

Bases: ScheduleBase

Base class for schedules that progress monotonically in value over time.

Initialize the scheduler.

Parameters:

config (noether.core.schemas.schedules.ProgressScheduleConfig) – Configuration for the progress schedule.
start_value (float) – The initial value of the scheduler.
delta (float) – The total change in value over the schedule.

start_value¶

delta¶

exclude_first¶

exclude_last¶

class noether.core.schedules.ScheduleBase(overhang_percent=None, overhang_steps=None)¶

Base class for schedules.

Initialize the scheduler.

Parameters:

overhang_percent (float | None) – The percentage by which the schedule is artificially prolonged.
overhang_steps (int | None) – The number of steps by which the schedule is artificially prolonged.

overhang_percent = None¶

overhang_steps = None¶

get_value(step, total_steps)¶

Get the value of the schedule at a given step.

This function includes the correction for overhangs in percent or steps and then calls the _get_value method. Therefore, it should not be overwritten by subclasses. Instead, the _get_value method should be implemented.

Parameters:

step (int) – The step for which to get the scheduler value.
total_steps (int) – The total number of steps.

Return type:

float

class noether.core.schedules.SequentialPercentSchedule(schedule_configs)¶

Bases: ScheduleBase

A scheduler that switches between multiple schedules based on the percentage of steps completed.

Initialize the scheduler.

Parameters:

schedule_configs (collections.abc.Sequence[SequentialPercentScheduleConfig]) – A list of schedule configurations.
**kwargs – Additional arguments to pass to the parent class.

schedule_configs¶

get_sequential_schedule_config(step, total_steps)¶

Parameters:

step (int)
total_steps (int)

Return type:

SequentialPercentScheduleConfig | None

class noether.core.schedules.SequentialPercentScheduleConfig¶

schedule: ScheduleBase¶

start_percent: float | None = None¶

end_percent: float | None = None¶

class noether.core.schedules.SequentialStepSchedule(schedule_configs, **kwargs)¶

Bases: ScheduleBase

A scheduler that switches between multiple schedules based on the step number.

Initialize the scheduler.

Parameters:

schedule_configs (collections.abc.Sequence[SequentialStepScheduleConfig]) – A list of schedule configurations.
**kwargs – Additional arguments to pass to the parent class.

schedule_configs¶

get_sequential_schedule_config(step)¶

Parameters:: step (int)
Return type:: SequentialStepScheduleConfig | None

class noether.core.schedules.SequentialStepScheduleConfig¶

schedule: ScheduleBase¶

start_step: int | None = None¶

end_step: int | None = None¶

class noether.core.schedules.ConstantSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

Constant value schedule that returns the same value for all steps.

Example

schedule_config:
    kind: noether.core.schedules.ConstantSchedule
    value: ${model.optim.lr}

Parameters:: config (noether.core.schemas.schedules.ConstantScheduleConfig) – Configuration of the constant schedule. See ConstantScheduleConfig for details.

value¶

class noether.core.schedules.ConstantScheduleConfig(/, **data)¶

Bases: ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.ConstantSchedule'] = 'noether.core.schedules.ConstantSchedule'¶: The fully qualified class name of the scheduler.

value: float¶: The constant value that will be returned for all steps. Value should be equal to the learning rate defined in the optimizer.

class noether.core.schedules.CosineDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

Cosine annealing scheduler with decreasing values.

Example

Initialize the scheduler.

Parameters:: config (noether.core.schemas.schedules.DecreasingProgressScheduleConfig)

class noether.core.schedules.CosineIncreasingSchedule(config)¶

Bases: noether.core.schedules.base.IncreasingProgressSchedule

Cosine annealing scheduler with increasing values.

Example

Initialize the scheduler.

Parameters:: config (noether.core.schemas.schedules.IncreasingProgressScheduleConfig)

class noether.core.schedules.CustomSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

Custom schedule that simply returns the values provided in the constructor.

Example

schedule_config:
    kind: noether.core.schedules.CustomSchedule
    values:
        - 1.0e-3
        - 5.0e-4
        - 1.0e-4

Parameters:: config (noether.core.schemas.schedules.CustomScheduleConfig) – Configuration of the custom schedule. See CustomScheduleConfig for details.

values¶

class noether.core.schedules.CustomScheduleConfig(/, **data)¶

Bases: ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.CustomSchedule'] = 'noether.core.schedules.CustomSchedule'¶: The fully qualified class name of the scheduler.

values: list[float]¶: The list of values that will be returned for each step. Values show ben as long as the number of steps.

noether.core.schedules.cosine(step, total_steps)¶

cosine schedule from [0 to 1]

Parameters:

step (int)
total_steps (int)

Return type:

float

noether.core.schedules.linear(step, total_steps)¶

linearly increasing from [0 to 1]

Parameters:

step (int)
total_steps (int)

Return type:

float

noether.core.schedules.polynomial(step, total_steps, power)¶

polynomial schedule from [0 to 1] https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.PolynomialLR.html

Parameters:

step (int)
total_steps (int)
power (float)

Return type:

float

class noether.core.schedules.LinearDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

A scheduler that decreases linearly from the maximum to minimum value over the total number of steps.

Example

Initialize the scheduler.

Parameters:: config (noether.core.schemas.schedules.DecreasingProgressScheduleConfig)

class noether.core.schedules.LinearIncreasingSchedule(config)¶

Bases: noether.core.schedules.base.IncreasingProgressSchedule

A scheduler that increases linearly from the minimum to maximum value over the total number of steps.

Example

>>> schedule_config:
>>>   kind: noether.core.schedules.LinearIncreasingSchedule
>>>   max_value: ${model.optim.lr}
>>>   start_value: 0.0

Initialize the scheduler.

Parameters:: config (noether.core.schemas.schedules.IncreasingProgressScheduleConfig)

class noether.core.schedules.LinearWarmupCosineDecaySchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A cosine annealing scheduler with linear increasing warmup phase.”

Example

Takes either warmup_steps or warmup_percent as argument to determine the length of the warmup phase.

Parameters:: config (noether.core.schemas.schedules.LinearWarmupCosineDecayScheduleConfig) – Configuration for the linear warmup cosine decay schedule. See LinearWarmupCosineDecayScheduleConfig for details.

schedule: noether.core.schedules.base.ScheduleBase¶

warmup_steps¶

warmup_percent¶

class noether.core.schedules.PolynomialDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

A scheduler that decreases polynomially from the maximum to minimum value over the total number of steps.

Parameters:: config (noether.core.schemas.schedules.PolynomialDecreasingScheduleConfig) – Configuration for the polynomial decreasing schedule. See PolynomialDecreasingScheduleConfig for details.

Example

schedule_config:
    kind: noether.core.schedules.PolynomialDecreasingSchedule
    power: 2.0
    start_value: ${model.optim.lr} # reference to the lr defined above
    end_value: 1e-6

power¶

class noether.core.schedules.StepDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

A scheduler that decreases exponentially from the maximum to minimum value over the total number of steps.

Example

schedule_config:
    kind: noether.core.schedules.StepDecreasingSchedule
    factor: 0.1
    decreases_interval: 0.01
    max_value: ${model.optim.lr}

I.e., after each 1% of the total training steps, the value is multiplied by 0.1.

Parameters:: config (noether.core.schemas.schedules.StepDecreasingScheduleConfig) – The configuration for the scheduler. See StepDecreasingScheduleConfig for details.

factor¶

decreases_interval¶

class noether.core.schedules.StepFixedSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A scheduler that progresses at fixed steps and increases or decreases by some factor at these steps.

Parameters:: config (noether.core.schemas.schedules.StepFixedScheduleConfig) – Configuration for the step fixed schedule.

Example: .. code-block:: yaml

schedule_config:
kind: noether.core.schedules.StepFixedSchedule factor: 0.1 start_value: ${model.optim.lr} steps:

0.01

0.02

0.03

Lower LR by factor 0.1 at 1%, 2%, and 3% of total training steps.

steps¶

start_value¶

factor¶

class noether.core.schedules.StepIntervalSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A scheduler that progresses at fixed intervals and increases or decreases by some factor at these intervals.

Parameters:: config (noether.core.schemas.schedules.StepIntervalScheduleConfig) – Configuration for the step interval schedule.

Example:

start_value¶

factor¶

update_interval¶