noether.core.schedules¶

Submodules¶

Attributes¶

AnyScheduleConfig

Classes¶

`DecreasingProgressSchedule`	Base class for schedules that monotonically decrease in value over time.
`IncreasingProgressSchedule`	Base class for schedules that monotonically increase in value over time.
`ProgressSchedule`	Base class for schedules that progress monotonically in value over time.
`ScheduleBase`	Base class for schedules.
`SequentialPercentSchedule`	A scheduler that switches between multiple schedules based on the percentage of steps completed.
`SequentialPercentScheduleConfig`
`SequentialStepSchedule`	A scheduler that switches between multiple schedules based on the step number.
`SequentialStepScheduleConfig`
`ConstantSchedule`	Constant value schedule that returns the same value for all steps.
`ConstantScheduleConfig`
`CosineDecreasingSchedule`	Cosine annealing scheduler with decreasing values.
`CosineDecreasingScheduleConfig`
`CosineIncreasingSchedule`	Cosine annealing scheduler with increasing values.
`CosineIncreasingScheduleConfig`
`CustomSchedule`	Custom schedule that simply returns the values provided in the constructor.
`CustomScheduleConfig`
`LinearDecreasingSchedule`	A scheduler that decreases linearly from the maximum to minimum value over the total number of steps.
`LinearDecreasingScheduleConfig`
`LinearIncreasingSchedule`	A scheduler that increases linearly from the minimum to maximum value over the total number of steps.
`LinearIncreasingScheduleConfig`
`LinearWarmupCosineDecaySchedule`	A cosine annealing scheduler with linear increasing warmup phase."
`LinearWarmupCosineDecayScheduleConfig`
`PolynomialDecreasingSchedule`	A scheduler that decreases polynomially from the maximum to minimum value over the total number of steps.
`PolynomialDecreasingScheduleConfig`
`PolynomialIncreasingSchedule`	A scheduler that increases polynomially from the minimum to maximum value over the total number of steps.
`PolynomialIncreasingScheduleConfig`
`DecreasingProgressScheduleConfig`
`IncreasingProgressScheduleConfig`
`ProgressScheduleConfig`
`ScheduleBaseConfig`
`SchedulerConfig`
`StepDecreasingSchedule`	A scheduler that decreases exponentially from the maximum to minimum value over the total number of steps.
`StepDecreasingScheduleConfig`
`StepFixedSchedule`	A scheduler that progresses at fixed steps and increases or decreases by some factor at these steps.
`StepFixedScheduleConfig`
`StepIntervalSchedule`	A scheduler that progresses at fixed intervals and increases or decreases by some factor at these intervals.
`StepIntervalScheduleConfig`

Functions¶

`cosine`(step, total_steps)	cosine schedule from [0 to 1]
`linear`(step, total_steps)	linearly increasing from [0 to 1]
`polynomial`(step, total_steps, power)	polynomial schedule from [0 to 1]

Package Contents¶

class noether.core.schedules.DecreasingProgressSchedule(config)¶

Bases: ProgressSchedule

Base class for schedules that monotonically decrease in value over time.

Initialize the scheduler.

Parameters:: config (noether.core.schedules.schemas.DecreasingProgressScheduleConfig)

class noether.core.schedules.IncreasingProgressSchedule(config)¶

Bases: ProgressSchedule

Base class for schedules that monotonically increase in value over time.

Initialize the scheduler.

Parameters:: config (noether.core.schedules.schemas.IncreasingProgressScheduleConfig)

class noether.core.schedules.ProgressSchedule(config, start_value, delta)¶

Bases: ScheduleBase

Base class for schedules that progress monotonically in value over time.

Initialize the scheduler.

Parameters:

config (noether.core.schedules.schemas.ProgressScheduleConfig) – Configuration for the progress schedule.
start_value (float) – The initial value of the scheduler.
delta (float) – The total change in value over the schedule.

start_value¶

delta¶

exclude_first¶

exclude_last¶

class noether.core.schedules.ScheduleBase(overhang_percent=None, overhang_steps=None)¶

Base class for schedules.

Initialize the scheduler.

Parameters:

overhang_percent (float | None) – The percentage by which the schedule is artificially prolonged.
overhang_steps (int | None) – The number of steps by which the schedule is artificially prolonged.

overhang_percent = None¶

overhang_steps = None¶

get_value(step, total_steps)¶

Get the value of the schedule at a given step.

This function includes the correction for overhangs in percent or steps and then calls the _get_value method. Therefore, it should not be overwritten by subclasses. Instead, the _get_value method should be implemented.

Parameters:

step (int) – The step for which to get the scheduler value.
total_steps (int) – The total number of steps.

Return type:

float

class noether.core.schedules.SequentialPercentSchedule(schedule_configs)¶

Bases: ScheduleBase

A scheduler that switches between multiple schedules based on the percentage of steps completed.

Initialize the scheduler.

Parameters:

schedule_configs (collections.abc.Sequence[SequentialPercentScheduleConfig]) – A list of schedule configurations.
**kwargs – Additional arguments to pass to the parent class.

schedule_configs¶

get_sequential_schedule_config(step, total_steps)¶

Parameters:

step (int)
total_steps (int)

Return type:

SequentialPercentScheduleConfig | None

class noether.core.schedules.SequentialPercentScheduleConfig¶

schedule: ScheduleBase¶

start_percent: float | None = None¶

end_percent: float | None = None¶

class noether.core.schedules.SequentialStepSchedule(schedule_configs, **kwargs)¶

Bases: ScheduleBase

A scheduler that switches between multiple schedules based on the step number.

Initialize the scheduler.

Parameters:

schedule_configs (collections.abc.Sequence[SequentialStepScheduleConfig]) – A list of schedule configurations.
**kwargs – Additional arguments to pass to the parent class.

schedule_configs¶

get_sequential_schedule_config(step)¶

Parameters:: step (int)
Return type:: SequentialStepScheduleConfig | None

class noether.core.schedules.SequentialStepScheduleConfig¶

schedule: ScheduleBase¶

start_step: int | None = None¶

end_step: int | None = None¶

class noether.core.schedules.ConstantSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

Constant value schedule that returns the same value for all steps.

Example

schedule_config:
    kind: noether.core.schedules.ConstantSchedule
    value: ${model.optim.lr}

Parameters:: config (ConstantScheduleConfig) – Configuration of the constant schedule. See ConstantScheduleConfig for details.

value¶

class noether.core.schedules.ConstantScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.ConstantSchedule'] = 'noether.core.schedules.ConstantSchedule'¶: The fully qualified class name of the scheduler.

value: float¶: The constant value that will be returned for all steps. Value should be equal to the learning rate defined in the optimizer.

class noether.core.schedules.CosineDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

Cosine annealing scheduler with decreasing values.

Example

schedule_config:
    kind: noether.core.schedules.CosineDecreasingSchedule
    max_value: ${model.optim.lr} # or just manually set the starting value
    end_value: 0.0

Initialize the scheduler.

Parameters:: config (noether.core.schedules.schemas.DecreasingProgressScheduleConfig)

class noether.core.schedules.CosineDecreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.base.DecreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.CosineDecreasingSchedule'] = 'noether.core.schedules.CosineDecreasingSchedule'¶: The fully qualified class name of the scheduler.

class noether.core.schedules.CosineIncreasingSchedule(config)¶

Bases: noether.core.schedules.base.IncreasingProgressSchedule

Cosine annealing scheduler with increasing values.

Example

schedule_config:
    kind: noether.core.schedules.CosineIncreasingSchedule
    max_value: ${model.optim.lr}
    start_value: 0.0

Initialize the scheduler.

Parameters:: config (noether.core.schedules.schemas.IncreasingProgressScheduleConfig)

class noether.core.schedules.CosineIncreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.IncreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.CosineIncreasingSchedule'] = 'noether.core.schedules.CosineIncreasingSchedule'¶: The fully qualified class name of the scheduler.

class noether.core.schedules.CustomSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

Custom schedule that simply returns the values provided in the constructor.

Example

schedule_config:
    kind: noether.core.schedules.CustomSchedule
    values:
        - 1.0e-3
        - 5.0e-4
        - 1.0e-4

Parameters:: config (CustomScheduleConfig) – Configuration of the custom schedule. See CustomScheduleConfig for details.

values¶

class noether.core.schedules.CustomScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.CustomSchedule'] = 'noether.core.schedules.CustomSchedule'¶: The fully qualified class name of the scheduler.

values: list[float]¶: The list of values that will be returned for each step. Values show ben as long as the number of steps.

noether.core.schedules.cosine(step, total_steps)¶

cosine schedule from [0 to 1]

Parameters:

step (int)
total_steps (int)

Return type:

float

noether.core.schedules.linear(step, total_steps)¶

linearly increasing from [0 to 1]

Parameters:

step (int)
total_steps (int)

Return type:

float

noether.core.schedules.polynomial(step, total_steps, power)¶

polynomial schedule from [0 to 1] https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.PolynomialLR.html

Parameters:

step (int)
total_steps (int)
power (float)

Return type:

float

class noether.core.schedules.LinearDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

A scheduler that decreases linearly from the maximum to minimum value over the total number of steps.

Example

schedule_config:
    kind: noether.core.schedules.LinearDecreasingSchedule
    max_value: ${model.optim.lr}
    end_value: 0.0

Initialize the scheduler.

Parameters:: config (noether.core.schedules.schemas.DecreasingProgressScheduleConfig)

class noether.core.schedules.LinearDecreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.base.DecreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.LinearDecreasingSchedule'] = 'noether.core.schedules.LinearDecreasingSchedule'¶: The fully qualified class name of the scheduler.

class noether.core.schedules.LinearIncreasingSchedule(config)¶

Bases: noether.core.schedules.base.IncreasingProgressSchedule

A scheduler that increases linearly from the minimum to maximum value over the total number of steps.

Example

schedule_config:
    kind: noether.core.schedules.LinearIncreasingSchedule
    max_value: ${model.optim.lr}
    start_value: 0.0

Initialize the scheduler.

Parameters:: config (noether.core.schedules.schemas.IncreasingProgressScheduleConfig)

class noether.core.schedules.LinearIncreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.base.IncreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.LinearIncreasingSchedule'] = 'noether.core.schedules.LinearIncreasingSchedule'¶: The fully qualified class name of the scheduler.

class noether.core.schedules.LinearWarmupCosineDecaySchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A cosine annealing scheduler with linear increasing warmup phase.”

Example

schedule_config:
    kind: noether.core.schedules.LinearWarmupCosineDecaySchedule
    warmup_percent: 0.05
    end_value: 1.0e-6
    max_value: ${model.optim.lr} # or just manually set the max value

Takes either warmup_steps or warmup_percent as argument to determine the length of the warmup phase.

Parameters:: config (LinearWarmupCosineDecayScheduleConfig) – Configuration for the linear warmup cosine decay schedule. See LinearWarmupCosineDecayScheduleConfig for details.

schedule: noether.core.schedules.base.ScheduleBase¶

warmup_steps¶

warmup_percent¶

class noether.core.schedules.LinearWarmupCosineDecayScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.ScheduleBaseConfig

Parameters:: data (Any)

max_value: float = None¶: The maximum value of the scheduler from which to start the cosine decay phase. This should be equal to the learning rate defined in the optimizer. I.e., max value is learning rate

kind: Literal['noether.core.schedules.LinearWarmupCosineDecaySchedule'] = 'noether.core.schedules.LinearWarmupCosineDecaySchedule'¶: The fully qualified class name of the scheduler.

warmup_steps: int | None = None¶: The number of steps to linearly increase the value from start to max.

warmup_percent: float | None = None¶: The percentage of steps to linearly increase the value from start to max.

validate_warmup()¶

Ensures that exactly one of ‘warmup_steps’ or ‘warmup_percent’ is specified.

Return type:: LinearWarmupCosineDecayScheduleConfig

class noether.core.schedules.PolynomialDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

A scheduler that decreases polynomially from the maximum to minimum value over the total number of steps.

Parameters:: config (PolynomialDecreasingScheduleConfig) – Configuration for the polynomial decreasing schedule. See PolynomialDecreasingScheduleConfig for details.

Example

schedule_config:
    kind: noether.core.schedules.PolynomialDecreasingSchedule
    power: 2.0
    start_value: ${model.optim.lr} # reference to the lr defined above
    end_value: 1e-6

power¶

class noether.core.schedules.PolynomialDecreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.DecreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.PolynomialDecreasingSchedule'] = 'noether.core.schedules.PolynomialDecreasingSchedule'¶: The fully qualified class name of the scheduler.

power: float = None¶: The power of the polynomial function.

class noether.core.schedules.PolynomialIncreasingSchedule(config)¶

Bases: noether.core.schedules.base.IncreasingProgressSchedule

A scheduler that increases polynomially from the minimum to maximum value over the total number of steps.

Parameters:: config (PolynomialIncreasingScheduleConfig) – Configuration for the polynomial increasing schedule. See PolynomialIncreasingScheduleConfig for details.

Example

schedule_config:
    kind: noether.core.schedules.PolynomialIncreasingSchedule
    power: 2.0
    start_value: 1e-6
    max_value: ${model.optim.lr} # reference to the lr defined above

power¶

class noether.core.schedules.PolynomialIncreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.IncreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.PolynomialIncreasingSchedule'] = 'noether.core.schedules.PolynomialIncreasingSchedule'¶: The fully qualified class name of the scheduler.

power: float = None¶: The power of the polynomial function.

class noether.core.schedules.DecreasingProgressScheduleConfig(/, **data)¶

Bases: ProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.DecreasingProgressSchedule'] = 'noether.core.schedules.DecreasingProgressSchedule'¶: The fully qualified class name of the scheduler.

max_value: float = None¶: Maximum (starting) value of the schedule.

end_value: float = None¶: Minimum (ending) value of the schedule.

class noether.core.schedules.IncreasingProgressScheduleConfig(/, **data)¶

Bases: ProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.IncreasingProgressSchedule'] = 'noether.core.schedules.IncreasingProgressSchedule'¶: The fully qualified class name of the scheduler.

start_value: float = None¶

max_value: float | None = None¶: Minimum (starting) value of the schedule.

class noether.core.schedules.ProgressScheduleConfig(/, **data)¶

Bases: ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.ProgressSchedule'] = 'noether.core.schedules.ProgressSchedule'¶: The fully qualified class name of the scheduler.

exclude_first: bool = None¶: Whether to exclude the first value of the schedule.

exclude_last: bool = None¶: Whether to exclude the last value of the schedule.

class noether.core.schedules.ScheduleBaseConfig(/, **data)¶

Bases: pydantic.BaseModel

Parameters:: data (Any)

kind: str | None = None¶: The fully qualified class name of the scheduler.

overhang_percent: float | None = None¶: The percentage by which the schedule is artificially prolonged. Mutually exclusive with overhang_steps.

overhang_steps: int | None = None¶: The number of steps by which the schedule is artificially prolonged. Mutually exclusive with overhang_percent.

start_value: float = None¶

end_value: float = None¶

weight_decay: float | None = None¶

start_percent: float | None = None¶: The percentage of steps at which the schedule starts.

end_percent: float | None = None¶: The percentage of steps at which the schedule ends.

start_step: int | None = None¶: The step at which the schedule starts.

end_step: int | None = None¶: The step at which the schedule ends.

interval: Literal['update', 'epoch'] = None¶: Whether the schedule is based on updates or epochs. Interval should be either “update” or “epoch”. Default is “update”. Under the hood steps is always used. However, when “epoch” is selected here, the step count is derived from epochs via the UpdateCounter.

check_mutual_exclusion()¶

Ensures that ‘overhang_percent’ and ‘overhang_steps’ are mutually exclusive.

Return type:: ScheduleBaseConfig

validate_start_end_steps()¶

Return type:: ScheduleBaseConfig

validate_start_end_percents()¶

Return type:: ScheduleBaseConfig

class noether.core.schedules.SchedulerConfig(/, **data)¶

Bases: ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.scheduler.SchedulerConfig'] = 'noether.core.schedules.scheduler.SchedulerConfig'¶: The fully qualified class name of the scheduler.

warmup_percent: float = None¶

end_value: float = None¶

class noether.core.schedules.StepDecreasingSchedule(config)¶

Bases: noether.core.schedules.base.DecreasingProgressSchedule

A scheduler that decreases exponentially from the maximum to minimum value over the total number of steps.

Example

schedule_config:
    kind: noether.core.schedules.StepDecreasingSchedule
    factor: 0.1
    decreases_interval: 0.01
    max_value: ${model.optim.lr}

I.e., after each 1% of the total training steps, the value is multiplied by 0.1.

Parameters:: config (StepDecreasingScheduleConfig) – The configuration for the scheduler. See StepDecreasingScheduleConfig for details.

factor¶

decreases_interval¶

class noether.core.schedules.StepDecreasingScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.DecreasingProgressScheduleConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.StepDecreasingSchedule'] = 'noether.core.schedules.StepDecreasingSchedule'¶: The fully qualified class name of the scheduler.

factor: float = None¶: The factor by which the value decreases.

decreases_interval: float = None¶: The interval in range [0, 1] at which the value decreases.

check_interval()¶

Ensures that ‘interval’ is a float in the range (0, 1).

Return type:: StepDecreasingScheduleConfig

class noether.core.schedules.StepFixedSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A scheduler that progresses at fixed steps and increases or decreases by some factor at these steps.

Parameters:: config (StepFixedScheduleConfig) – Configuration for the step fixed schedule.

Example:

schedule_config:
    kind: noether.core.schedules.StepFixedSchedule
    factor: 0.1
    start_value: ${model.optim.lr}
    steps:
        - 0.01
        - 0.02
        - 0.03

Lower LR by factor 0.1 at 1%, 2%, and 3% of total training steps.

steps¶

start_value¶

factor¶

class noether.core.schedules.StepFixedScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.StepFixedSchedule'] = 'noether.core.schedules.StepFixedSchedule'¶: The fully qualified class name of the scheduler.

start_value: float = None¶: The initial value of the scheduler.

factor: float = None¶: The factor by which the value is multiplied after reaching the next step provided in steps.

steps: list[float] = None¶: The steps at which the value changes, must be a list of floats in the range (0, 1).

validate_steps()¶

Ensures that ‘steps’ is a non-empty list of floats in the range (0, 1).

Return type:: StepFixedScheduleConfig

class noether.core.schedules.StepIntervalSchedule(config)¶

Bases: noether.core.schedules.base.ScheduleBase

A scheduler that progresses at fixed intervals and increases or decreases by some factor at these intervals.

Parameters:: config (StepIntervalScheduleConfig) – Configuration for the step interval schedule.

Example:

schedule_config:
    kind: noether.core.schedules.StepIntervalSchedule
    start_value: 1.0
    factor: 0.5
    update_interval: 0.01

start_value¶

factor¶

update_interval¶

class noether.core.schedules.StepIntervalScheduleConfig(/, **data)¶

Bases: noether.core.schedules.schemas.ScheduleBaseConfig

Parameters:: data (Any)

kind: Literal['noether.core.schedules.StepIntervalSchedule'] = 'noether.core.schedules.StepIntervalSchedule'¶: The fully qualified class name of the scheduler.

start_value: float = None¶: The initial value of the scheduler. I.e, the learning rate at step 0.

factor: float = None¶: The factor by which the value is multiplied after reaching the next interval.

update_interval: float = None¶: The interval in range (0, 1) at which the value changes.

check_update_interval(v)¶

Ensures that ‘update_interval’ is a float in the range (0, 1).

Parameters:: v (float)
Return type:: float

noether.core.schedules.AnyScheduleConfig¶