noether.core.providers

Submodules

Classes

MetricPropertyProvider

Provider for properties about metrics, mainly the ordinality of a metric, i.e., is it better if it is higher

PathProvider

Provider that defines at which locations things are stored on the disk. The basic layout is that every training

Package Contents

class noether.core.providers.MetricPropertyProvider

Provider for properties about metrics, mainly the ordinality of a metric, i.e., is it better if it is higher (e.g, accuracy of a classifier) or better if it is lower (e.g., a mean-squared error or any other loss). Additionally, a concept of “neutral” metrics is introduced, which are things that are logged throughout training but there is no concept of “best” metric (e.g., a learning rate that is scheduled, weight decay, …).

By default, the following patterns are handled: - “loss” (lower is better) - “error” (lower is better) - “accuracy” (higher is better)

Callbacks or trainers can freely add patterns if they introduce new metrics that are not handled.

The MetricPropertyProvider is used by the experiment tracker to automatically create summary values and log them to the online interface/log files and also by early stopping mechanisms to enable flexible early stopping where the early stopping criteria can use loss values, but also, e.g., validation accuracies.

static register_pattern(pattern, ordinality)

Allows users (Trainers/Callbacks) to add new patterns easily.

Parameters:
logger
static get_ordinality(key)

Determines the ordinality of a metric key based on registered patterns. The search is case-insensitive.

Parameters:

key (str)

Return type:

Ordinality | None

is_neutral_key(key)

Returns True if the passed key is a neutral metric.

Parameters:

key (str)

Return type:

bool

higher_is_better(key)

Returns whether or not the passed key is better if the metric is higher. Logs a warning if the key is unmatched.

Parameters:

key (str) – The key to check.

Return type:

bool

class noether.core.providers.PathProvider(output_root_path, run_id, stage_name=None, debug=False)

Provider that defines at which locations things are stored on the disk. The basic layout is that every training run is identified by a stage_name (e.g., “pretrain” or “finetune”) and run_id, a string that is unique for each training run. All outputs are stored in a directory defined in the configuration that is located in output_path. The outputs of a single run will be stored in output_path/stage_name/run_id.

Parameters:

  • output_root_path (pathlib.Path) – The base output directory where outputs should be stored (e.g., /save).

  • run_id (str) – Unique identifier of the training run.

  • stage_name (str | None) – Optional identifier for the training stage for easier categorization (e.g., “pretrain” or “finetune”).

  • debug (bool) – If True, outputs are stored in a “debug” subfolder.

output_root
stage_name = None
run_id
debug = False
with_run(run_id=None, stage_name=None)
Parameters:

  • run_id (str | None)

  • stage_name (str | None)

Return type:

PathProvider

property run_output_path: pathlib.Path

Returns the output_path for a given stage_name and run_id.

Returns:

The output path for the current run.

Return type:

pathlib.Path

property logfile_uri: pathlib.Path

Returns the URI where the logfile should be stored (the file where stdout messsage are stored).

Return type:

pathlib.Path

property checkpoint_path: pathlib.Path

Returns the checkpoint path of the current run.

Return type:

pathlib.Path

property basetracker_config_uri: pathlib.Path

Independent of whether or not (or which) online tracker is used, the log entries are also written to disk. This property defines where the config is written to.

Return type:

pathlib.Path

property basetracker_entries_uri: pathlib.Path

Independent of whether or not (or which) online tracker is used, the log entries are also written to disk. This property defines where the log entries are written to.

Return type:

pathlib.Path

property basetracker_summary_uri: pathlib.Path

Independent of whether or not (or which) online tracker is used, the log entries are also written to disk. This property defines where the summary is written to.

Return type:

pathlib.Path

static generate_run_id(seed=None)

Generate a random run ID.

Parameters:

seed – Optional seed for reproducibility.

Returns:

A random run ID.

Return type:

str

Create a symlink from the current run output to the ancestor run output.

Parameters:

ancestor (PathProvider) – The ancestor PathProvider to link to.

Return type:

None