noether.core.schemas.models.transformer

Classes

TransformerConfig

Configuration for a Transformer model.

Module Contents

class noether.core.schemas.models.transformer.TransformerConfig(/, **data)

Bases: noether.core.schemas.modules.blocks.TransformerBlockConfig, noether.core.schemas.models.base.ModelBaseConfig

Configuration for a Transformer model.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (Any)

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

depth: int

Number of transformer blocks in the model.

mlp_expansion_factor: int = 4

Expansion factor for the MLP hidden dimension relative to the hidden dimension. If ‘mlp_hidden_dim’ is not set, this factor is used to compute it as hidden_dim * mlp_expansion_factor.

set_mlp_hidden_dim()