noether.modeling.modules.layers.rope_frequency

Classes

RopeFrequencyConfig	Configuration for RoPE frequency settings.
RopeFrequency	Creates frequencies for rotary embeddings (RoPE) from https://arxiv.org/abs/2104.09864 for variable positions.

Module Contents

class noether.modeling.modules.layers.rope_frequency.RopeFrequencyConfig(/, **data)

Bases: pydantic.BaseModel

Configuration for RoPE frequency settings.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (Any)

hidden_dim: int = None

Dimensionality of frequencies (in transformers this should be the head dimension).

input_dim: int = None

Dimensionality of the coordinates (e.g., 2 for 2D coordinates, 3 for 3D coordinates).

max_wavelength: int = None

10000.0

Type:

Theta parameter for the transformer sine/cosine embedding. Default

implementation: Literal['real', 'complex'] = None

“real” -> basic implementation using real coordinates (this is slow and only here for backward compatibility). “complex” -> fast implementation of rotation via complex multiplication. Default: “real”.

class noether.modeling.modules.layers.rope_frequency.RopeFrequency(config)

Bases: torch.nn.Module

Creates frequencies for rotary embeddings (RoPE) from https://arxiv.org/abs/2104.09864 for variable positions.

Parameters:

config (RopeFrequencyConfig) – Configuration for RoPE frequency settings. See RopeFrequencyConfig for available options.

omega: torch.Tensor

hidden_dim

input_dim

implementation

ndim_padding

sincos_padding

max_wavelength

padding

forward(coords)

Parameters:

coords (torch.Tensor) – coordinates to create RoPE frequencies for. Expected shape is (…, input_dim).

Return type:

torch.Tensor | tuple[torch.Tensor, Ellipsis]