noether.modeling.modules.layers.rope_frequency¶

Classes¶

RopeFrequency

Creates frequencies for rotary embeddings (RoPE) from https://arxiv.org/abs/2104.09864 for variable positions.

Module Contents¶

class noether.modeling.modules.layers.rope_frequency.RopeFrequency(config)¶

Bases: torch.nn.Module

Creates frequencies for rotary embeddings (RoPE) from https://arxiv.org/abs/2104.09864 for variable positions.

Parameters:: config (noether.core.schemas.modules.layers.RopeFrequencyConfig) – Configuration for RoPE frequency settings. See RopeFrequencyConfig for available options.

omega: torch.Tensor¶

hidden_dim¶

input_dim¶

implementation¶

ndim_padding¶

sincos_padding¶

max_wavelength¶

padding¶

forward(coords)¶

Parameters:: coords (torch.Tensor) – coordinates to create RoPE frequencies for. Expected shape is (…, input_dim).
Return type:: torch.Tensor | tuple[torch.Tensor, Ellipsis]