• Import
• Transformer Configuration Structure
• Transformer Tuner

Transformer Guide

ForeBlocks ships a flexible encoder-decoder transformer stack centered on TransformerEncoder and TransformerDecoder.

The current implementation supports:

• Training optimization: Layer-wise LR decay (LLRD) + warmup-cosine scheduler for SOTA fine-tuning
• Per-layer dropout schedule: Depth-scaled attention dropout (deeper layers → higher dropout)
• multiple attention backends and per-layer attention routing
• encoder and decoder patching
• CT-PatchTST-style encoder tokenization
• paper-style Attention Residuals
• GateSkip (residual path gating)
• Mixture-of-Depths (MoD) dynamic layer skipping
• mHC residual stream mixing (manifold-constrained hyper-connections)
• MoE feedforward blocks with multiple routers and load-balancing
• gradient checkpointing and shared-layer reuse

Related docs:

• Documentation Overview
• Getting Started
• Custom Blocks
• Advanced Transformer Features — LLRD, per-layer dropout, GateSkip, MoD, mHC, attention variants
• Advanced MoE — routers, load-balancing, expert types, production tuning
• MoE
• DARTS

Import

from foreblocks import TransformerEncoder, TransformerDecoder
```text

## Baseline decoder

```python
decoder = TransformerDecoder(
    input_size=1,
    output_size=1,
    d_model=256,
    nhead=8,
    num_layers=4,
    patch_decoder=False,
    informer_like=False,
)
```text

#### `attention.architecture` — routing schedule

Supported `attention.architecture` values currently include:

- `standard`
- `linear`
- `sype`
- `hybrid`
- `kimi`
- `hybrid_kimi`
- `kimi_3to1`
- `gated_delta`
- `hybrid_gdn`
- `gdn_3to1`
- `gla`
- `gla_hybrid`
- `gla_3to1`
- `deltanet`
- `deltanet_hybrid`
- `deltanet_3to1`
- `gated_deltanet`
- `gated_deltanet_hybrid`
- `gated_deltanet_3to1`

Important behavior:

- if `attention.architecture="standard"` but the variant `name` is a routed type such as `linear`, `sype`, `kimi`, `gated_delta`, `gla`, `deltanet`, or `gated_deltanet`, the model promotes the architecture automatically

### Patching

- `patch_encoder`
- `patch_decoder`
- `patch_len`
- `patch_stride`
- `patch_pad_end`

### Efficiency

- `use_gradient_checkpointing`
- `share_layers`

### Advanced modules

- Attention Residuals:
  `use_attention_residual`, `attn_residual_type`, `attention_residual_block_size`
- GateSkip:
  `use_gateskip`, `gate_budget`, `gate_lambda`
- MoD:
  `use_mod`, `mod_mode`, `mod_lambda`, `mod_budget_scheduler`
- mHC:
  `use_mhc`, `mhc_n_streams`, `mhc_sinkhorn_iters`, `mhc_collapse`
- MoE:
  `use_moe`, `num_experts`, `top_k`, `moe_aux_lambda`

## Recommended patching strategy

The recommended pattern for forecasting is:

- `patch_encoder=True`
- `patch_decoder=False`

Why:

- the encoder benefits from shorter token sequences
- the decoder stays easier to reason about
- autoregressive decoding stays compatible with `forward_one_step(...)`

`patch_decoder=True` is supported for full-sequence decoding, but it is not compatible with KV-cached incremental decoding.

When the encoder is patched, the memory sequence length becomes patch-token length. The decoder validates that `memory_key_padding_mask` matches the actual memory length, so patched and unpatched masks cannot be mixed silently.

## CT-PatchTST encoder mode

The encoder also supports a channel-token PatchTST-style path:

```python
encoder = TransformerEncoder(
    input_size=8,
    ct_patchtst=True,
    ct_patch_len=16,
    ct_patch_stride=8,
    ct_patch_pad_end=True,
    ct_patch_fuse="linear",  # or "mean"
    d_model=256,
)
```text

### Informer-like decoding

`model_type="informer-like"` changes defaults so that:

- encoder time encoding is enabled
- decoder informer-like behavior is enabled
- decoder prompt masking follows `label_len`

Typical setup:

```python
decoder = TransformerDecoder(
    input_size=1,
    output_size=1,
    model_type="informer-like",
    label_len=12,
    d_model=256,
    nhead=8,
    num_layers=4,
)
```text

Recommended usage:

- first call: pass the available prefix
- later calls: pass either the growing prefix or only the newest token
- once cache exists, the implementation consumes only the newest step

Current constraints:

- requires `patch_decoder=False`
- does not support `use_mod=True`
- does not support `use_mhc=True`

## Active-position masks for time series

Both GateSkip and MoD operate over active positions. The public runtime input is:

- encoder: `gateskip_active_mask`
- decoder: `gateskip_active_mask`

For time series, the intended meaning is:

- `True`: this timestep or token participates in budgeting or routing
- `False`: inactive position such as padding or masked-out region

Default behavior:

- encoder: active positions are derived from `src_key_padding_mask` when available
- decoder: active positions are derived from the user-provided target padding mask
- the auto-generated Informer forecast mask is intentionally not treated as inactivity for GateSkip or MoD

With patching enabled, the active mask is patchified too, so routing stays aligned with patch tokens.

## Attention Residuals

The transformer now implements paper-style Attention Residuals rather than the older local residual trick.

Controls:

- `use_attention_residual`
- `attn_residual_type`: `full` or `block`
- `attention_residual_block_size`

Behavior:

- `full`: aggregates over the running layer history
- `block`: aggregates over block summaries

Notes:

- this is enabled by default
- it replaces the normal residual path for the affected blocks

Current compatibility rules:

- not compatible with `use_gateskip=True`
- not compatible with `use_mhc=True`
- not compatible with `use_mod=True`

If you want GateSkip, MoD, or mHC, disable Attention Residuals explicitly:

```python
use_attention_residual=False
```text

See the dedicated guide for routing and auxiliary-loss details:

- [MoE Guide](moe)

## Integration with `ForecastingModel`

```python
from foreblocks import ForecastingModel, TransformerEncoder, TransformerDecoder

encoder = TransformerEncoder(
    input_size=8,
    d_model=128,
    nhead=4,
    num_layers=3,
    patch_encoder=True,
    patch_len=16,
    patch_stride=8,
)

decoder = TransformerDecoder(
    input_size=1,
    output_size=1,
    d_model=128,
    nhead=4,
    num_layers=3,
    patch_decoder=False,
    informer_like=False,
)

model = ForecastingModel(
    encoder=encoder,
    decoder=decoder,
    forecasting_strategy="transformer_seq2seq",
    model_type="transformer",
    target_len=24,
    output_size=1,
)

Transformer Configuration Structure

ForeBlocks now uses a structured configuration system for transformers and attention:

• TransformerConfig: Main transformer configuration
• AttentionConfig: Attention settings with sub-configs:
  • shape: AttentionShapeConfig (d_model, n_heads, n_kv_heads, dropout, max_seq_len, cross_attention)
  • architecture: Attention mode (standard, linear, gla, deltanet, etc.)
  • cache: AttentionCacheConfig (use_paged_cache, block_size, max_blocks, use_mla, attention_matching)
  • position: AttentionPositionConfig (encoding, rope_base, rope_scaling_type, rope_scaling_factor)
  • variant: AttentionVariantConfig (name, backend, window_size, chunk_size, frequency_modes, use_flash_sliding, use_swiglu, nsa_block_size, moba_topk, dilation)
  • features: AttentionFeatureConfig (qk_norm, logit_softcap, learned_temperature, gated_attention, normalized_output, head_importance, multiscale_mask, subquery_norm)

Transformer Tuner

ForeBlocks provides ModernTransformerTuner for auto-hyperparameter selection and feature analysis. The tuner has been modernized and now includes:

• Lempel-Ziv complexity analysis for sequence structure
• Continuous Wavelet Transform (CWT) energy features for frequency domain analysis

Note: TransformerTuner has been removed; use ModernTransformerTuner instead.