DocumentationArchitecture

Attention

A zoomed-in view of one operation in Piro’s stateful inference loop.

Architecture detail

Attention

Deferred CTM architecture: retrieves working history with synchronization-aware attention; not part of the baseline contract.

← Full model

Attention(
    hₖ,
    historyₖ,
    x,
    k,
    weights
):
    memoryₖ = BuildMemorySlots(
        historyₖ,
        k
    )
    syncFeaturesₖ = SummarizeSynchronization(
        hₖ,
        historyₖ,
        weights
    )
    attentionShape = GetAttentionShape(weights)
    d_head = attentionShape.d_head
    queryₖ = QueryProjection(
        Normalize(Concatenate(hₖ, x, syncFeaturesₖ)),
        weights
    )
    keysₖ = KeyProjection(
        memoryₖ,
        weights
    )
    valuesₖ = ValueProjection(
        memoryₖ,
        weights
    )
    contentScoresₖ = queryₖ · keysₖᵀ / sqrt(d_head)
    timeBiasₖ = RelativeTimeBias(
        memoryₖ.age,
        weights
    )
    syncBiasₖ = SynchronizationBias(
        hₖ,
        memoryₖ,
        weights
    )
    contextₖ = OutputProjection(
        contentScoresₖ,
        timeBiasₖ,
        syncBiasₖ,
        valuesₖ,
        weights
    )
    readGateₖ = ReadGate(
        hₖ,
        x,
        contextₖ,
        weights
    ))
    return readGateₖ ⊙ contextₖ

What it does

Deferred CTM architecture: builds a recurrent query and scores timestamped working-memory slots.

Why it exists

The baseline can test self-updating weights without requiring a specialized memory-attention mechanism.

← Back to architecture