Which innovation is at the core of the transformer architecture and enables modeling long-range dependencies effectively?

Question

BlackTom AI · Accepted Answer

The question asks which innovation sits at the core of the transformer architecture and enables modeling long-range dependencies effectively.
Option 1: Recurrent hidden states. While recurrent networks rely on hidden states to carry information across timesteps, transformers eschew recurrence entirely. This makes recurrent hidden states an approach not characteristic of the transformer design.
Option 2: The attention mechanism. This is the hallmark of transformers: attention allows the model to weigh and aggregate information from all positions in the input sequence, enabling direct modeling of long-range dependencies without sequential processing. This mechanism replaces the need for maintaining a persistent hidden state over time.
Option 3: Backpropagation. Backpropagation is a general learning algorithm used to train many neural networks, including transformers, but it is not unique to transformers and does not confer the ability to model long-range dependencies on its own. It’s a training method rather than a core architectural innovation.
Option 4: Reinforcement learning. Reinforcement learning concerns learning via trial-and-error interaction with an environment and a reward signal. It is not the defining innovation of the transformer architecture, which is based on attention and feed-forward processing rather than RL frameworks.
In summary, the central feature that enables transformers to model dependencies across distant tokens and parallelize computation is the attention mechanism, whereas the other options describe concepts that are either unrelated to the transformer’s core design or represent broader training paradigms.

Fall 2025.FIN.5321.02 Final Exam

Which innovation is at the core of the transformer architecture and enables modeling long-range dependencies effectively?

View Explanation

Log in for full answers

Similar Questions

Which of the following attention models uses a subset of the input to derive the output, and can not be trained directly with gradient methods?

Why is the attention mechanism particularly suitable for modeling financial time series?

Which of the following statements is correct about query, key, and value in transformer models?

Consider a single-headed attention layer. What happens to the dimensions of the value weight matrix Wv, when we double the maximum input sequence length? Select all that apply

Please read the following paper to answer the below question. https://arxiv.org/pdf/1409.0473.pdf Links to an external site. Based on your reading of the paper, which of the following are true?

More Practical Tools for Students Powered by AI Study Helper

Homework AI Solver

Stylized AI Paper Writer

Plagiarism Checker Assistant

Citation AI Academic Writing Tool

In-Class Translation Assistant

AI Note Generator

AI Quiz Answers

Past Exam Questions from University Test Bank

Smart Practice Assistant

Adaptive Practice

Making Your Study Simpler