Attention Mechanisms

Study Finds Transformers Collapse on Classic Stroop Test

The paper says a length-linked failure shows transformer attention lacks top-down executive control and recommends adding such mechanisms to improve conflict resolution in models.