Citation Alert: Vaswani 2017 + 2 more

New protein preprint extends Vaswani's Transformer and benchmarks against AlphaFold2 speed

This week's verified citers in protein modelling mostly use the Transformer as a building block and AlphaFold2 as a data source or speed baseline. Disambiguation: Yann LeCun resolved to OpenAlex author A5001226970 (ORCID 0000-0002-1992-2684); no verified new citers were found for him this week.

The 30-second version
  1. Pourmirzaei et al. train protein autoregressive transformers on latent tokens and report sampling about 1,000x faster than AlphaFold2.
  2. Feller et al. use a Vaswani-style transformer decoder over ESMC embeddings for stability prediction, beating ProSST-2048 on ProteinGym.
  3. Liu et al. build IDiom, a protein language model trained on 54M disordered regions curated from AlphaFold2 predictions.
01arXiv cs.LG · Oct 2

Latent protein languages make autoregressive transformers competitive, and ~1,000x faster than AlphaFold2

"Learning Latent Protein Languages for Autoregressive Generation" by Mahdi Pourmirzaei, Farzaneh Esmaili, Amir Ziashahabi et al., arXiv:2610.03978, posted Oct 2, 2026. Not peer-reviewed (preprint).

The authors introduce two learned token alphabets for proteins: a 4,096-state sequence language built on a frozen ESM-2 encoder, and a structure language that decodes to backbone coordinates. Each is pretrained with a decoder-only autoregressive transformer. Vaswani (extends): the paper says this "follows the standard Transformer and GPT-2 recipe, while changing only the protein token space." AlphaFold2 (compares against): the authors state candidate-token generation for long proteins is approximately 1,000 times faster than MSA-based AlphaFold2 on an NVIDIA A100 GPU, in their measurements.

For your question: this is the clearest extension of the Transformer recipe to protein sequence and structure this week; check the speed claim against AF2 quality before citing it.

02arXiv q-bio.BM · Sep 30

StabilityArc applies a Vaswani-style transformer decoder to protein stability prediction

"StabilityArc: Decoding Protein Sequence Embeddings into Generalizable Stability Landscapes" by Aaron L. Feller, Andrew D. Ellington, Claus O. Wilke, arXiv:2610.00742, posted Sep 30, 2026. Not peer-reviewed (preprint).

A shared four-layer transformer, built on the Vaswani et al. architecture with rotary embeddings, decodes frozen ESMC-600M residue embeddings into substitution effects. Vaswani (applies): the layers are cited for the decoder itself. On 66 leave-one-protein-out ProteinGym stability assays (134,794 variants) the authors report 0.7134 Spearman against 0.6526 for ProSST-2048. They note ProSST uses AlphaFold2-derived structure tokens, though this paper does not cite Jumper et al. for that. As a prior for Kermut, they report 0.8280 versus Kermut's reported 0.8167.

For your question: a sequence-only transformer decoder is reported to beat structure-token baselines on stability; replicate on your own assays before relying on the gap.

03arXiv cs.LG · Oct 1

IDiom language model for disordered regions is trained on AlphaFold2-derived data

"Generative modeling of intrinsically disordered protein regions by reinforcing sparse autoencoder features" by Jason X. Liu, Sebastian Ibarraran, Frank Hu et al., arXiv:2610.02189, posted Oct 1, 2026. Not peer-reviewed (preprint).

The authors present IDiom, an autoregressive protein language model trained on IDiom-DB, 54 million predicted disordered regions curated from the AlphaFold Database, plus an RL method using sparse-autoencoder features to steer generation. Jumper et al. is cited in the introduction as driving progress in folded-protein design. AlphaFold2 (applies): low AF2 pLDDT values are used as the disorder predictor for curating the data, and ColabFold AF2 predictions score generated sequences.

For your question: AF2 confidence scores are being reused as training-data filters for language models; note that pLDDT is a proxy for disorder, not an experimental label.

04Front. Mol. Biosci. · Oct 5

AIPFusion-Net uses transformer attention pooling over ESM2 embeddings for anti-inflammatory peptides

"AIPFusion-Net: an interaction-fusion framework for anti-inflammatory peptide prediction using protein language model embeddings, local sequence patterns, and handcrafted descriptors" by Wang W, Yang W, Li H, Cui Z and Shen D, Frontiers in Molecular Biosciences 13:1952395, published Oct 5, 2026 (peer-reviewed; received Jul 29, accepted Aug 30).

The model fuses ESM2 embeddings, local window patterns and handcrafted descriptors. Vaswani (applies): window embeddings pass through a Transformer encoder and attention pooling is cited to Vaswani et al. 2017. The authors report 84.6% accuracy and AUC 0.884 on Ind-1049, and 82.3% accuracy and AUC 0.882 on Ind-426.

For your question: a routine application of Transformer components to peptide classification; useful as a benchmark only if you work on anti-inflammatory peptides.

Several AlphaFold2 citers (PNAS, ACS and Elsevier journals) could not be opened for citing passages, so they are unclassified and omitted; no LeCun citers were verified.

Coverage: Sep 30 – Oct 6, 2026

Researched with GREP. Every story links to its sources.

Enable JavaScript to read this edition.