Strip a transformer down and you find a modest bargain: every element in a sequence gets to look at every other element and decide how much it cares. That bargain was proposed for machine translation, which is why the architecture got filed under "language." Read the mechanism itself, though, and it never mentions words. It mentions vectors, positions, and weights.

That is the loophole people have been exploiting ever since. An image is a sequence of patches. A recording is a sequence of frames. A protein is a sequence of residues. Once you accept that framing, the same stack of attention layers, feed-forward blocks and residual connections starts showing up in places that have nothing to do with sentences.

Attention is a routing rule, not a language rule

Self-attention does one thing: for each position in a sequence, it computes a weighted average of every other position, where the weights come from learned similarity. Stack that a dozen times and the network can move information across long distances in a single hop. Compare that with a recurrent network, which has to carry meaning forward step by step and slowly forgets, or a convolutional network, which only sees a local window at each layer.

The catch is that attention has no built-in sense of order or geometry. Shuffle the tokens and the output barely changes unless you tell it otherwise. That is why positional information — one-dimensional for text, two-dimensional for images, time-frequency for audio — became such an important design decision. The second catch: attention cost grows with the square of sequence length, which shapes everything downstream.

Vision: patches became the new tokens

The Vision Transformer made the translation literal. Take an image, cut it into a grid of square patches — sixteen by sixteen pixels is the classic choice — flatten each patch, project it to a vector, add a positional embedding, and feed the whole thing to a standard encoder. A special token at the front collects a summary used for classification. No convolutions required.

What made this interesting was not the architecture but the trade-off it exposed. Convolutions bake in useful assumptions: nearby pixels matter to each other, and a cat is a cat wherever it appears in the frame. Vision Transformers have to learn those assumptions from data. At small dataset sizes, convolutional models still win. Push the data up, or add strong augmentation and distillation, or pretrain with self-supervision, and the gap closes and reverses.

Three lines of work followed from that observation:

• Hybrid and hierarchical designs. Variants such as Swin reintroduce local windows and multi-scale feature maps, which helps dense tasks like segmentation and detection where you need fine spatial detail, not just one label per image.
• Self-supervised pretraining. Masked autoencoders hide most patches and ask the model to reconstruct them; self-distillation methods train on augmented views of the same image. Both produce features that transfer well without labels.
• Vision-language coupling. Contrastive training on image-caption pairs aligns both modalities in one embedding space, which is what enables zero-shot classification and text-driven retrieval.

Audio: long sequences with no obvious words

Sound is harder than it looks. A minute of raw audio at sixteen kilohertz is roughly a million samples — far too long for attention over individual values. Teams handled this in two ways: convert audio to a time-frequency representation such as a spectrogram and treat it like an image, or learn discrete units from the waveform and treat those like tokens.

Speech recognition models such as Whisper use an encoder-decoder trained on a very large body of weakly labeled audio, with a single model handling transcription, translation and language identification as different output prompts. Pretraining approaches like wav2vec 2.0 learn from unlabeled audio by masking spans and predicting quantized units, then fine-tune on a small labeled set. The Conformer combined convolution with self-attention, keeping the local sensitivity that phonetics demands while still modeling long-range context. In music, attention made it feasible to keep a generated piece coherent across minutes rather than bars.

Biology: proteins and genomes as sequences

A protein is a string over an alphabet of twenty amino acids, and its function depends on how that string folds. Protein language models borrow the masked-language-model recipe directly: hide residues, predict them from context, and the resulting embeddings turn out to encode structural and functional properties well enough to be useful for downstream prediction tasks with little or no labeled data. Structure-prediction systems such as AlphaFold2 went further, using attention over both the sequence and the multiple sequence alignment of homologous proteins to refine residue-pair relationships.

Genomics followed a similar path, with k-mer tokenization standing in for words and models predicting regulatory activity or gene expression from DNA sequence. One caution applies more strongly here than anywhere else: a confident prediction is not a validated biological result. Benchmarks in this space are noisy, distribution shift is severe, and any clinical or diagnostic application needs wet-lab confirmation, domain experts and regulatory review.

What actually transfers between modalities

1. Tokenization is the real decision. Patch size, spectrogram hop length, k-mer length — these choices often matter more than adding layers.
2. Positional encoding must match the geometry. Two dimensions for images, time for audio, relative offsets for very long sequences.
3. Masked reconstruction is the default pretext task. It works almost everywhere because it forces the model to use context.
4. Pre-trained checkpoints beat architecture tinkering. Fine-tuning a strong checkpoint on a modest dataset usually outperforms training a clever new design from scratch.
5. Memory scales quadratically. Budget for sequence length before you budget for parameters.

Where the cracks show

Long sequences remain the weak point. Attention over an entire genome or hour-long recording is still expensive, which is why sliding windows, sparse attention and state-space alternatives keep gaining ground. Interpretability is another soft spot: attention weights look like explanations but often are not, and a heat map over image patches tells you where the model looked, not why it decided. In scientific domains, the risk is subtler — a model can memorize splits of public benchmarks and look far better than it is.

If you are starting on a non-text problem

Begin by writing down what a token is and how many you will have per example. That single number predicts your compute budget, your batch size and your debugging pain. Then build the simplest convolutional or recurrent baseline you can — it is your reality check, and on small datasets it may win. Fine-tune a pretrained checkpoint in the same modality before training anything from scratch, and verify that the pretraining data resembles your data rather than assuming it does.

Finally, evaluate on your own distribution with a metric that reflects the decision you actually need to make, not just the training loss. Transformers travel well between domains, but the habits that make them work — careful tokenization, honest baselines, respect for data limits — do not transfer automatically. They are still your job.

Photo: coyot / Pixabay