22  Images, audio, and tables: Representations and tasks

Images, audio, and table fields need representations that preserve the structure required by their distinct output tasks.

Pixels have spatial neighbors, audio samples have a time order, and table columns have different meanings and units. A representation that ignores those relationships can discard useful structure. A model first needs a numerical representation that preserves the distinctions required by its task.

Classification, matching, reconstruction, and generation require different outputs and learning signals, even when their internal vectors have the same width. Image classification returns categories. Object detection predicts categories and locations of individual objects, commonly as labeled boxes. Semantic segmentation assigns a category to each image pixel. A representation useful for one output still needs the appropriate prediction head and reference labels for another.

Six linked panels show local image filters and patch tokens, image-text matching versus generated outputs, waveforms becoming time-frequency features, table fields receiving field-specific handling, task outputs selecting evaluation criteria, and sixteen flattened image patches projected to sixteen width-64 vectors.
Figure 22.1: Local filters and projected patches preserve image structure. Matching and reconstruction require different outputs. Audio features retain time and frequency, speech models learn task-specific representations, and table fields need typed preprocessing and a measured baseline.

22.1 Local image operations and learned filters

An image classifier must relate nearby pixel values while recognizing patterns at different locations. An image channel is one measurement coordinate at each pixel, such as red, green, or blue intensity. A batch in PyTorch commonly has shape \([B,C,H_{\mathrm{img}},W_{\mathrm{img}}]\): examples, channels, height and width.

A convolution applies a local filter at many spatial locations, reusing its weights. Training a convolutional layer adjusts those weights. Deep-learning libraries usually implement cross-correlation: the filter is not spatially reversed. The expression below uses that convention with unit stride. Padding supplies any added border values.1

To focus on one image, the expression omits the batch axis and places spatial indices before the channel index. Its \(X_{i,j,c}\) corresponds to PyTorch indexing X[b, c, i, j]. For groups=1, its kernel \(W_{u,v,c_{\mathrm{in}},c_{\mathrm{out}}}\) corresponds to Conv2d.weight[c_out, c_in, u, v]. The equation and library use the same local products with different axis orders. Here \(W\) denotes kernel weights.

\[ Y_{i,j,c_{\mathrm{out}}}=b_{c_{\mathrm{out}}}+\sum_{u,v,c_{\mathrm{in}}}W_{u,v,c_{\mathrm{in}},c_{\mathrm{out}}}X_{i+u,j+v,c_{\mathrm{in}}} \tag{22.1}\]

Here \(X\) is the input, \(W\) the kernel, \(b\) an output-channel bias, and \(Y\) the output. Indices \(i,j\) locate the output. Offsets \(u,v\) range over the kernel, while \(c_{\mathrm{in}}\) and \(c_{\mathrm{out}}\) identify channels. Each output adds products across the local spatial window and all input channels.

Fixed filters show how the weights select a property before any training. A three-by-three kernel with one at its center and zeros elsewhere copies the center value. A kernel with every entry \(1/9\) averages the nine values to blur local variation. Setting the center to 5, its four immediate neighbors to \(-1\), and the corners to zero sharpens the center against those neighbors. Its weights sum to one, so a constant region stays constant away from borders. The chosen boundary policy and output range still matter. Learning replaces these fixed choices with weights adjusted for a task.

For a signed-filter calculation, use this six-by-six input and three-by-three kernel:

\(X=\begin{bmatrix}0&0&0&0&0&0\\0&1&2&2&1&0\\0&2&4&4&2&0\\0&1&3&3&1&0\\0&1&2&3&1&0\\0&0&1&1&0&0\end{bmatrix}\), \(W=\begin{bmatrix}0&1&0\\1&-4&1\\0&1&0\end{bmatrix}\).

With stride one, no added padding, and zero bias, the first window gives \(0+0-4(1)+2+2=0\). Shifting one column gives \(0+1-4(2)+2+4=-1\). The filter compares each center with its four immediate neighbors: their sum minus four times the center. In the first, nonuniform window the neighbors average to the center, giving zero. In the second, their sum is one less than four times the center, giving \(-1\).

The spatial stride is the number of input positions between successive window starts. It differs from storage stride, developed later in §23.3. For kernel width \(k\), positive stride \(s\), and \(p\) added positions on each side, one output dimension is

\[ n_{\mathrm{out}}=\left\lfloor\frac{n_{\mathrm{in}}+2p-k}{s}\right\rfloor+1 \tag{22.2}\]

Here \(n_{\mathrm{in}}\) is an input width or height. Dilation one means adjacent kernel entries read adjacent input positions. Under that assumption, a legal kernel start cannot exceed \(n_{\mathrm{in}}+2p-k\). Starts are spaced by \(s\), and counting the start at zero gives the final \(+1\). The padded dimension must accommodate the kernel. The formula counts spatial positions, not output channels, and applies separately to height and width.

Example: Equal output sizes from different convolution settings

Use \(X=\begin{bmatrix}1&0&1&0&1\\0&1&1&0&1\\1&0&1&1&0\\1&0&1&1&1\\0&1&1&0&0\end{bmatrix}\) and \(W=\begin{bmatrix}1&2&1\\0&1&0\\1&2&1\end{bmatrix}\). Compare (stride=1, padding=0) with (stride=2, padding=1), using zero padding and no bias. Their sizes are \(\lfloor(5-3)/1\rfloor+1=3\) and \(\lfloor(5+2-3)/2\rfloor+1=3\).

Stride 1 without padding gives [[5,6,5],[5,7,7],[5,7,5]]. Stride 2 with one zero border gives [[2,4,3],[4,7,5],[2,4,3]].

Conclusion: Both settings produce three-by-three arrays, but different window starts and border values give different entries.

Spatial pooling reduces a window within each channel, rather than the whole token sequence used in Chapter 13. Maximum pooling retains the largest local value. Mean pooling averages that window. Consider nonoverlapping two-by-two windows with stride two on

\(X=\begin{bmatrix}8&7&5&3\\12&9&5&7\\13&2&10&3\\9&4&5&14\end{bmatrix}\).

The upper-left maximum is 12, while its mean is \((8+7+12+9)/4=9\). Repeating the reductions gives maxima [[12,7],[13,14]] and means [[9,5],[7,8]].

Conclusion: Both spatial axes shrink from four to two, while channels remain separate. Maximum pooling preserves local peaks, and mean pooling preserves local averages. Neither retains every pixel or its within-window position.

A convolutional neural network, abbreviated CNN, composes local filters and nonlinear transformations. For classification, local filter responses pass through activations and further filters or pooling. A final aggregation produces features for a category-scoring head, and labeled images supply its training loss. This connects the local operation to the required output.

An output’s receptive field is the input region that can affect it. Two successive three-by-three convolutions with stride one cover a five-by-five region in the original image: each second-layer window combines neighboring first-layer windows. Larger context therefore develops through composition. With shared filters and stride one, shifting the input shifts the response away from boundaries. That property is translation equivariance. It does not by itself make the final category invariant to shifts, especially after subsampling or cropping.

Recognizing a pattern does not establish which part of the image supplied the evidence. Suppose wolf photographs usually contain snow while husky photographs usually contain grass. A classifier could separate those training examples using their backgrounds. Shortcut learning uses a predictive cue that works in the observed data but fails when the intended task requires a different relationship.2

Testing both animals across both backgrounds can expose this problem. Changing a background while preserving the animal provides another comparison, but artifacts from the edit can also change predictions. The result supports a background-dependence hypothesis under the controlled conditions, rather than proving what every internal feature represents.

Limited image variety can also make a classifier sensitive to an object’s position or orientation. Data augmentation creates transformed training examples with targets retained or adjusted to match the transformation. Random crops and horizontal reflections were part of AlexNet’s image-classification training.3 A crop retaining one animal can preserve its category. Cropping two cups down to one changes a counting answer, and reflecting written text can change its meaning. Boxes and segmentation masks must follow the corresponding image transformation.

Original records belong to their training, validation, or test split before their transformed copies are created. Otherwise related copies can cross the evaluation boundary. Augmented copies remain dependent on their originals. More transformations add training work, and their value needs a held-out comparison under the intended input conditions. This addresses variation in the data. Training a deep network also depends on the paths available to loss derivatives.

A ResNet is a residual network with bypasses around learned transformations. An identity bypass needs matching shapes. A projection can supply compatible spatial size or channel width when those change.4

Residual-network drawing with bypasses and elementwise additions. Shapes are unspecified and some pictured stacks change size.
Figure 22.2: Bypass paths enter additions beside learned convolutional transformations.

The residual derivative in §17.3 supplies an identity contribution when shapes match. The total gradient also depends on the learned branch.

22.2 Patch order and shared projection

Attention needs a sequence of vectors rather than a two-dimensional pixel grid. Projecting image regions supplies those positions while retaining their spatial order.

A patch token is a vector representing one image region. A vision transformer, abbreviated ViT, projects patches to a common width and applies transformer blocks with position information.5 Nonoverlapping square patches of side \(P\) give

\[ N_{\mathrm{patches}}=\left(\frac{H_{\mathrm{img}}}{P}\right)\left(\frac{W_{\mathrm{img}}}{P}\right) \tag{22.3}\]

Both image height \(H_{\mathrm{img}}\) and width \(W_{\mathrm{img}}\) must be divisible by \(P\). Otherwise preprocessing needs a declared crop, resize, or padding policy. These remove content, change sampling, or add positions, respectively.

For a single-channel patch [[1,2],[3,4]], row-wise flattening gives \((1,2,3,4)\). A projection of shape \([4,D]\) maps those four values to one width-\(D\) patch vector.

A 16-by-16 RGB patch contains \(16(16)(3)=768\) raw values. Flattening fixes their order, and a shared linear map projects them to model width \(D\). That width need not equal 768. Position information distinguishes patch locations.

A class token is an added learned input vector whose final representation can feed an image classifier. Attention lets it incorporate patch information. It is not a pixel and must be inserted if the output uses that position.

Example: Patch positions for a 224-pixel image

A 224-by-224 image with 16-by-16 patches has 14 patches per side and \(14(14)=196\) patch positions. Input shape \([B,3,224,224]\) becomes projected shape \([B,196,D]\). Adding one class token gives 197 positions.

Treating each pixel as one position would instead give \(224(224)=50{,}176\) positions. Full attention would compare \(50{,}176^2=2{,}517{,}630{,}976\) position pairs per head, versus \(196^2=38{,}416\) for the patches. This is 65,536 times as many pairs. These counts exclude a class token and count pairs rather than complete arithmetic operations: each score also depends on the query/key width, and the value combination requires further calculation.

Halving the patch side to eight gives 784 patch positions, four times as many. Ignoring an optional class token, full attention has sixteen times as many position pairs at the same width.

Conclusion: Patch size changes both sequence length and the raw width entering each projection. Prediction quality still requires evaluation on the intended task.

RGB patch contents appear above a Base, Large, and Huge model table. The learned patch projection is not shown.
Figure 22.3: Image regions become ordered flattened rows before a shared projection maps them to model width. ViT variants can change depth and width while using the same patch size.

Flattening precedes the learned projection. The published configurations below compare depth and width while using the stated patch size.6

Table 22.1: Published ViT variants increase depth, width, and parameter count together.
Variant Layers Width MLP width Heads Parameters
Base 12 768 3072 12 86 million
Large 24 1024 4096 16 307 million
Huge 32 1280 5120 16 632 million

A separate 32-by-32 RGB case makes the flattening order explicit before the projection is applied.

With patch side eight, the count formula gives \(N=(32/8)(32/8)=16\) positions. Each patch contains \(8(8)(3)=192\) raw values. A fixed channel/spatial order defines its row, and the projection must use that same order. For channel-first input, let \(c\in\{0,1,2\}\) select the channel and \(u,v\in\{0,\ldots,7\}\) the row and column inside one patch. Flattening in that order assigns index \(j=64c+8u+v\). Channel 1, row 2, column 3 therefore contributes at index 83. Projection row 83 must multiply that same pixel value. If input order changes, the corresponding rows of the projection must be permuted with it to preserve the operation.

\[ Z=\operatorname{flatten}(\mathrm{patches})W_{\mathrm{patch}},\quad Z\in\mathbb{R}^{B\times N\times D} \tag{22.4}\]

Flattened patches have shape \([B,N,192]\), \(W_{\mathrm{patch}}\) has shape \([192,64]\), and \(Z\) has shape \([B,16,64]\). The equation omits bias. A bias-enabled projection adds one width-64 vector at each position, as in the Conv2d example below.

The multiplication contracts the 192-value axis, leaving examples and patch positions separate. Adding a class token would give 17 positions. Position information still requires its configured encoding before order-dependent model use.

Conclusion: Sixteen patches of 192 values become sixteen width-64 vectors. This checks the input representation that an image transformer could receive. Classification, matching, or generation still requires the appropriate model and output head.

The runnable example uses nn.Conv2d with kernel and stride both 16. Each filter covers one nonoverlapping RGB patch. Flattening its weights gives a shared linear patch projection, with one bias per output coordinate.

Code example: Patchifying an image with a strided convolution

import torch
import torch.nn as nn

images = torch.randn(2, 3, 224, 224)
patch_projection = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=16, stride=16)

patch_grid = patch_projection(images)          # [2, 64, 14, 14]
patch_sequence = patch_grid.flatten(2).transpose(1, 2)  # [2, 196, 64]

print(patch_grid.shape)
print(patch_sequence.shape)

The printed shapes are [2,64,14,14] and [2,196,64]. flatten(2) joins spatial patch axes, and transpose(1,2) places positions before features. Random inputs and weights make values variable. This checks construction without a class token, position encoding, transformer, or trained classifier.

The local runnable check turns a four-by-four grayscale image into four patches and projects each to width three. It contains no text prompt or generation model.

Code example: Four grayscale patches projected to width three

import torch

image = torch.arange(16.0).reshape(1, 1, 4, 4)
patches = image.unfold(2, 2, 2).unfold(3, 2, 2)
patches = patches.contiguous().reshape(1, 4, 4)
projection = torch.ones(4, 3)
patch_tokens = patches @ projection

assert patches.shape == (1, 4, 4)
assert patch_tokens.shape == (1, 4, 3)
print(patch_tokens)

The patches are [0,1,4,5], [2,3,6,7], [8,9,12,13], and [10,11,14,15]. Multiplication by the all-ones [4,3] projection repeats each sum three times. The output rows are (10,10,10), (18,18,18), (42,42,42), and (50,50,50).

Conclusion: The output has shape [1,4,3] and preserves the declared patch order. It verifies input conversion, not captioning, VQA, or retrieval accuracy.

A patch sequence supplies an image representation. A task involving language still needs a connection between image and text representations.

22.3 Image-text matching, joint processing, and generation

Finding a caption that matches an image needs comparable scores across candidates. Producing a new caption also needs a generator. These are different output tasks.

CLIP, Contrastive Language-Image Pretraining, trains separate image and text encoders using paired examples. Each returns one vector per example. Dividing each nonzero vector by its Euclidean norm makes their dot product a cosine similarity. Zero vectors need a handling policy before division.7

For \(B\) matched pairs, \(S_{ij}\) compares image \(i\) with text \(j\). This \([B,B]\) matrix has intended matches on its diagonal. Temperature \(\tau>0\) divides these scores as in §7.2 before two classification losses:

\[ L_{\mathrm{CLIP}}=\frac{1}{2}\left[\operatorname{CE}\!\left(\frac{S}{\tau},y\right)+\operatorname{CE}\!\left(\frac{S^{\top}}{\tau},y\right)\right] \tag{22.5}\]

Each CE receives logits and calculates row-wise cross-entropy as in §8.1, then averages the \(B\) rows using §8.5’s included-count convention. The first direction chooses text for an image. Transposition makes each text choose an image. Labels \(y=(1,\ldots,B)\) select the diagonal, while zero-based code uses torch.arange(B).

Example: Both directions from one score matrix

Choose scaled logits \(S/\tau=\log\begin{bmatrix}0.9&0.1\\0.2&0.8\end{bmatrix}\), with an elementwise logarithm. These constructed scores are not measured encoder outputs. They are compatible with normalized vectors at a suitable temperature.

For example, choose \(\tau=0.1\) and image vectors as the first two coordinate basis vectors in three dimensions. Each text vector takes its corresponding column of \(S\) as its first two coordinates. A third coordinate completes its norm to one because their squared sum is below one.

Row normalization gives matching probabilities 0.9 and 0.8. Their mean loss is \((-\log0.9-\log0.8)/2\approx0.164252\), or 0.164 nats.

Column normalization uses that same matrix. Text 1’s matching probability is \(0.9/1.1=9/11\), and text 2’s is \(0.8/0.9=8/9\). Their mean loss is approximately 0.159227 nats. Averaging both directions gives approximately 0.161739 nats.

Conclusion: Normalizing across different candidate axes changes probabilities. Copying the image-row probabilities into the reverse direction would give the wrong symmetric loss.

Four-by-four similarity grid without axes, numerical scores, a marked matching diagonal, temperature, or directional losses.
Figure 22.4: Image and text vectors form a candidate score matrix. The paired diagonal supplies reference choices in both directions.

The objective favors paired entries relative to other batch candidates. Another caption may still describe the same image, so an off-diagonal pair is not necessarily semantically wrong. Because the encoders share parameters across examples, an update need not improve every individual pair’s distance.

Separate encoders support retrieval because stored candidates can be encoded in advance. ViLT, the Vision-and-Language Transformer, instead sends projected image patches and word embeddings obtained by §13.1’s lookup into one encoder. Position and modality information distinguish their locations and input types. Common width permits joining the sequence.8

A two-stream fusion model retains separate image and text sequences, then exchanges information through cross-attention. For example, text queries can read image keys and values, while a separate cross-attention operation lets image queries read text. This differs from a joint encoder, where one self-attention calculation operates over the combined positions. It also differs from CLIP’s independently encoded vectors, which interact only in the comparison score. The chosen task determines whether this pair-specific interaction is worth recomputing for each candidate.

Visual grounding locates the image region described by text, such as the box around “the red cup.” It needs spatial outputs and reference regions during supervised training. A high image-text matching score alone gives neither coordinates nor a verified location.

Some image-text tasks require judging a claim about the scene. Visual reasoning draws conclusions about the depicted objects and their relationships. For “There are two cups,” a true-or-false output must agree with the visible count. Supervised training needs reference judgments for the claims. This differs from choosing a similar caption or generating a description: a plausible description can still get that count wrong.

ViLT trains image-text matching and selected masked-word prediction. A matching head judges the pair, while a masked-word head predicts an original word using joint context. For example, if the input hides office, the masked-word head assigns scores to possible replacements.9

Supporting objective: Word-patch alignment

A pair-level matching label does not specify which words correspond to image patches. ViLT adds an alignment term using the contextualized word and patch vectors. An optimal-transport comparison assigns nonnegative weights to word-patch pairs while constraining the total weight associated with each side. It seeks a low weighted matching cost. The paper approximates this distance with the IPOT algorithm and adds it to the image-text matching loss with weight 0.1. These weights describe the alignment objective, rather than the attention operation.10

For captioning or open-ended visual question answering, abbreviated VQA, a generation component produces text. A language decoder can read encoded image features through cross-attention. Another arrangement projects visual features into its input sequence beside the prompt. It then generates tokens using image information and earlier output. A matching encoder alone does not implement this loop.

22.4 Reconstructed pixels and discrete image codes

A coarse image does not uniquely determine its missing fine detail. Super-resolution estimates a higher-resolution image from a lower-resolution observation. A larger grid alone does not identify the missing detail. If one coarse grayscale pixel is the average of four fine pixels, both (0.25,0.25,0.25,0.25) and (0,0,0.5,0.5) produce 0.25. Either fine pattern is compatible with the same coarse value.

A supervised method learns from aligned low- and high-resolution pairs. SRCNN first enlarges the input using bicubic interpolation, which estimates added pixel values from neighboring ones. Learned convolutions then predict the reconstruction. Training minimizes squared pixel error on corresponding central regions because the original unpadded convolutions reduce the output border.11 The blur and downsampling rule used to make training inputs is part of the task. A model trained on one rule may perform poorly on another.

Doubling both grid dimensions produces four times as many pixels. Compute cost also depends on the model. Plausible fine detail is an estimate from learned regularities, rather than additional measurement of the scene. Reconstruction error against a high-resolution reference and the intended use determine whether that estimate is useful.

Discrete image codes serve another purpose: reconstruction and later modeling of image content. A codebook is a finite table of fixed or learned reconstruction vectors. A VQ-VAE, vector-quantized variational autoencoder, maps an encoder vector to its nearest codebook vector, then decodes from that selection. Training adjusts this VQ-VAE’s codebook entries. The integer code identifies the selected vector but is not the vector itself.12

For a chosen encoder output \((0.8,0.1)\) and codebook vectors \((0,0)\), \((1,0)\), and \((0,1)\), squared distances are 0.65, 0.05, and 1.45. Selection returns the second vector, \((1,0)\). The decoder receives it, losing the original difference \((-0.2,0.1)\) from that entry.

Hard selection has no useful ordinary encoder derivative. A straight-through approximation passes the decoder gradient to the encoder as though selection were an identity map. Reconstruction trains the decoder and, through that approximation, the encoder. A codebook term moves the selected vector toward a fixed encoder output. A commitment term moves the encoder output toward the fixed selected vector. Stopping gradients on the opposite side separates these roles.13

Conclusion: Reconstruction needs a reference image or a defined decoding path, rather than a matching score. The VQ-VAE trades continuous encoder values for selected codebook entries. Audio adds a time axis with its own conversion requirements.

22.5 Sampled sound and spectral features

A recording can contain the same frequencies at different times, producing different words. One spectrum for the entire recording would discard when those components occur. Short overlapping windows retain a time index alongside local frequency content.

A waveform is a sequence of sampled amplitudes. Its sampling rate \(f_s\) counts samples per second in each channel. Sample \(n\) occurs at time \(n/f_s\) under a fixed-rate convention. Amplitude units or normalization depend on the recording system.

Sampling also limits which frequencies can be distinguished. Without suitable filtering before sampling, components above half the sampling rate can appear as lower frequencies, a distortion called aliasing. At 16,000 samples per second, preserving arbitrary content above 8,000 Hz requires a different sampling setup. A later spectral transform cannot recover distinctions already lost in sampling.

PCM, pulse-code modulation, stores quantized amplitudes at those sample times. Bit depth determines the available codes. Stereo has two channels, with one sample from each in an audio frame. That frame differs from the longer analysis window below.

PCM page with 44.1 kHz, two channels, and 16 bits per channel sample. Its wording calls a stereo frame one sample.
Figure 22.5: The page identifies sampling rate, channels, and amplitude storage.

At 44,100 sample times per second, two channels, and 16 bits per channel sample, one second uses \(44{,}100(2)(16/8)=176{,}400\) payload bytes. Headers and compression are excluded. Calling one stereo frame a single per-channel sample would lose the factor of two.

An analysis window selects and weights a local run of samples. Its hop size is the number of samples between successive starts. A rectangular window weights each included sample by one. A tapered window reduces discontinuities at the window edges, while changing the separation of nearby frequencies.

A short-time Fourier transform, abbreviated STFT, calculates a frequency transform for each weighted window. For transform length \(N\) equal to window length and hop \(H_{\mathrm{hop}}\), write its real and imaginary coordinates as \(a_{m,k}\) and \(b_{m,k}\):

\[ X[m,k]=a_{m,k}+i b_{m,k}=\sum_{n=0}^{N-1}x[mH_{\mathrm{hop}}+n]w[n]\exp\!\left(-\frac{2\pi i kn}{N}\right) \tag{22.6}\]

Frame index \(m\) selects the window start, \(n\) indexes its samples, and \(w[n]\) supplies the weights. Bin \(k\) compares that window with a frequency pattern using both sine and cosine components. The real-sum expansion below is equivalent notation, not another measurement.

Supporting calculation: Real coordinates of one spectral coefficient

\[ \begin{aligned}a_{m,k}&=\sum_{n=0}^{N-1}x[mH_{\mathrm{hop}}+n]w[n]\cos\!\left(\frac{2\pi kn}{N}\right),\\ b_{m,k}&=-\sum_{n=0}^{N-1}x[mH_{\mathrm{hop}}+n]w[n]\sin\!\left(\frac{2\pi kn}{N}\right).\end{aligned} \tag{22.7}\]

Here \(X[m,k]=a_{m,k}+i b_{m,k}\). The cosine sum supplies its real part and the negative sine sum its imaginary part, using the same window samples \(x[mH_{\mathrm{hop}}+n]\).

Bin \(k\) represents frequency \(k f_s/N\) before accounting for equivalent negative-frequency bins. For real part \(a\) and imaginary part \(b\), magnitude \(\sqrt{a^2+b^2}\) measures response strength. Phase is the angle of the pair \((a,b)\), describing its sine-cosine alignment. Power is squared magnitude, a different quantity. For real input, positive and negative frequency coefficients form conjugate pairs.

Example: Zero mean does not mean silence

Use \((1,0,-1,0)\), a rectangular window, \(N=4\), and no centering or padding. At \(k=0\), all exponential weights are one. The constant, or DC, coefficient is \(1+0-1+0=0\).

At \(k=1\), the weights are \((1,-i,-1,i)\). The contributions \((1,0,1,0)\) sum to 2. The full transform is \((0,2,0,2)\), with the last entry the conjugate-frequency counterpart of the second.

For samples \((1,0,-1,0,1,0)\) and hop two, the second window is \((-1,0,1,0)\). Its \(k=1\) coefficient is \(-2\). Both frames have magnitude 2 at that bin. Their coefficients 2 and \(-2\) differ by a half-cycle phase angle.

Conclusion: Zero DC indicates cancellation of the constant component. The nonzero frequency response retains the oscillation. Moving the frame changes the samples and can change phase even when magnitude stays equal.

Counting only complete windows without centering or padding gives

\[ \mathrm{frames}=\left\lfloor\frac{\mathrm{samples}-\mathrm{window}}{\mathrm{hop}}\right\rfloor+1 \tag{22.8}\]

Sample count must be at least the positive window length, and hop must be positive. Six samples, window four, and hop two give \(\lfloor(6-4)/2\rfloor+1=2\) frames. Different padding, centering, or transform-length policies need their own count. The full manual transform can be stored as two time frames by four frequency bins. For unbatched real input and a real window, torch.stft with return_complex=True returns frequency before frame. With the default onesided=None, it uses one-sided treatment for this real input and keeps \(\lfloor N/2\rfloor+1\) nonredundant bins. With n_fft=4, hop_length=2, and center=False, this six-sample example therefore returns shape [3,2]. Setting onesided=False retains all four bins and returns [4,2].14

A fast Fourier transform, abbreviated FFT, is an algorithm for calculating the discrete Fourier transform efficiently. It does not itself choose or slide windows. STFT applies a transform separately to successive windows. Recovering a waveform from a full transform needs phase as well as magnitude and compatible inverse/window rules. Magnitude alone generally cannot supply exact reconstruction.

A mel spectrogram groups spectral energy into overlapping bands spaced on a perceptual frequency scale. The filterbank forms weighted sums of magnitude or power under a declared convention. A logarithm compresses their range after a floor prevents taking the logarithm of zero. These log-mel features are not raw amplitudes. Window, hop, filterbank, power convention, and log scaling must match the trained model.

22.6 Speech representations and audio outputs

Preparing transcripts for a large recording collection is costly. Speech representations can first be learned from recordings without transcripts, then used in a supervised recognition task. The original wav2vec uses convolutional networks on raw waveform samples. Its training objective distinguishes future learned features from distractors, and a speech recognizer consumes the resulting representations.15

wav2vec 2.0 also starts with a learned waveform feature encoder, but uses a different pretraining task. It masks spans of feature vectors before a transformer processes their context. At masked positions, the output must distinguish the corresponding quantized, unmasked feature target from distractors. A diversity term encourages use of the available codes. The targets come from the audio itself, rather than supplied transcripts.16

For wav2vec 2.0, transcription adds a symbol-scoring head and supervised training with transcripts. The original wav2vec 2.0 procedure keeps the convolutional feature encoder frozen during this stage and fits the contextual network and new head. Transcription quality must be measured for the relevant language and recording conditions.17

The transcript usually has fewer symbols than the audio has feature positions. Connectionist temporal classification, or CTC, trains from a transcript without requiring a reference symbol at every position. Each position returns probabilities for the symbols and a blank. Given the full audio input, CTC treats the position outputs as conditionally independent, so a path probability multiplies their symbol probabilities across positions. The transcript probability adds the paths that produce that transcript. Training minimizes the negative logarithm of the reference transcript probability.18

For one path (a,a,blank,a,b,b), merging consecutive repeats gives (a,blank,a,b). Removing blanks then gives aab. The blank preserves the second a as a separate occurrence. For two positions each assigning blank probability 0.6 and a probability 0.4, the most probable path is (blank,blank). Its probability is 0.36, and its transcript is empty. The paths (blank,a), (a,blank), and (a,a) all give a. Their probabilities sum to \(0.24+0.24+0.16=0.64\). Choosing the best path therefore misses the more probable transcript. This separates alignment during training from a particular decoding approximation.

These waveform encoders produce learned representations for recognition, rather than requiring an FFT for each window or generating sound. A spectrogram-based encoder-decoder takes a different route.

Whisper is an encoder-decoder speech model mapping log-mel features to text tokens. Its encoder receives the features through convolutional processing. Its causal decoder uses cross-attention to encoded audio and generates text with task, language, and optional timestamp markers.19

Whisper source diagram with audio features, convolutions, encoder, decoder cross-attention, and shifted task and text token rows.
Figure 22.6: Log-mel features feed an encoder whose outputs condition a text decoder.

Whisper uses the source-conditioned encoder/decoder roles from §18.2. Task and language tokens identify the requested behavior. Timestamp tokens describe time locations. Recorded transcripts supply training targets, while generated tokens supply later decoder inputs at inference.

A transcript can be compared with reference words using word error rate, or WER. After a declared word normalization and alignment, count substitutions \(S\), deletions \(D\), and insertions \(I\), then divide \(S+D+I\) by the reference word count \(N>0\). One substitution and one deletion against ten reference words give WER 0.2. Extra insertions can make this ratio exceed one. It measures transcription edits, not every aspect of speech understanding. Quality depends on language, recording conditions, task, and training data. A result for one language cannot establish equal quality for all languages.

A reference-conditioned speech generator has a different input-output contract. Reference audio supplies an encoded description of the voice or speaking style, while supplied text specifies what to say. A generation component combines that conditioning with the text representation and produces an acoustic representation or waveform. A waveform decoder is needed when the intermediate output is acoustic features or codes. This process generates sound rather than a transcript, so its evaluation includes intelligibility and voice or style matching, not just transcription error.

Other systems use different representations and outputs. SpeechVerse combines continuous audio features with text instructions and returns text. Its frozen audio encoder feeds trainable one-dimensional downsampling, which shortens the sequence and matches text-embedding width. The resulting vectors join text embeddings before a language model with an adapter.20

Its first stage fits the audio connection and layer-combination weights on transcription. A later stage adds LoRA for broader tasks while pretrained audio and language weights remain frozen. A keyword-extraction task returns a list of words. Generating a waveform requires a different output path. The two training stages do not update identical parameter sets.21

AudioLM models tokenized audio and can decode a waveform. Its w2v-BERT route discretizes learned features into semantic tokens. Its SoundStream codec route supplies acoustic tokens at several quantization levels. These emphasize longer content structure and signal detail without guaranteeing a perfect separation.22

Generation predicts semantic tokens, then coarse acoustic tokens conditioned on them. Fine acoustic prediction adds detail, and codec decoding reconstructs a waveform. Each conditional stage reuses autoregressive prediction ideas. The semantic and acoustic token streams identify the intermediate representations. The staged generation procedure and listening evaluation establish how the system uses them.23

Conclusion: Amplitudes, spectral features, continuous learned vectors, and discrete audio codes are different objects. Their consumers and target outputs determine the required conversion. Speech models do not all require the same spectrogram front end.

22.7 Heterogeneous fields and tabular baselines

A table row can combine a category, a measured quantity, and a missing value. Its column positions identify fields, rather than neighboring times or image regions. Treating column order as a natural sequence would add an assumption the data does not supply.

Consider three fictional rows: Leia/female/150/49, Luke/male/172/77, and Han/male/180/80. Height is in centimeters and weight in kilograms. The first field identifies a row, while the second is categorical. No target is supplied, so these records alone define neither classification nor regression. An identifier is not automatically a useful prediction feature.

Categories can select embedding rows. Numerical fields need a declared treatment of scale. Missing values need a policy, such as imputation with a missingness indicator. Fitted statistics and mappings must use training data only, and unseen categories need a fallback. The task and available evidence govern whether a sensitive field belongs in the model.

TabTransformer contextualizes categorical embeddings with attention, then joins them with continuous features for an MLP. In the original architecture, normalized continuous values join after attention. They do not all enter it as tokens.24

With \(m\) categorical fields of embedding width \(d\) and \(c\) continuous values, the final width is \(md+c\). For a separate six-field record, consider city=London, job=engineer, plan=basic, age=35, income=50000 and visits=3. The first three fields are categorical and the last three numerical. At embedding width four, the concatenated width is \(3(4)+3=15\). Field identity remains associated with each category embedding. Other tabular transformers may tokenize numerical fields, which is a different design.

A regularized linear model provides an inspectable baseline after compatible encoding. A decision tree instead splits records through successive feature tests and predicts at a leaf. Its thresholds can model nonlinear interactions without treating fields as sequence neighbors.

Gradient-boosted decision trees add trees sequentially to improve predictions under a chosen loss. At round \(m\), let the current function be \(F_{m-1}\). Each record supplies the negative loss derivative with respect to its prediction. A new tree \(h_m\) fits those values, and \(F_m=F_{m-1}+\eta h_m\) adds a scaled correction.25

For half-squared error \(L=(y-F)^2/2\), that negative derivative is the residual \(y-F\). Targets \((3,1)\) and current predictions \((2,2)\) give residuals \((1,-1)\). If a chosen tree fits them exactly, rate 0.1 gives predictions \((2.1,1.9)\). The summed half-squared loss falls from 1 to 0.81.

Conclusion: Residual fitting describes this squared-error case. Other losses supply different negative derivatives. The chosen correction demonstrates an update, not a trained benchmark result or a guaranteed held-out improvement.

Compare linear, tree, and attention models using the same split, target, metric, preprocessing boundary, and search budget. Record fit time, inference cost, and missing/noisy-field behavior alongside quality. Interactions or pretraining can motivate an attention candidate, but a ranking from one dataset and setting cannot establish superiority across other tasks.

22.8 Choosing representations by the required output

Input structure determines which information a representation can preserve. The task determines whether that information helps. Image, audio, and table models therefore need comparisons on common criteria without requiring one architecture for every modality.

The spatial outputs introduced at the chapter opening need matching reference annotations. Object detection commonly uses a category and bounding box for each object, while semantic segmentation uses a reference category at each pixel. A single category score or pooled image vector cannot by itself supply those locations. Text-to-image generation reverses the output requirement: supplied text conditions a generated image. Its image-generation mechanism is separate from the classification and text-generation paths developed here.

A representation alone does not supply the task output. Transcription needs text, while audio continuation needs a waveform path. Table prediction needs a chosen target and training-only preprocessing. Their costs and held-out results can then be compared under the same task criteria.

Separate image-text encoders can reuse stored candidate vectors. Joint processing recomputes interactions for each pair. Captioning and VQA additionally need text generation. Output requirements, available pretraining, training data, and measured resource limits determine which comparison is useful.

Logical axes specify those computations. Chapter 23 follows how the axes refer to stored values, including shared storage, copies and device placement.

Chapter checkpoint

Does zero DC in \((1,0,-1,0)\) mean every Fourier coefficient is zero? Does a CLIP matching score generate a caption? Do all original TabTransformer fields enter attention?

Answer: No. The transform is \((0,2,0,2)\), so the oscillation remains. Matching ranks pairs, while captioning needs a generation component. Original TabTransformer attention processes categorical embeddings, then joins their outputs with continuous values before an MLP.

Why can two convolution settings return the same three-by-three size yet different values?

Answer: The size formula counts legal window starts. Stride and padding can change the windows despite equal counts. The kernel products must be checked at their actual locations.


  1. PyTorch Contributors. (2026). Conv2d. PyTorch 2.12 documentation. The operation uses cross-correlation without spatial kernel reversal.↩︎

  2. Geirhos, R., et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2, 665–673. The animal/background example here is an assumed situation, not a measured result from this paper.↩︎

  3. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25. Section 4.1 uses random crops and horizontal reflections for image classification. It does not establish that every transformation preserves every task target.↩︎

  4. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778. The original blocks use identity shortcuts when shapes match and learned projections in selected shape-changing paths.↩︎

  5. Dosovitskiy, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, §3 and Table 1. ↩︎

  6. Dosovitskiy, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, §3 and Table 1. ↩︎

  7. Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. Proceedings of Machine Learning Research, 139, 8748–8763, Figure 3. The score matrix here is a constructed calculation.↩︎

  8. Kim, W., Son, B., & Kim, I. (2021). ViLT: Vision-and-language transformer without convolution or region supervision. Proceedings of Machine Learning Research, 139, 5583–5594, §3 and Figure 3.↩︎

  9. Kim, W., Son, B., & Kim, I. (2021). ViLT: Vision-and-language transformer without convolution or region supervision. Proceedings of Machine Learning Research, 139, 5583–5594, §3 and Figure 3.↩︎

  10. Kim, W., Son, B., & Kim, I. (2021). ViLT: Vision-and-language transformer without convolution or region supervision. Proceedings of Machine Learning Research, 139, 5583–5594, §3 and Figure 3.↩︎

  11. Dong, C., Loy, C. C., He, K., & Tang, X. (2015). Image super-resolution using deep convolutional networks. arXiv:1501.00092, version 3. Sections 3.1 and 3.3 describe bicubic enlargement, learned convolutions, paired targets and the central comparison region.↩︎

  12. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, §§3.1–3.2. The distance example uses chosen vectors.↩︎

  13. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, §§3.1–3.2. The distance example uses chosen vectors.↩︎

  14. PyTorch Contributors. (2026). STFT. PyTorch 2.12 documentation. The book’s frame count assumes complete windows without centering or padding.↩︎

  15. Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019). wav2vec: Unsupervised pre-training for speech recognition. Proceedings of Interspeech 2019, 3465–3469. DOI: 10.21437/Interspeech.2019-1873. Sections 2.1–2.2 learn raw-audio features with a contrastive future-feature objective for a separate recognizer.↩︎

  16. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460. Sections 2–3 distinguish masked context inputs from unmasked quantized targets. Section 3.3 describes the CTC head and frozen feature encoder during fine-tuning.↩︎

  17. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460. Sections 2–3 distinguish masked context inputs from unmasked quantized targets. Section 3.3 describes the CTC head and frozen feature encoder during fine-tuning.↩︎

  18. Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, 369–376. Sections 3.1–3.2 distinguish the sum over transcript alignments from best-path decoding.↩︎

  19. Radford, A., et al. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of Machine Learning Research, 202, 28492–28518, §§2.2–2.3 and Figure 1.↩︎

  20. Das, N., et al. (2024). SpeechVerse: A large-scale generalizable audio language model. arXiv:2405.08295, version 2, §§2.1–2.3 and Figure 2.↩︎

  21. Das, N., et al. (2024). SpeechVerse: A large-scale generalizable audio language model. arXiv:2405.08295, version 2, §§2.1–2.3 and Figure 2.↩︎

  22. Borsos, Z., et al. (2023). AudioLM: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 2523–2533, §III.↩︎

  23. Borsos, Z., et al. (2023). AudioLM: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 2523–2533, §III.↩︎

  24. Huang, X., Khetan, A., Cvitkovic, M., & Karnin, Z. (2020). TabTransformer: Tabular data modeling using contextual embeddings. arXiv:2012.06678, §2.↩︎

  25. Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5), 1189–1232.↩︎