23 Tensor storage: Shapes, views, devices, and lifetime
Tensor memory and execution claims need storage, layout, numerical type, device, and state lifetime beyond logical shape.
Two tensors can describe the same mathematical array while sharing storage differently or requiring different copies. Shape alone cannot reveal that difference. Layout, numerical type, device, and retained gradient state determine which values are stored and which operations can execute.
23.1 Logical elements and underlying storage
A repeated row can appear twice in an array without its values being stored twice. Conversely, a small slice can keep a much larger allocation alive. Memory accounting therefore needs both the logical shape and the underlying storage.
The Roadmap’s tensor primer introduced shape, dtype and device. Here their storage roles matter: shape counts logical positions, dtype sets their element size, and device identifies where the values reside and operations execute.
For axis lengths \(d_i\), the logical element count and dense payload are:
\[ \operatorname{numel}(X)=\prod_i d_i \tag{23.1}\]
\[ \mathrm{logical\ payload}(X)=\operatorname{numel}(X)\cdot\mathrm{bytes\ per\ element} \tag{23.2}\]
A scalar tensor has shape [] and one element. An axis of length zero makes the count zero. The element size in bytes is fixed by dtype. The payload equals value storage for a separate dense allocation with no extra capacity. A view is a tensor that shares another tensor’s storage. It can refer to fewer distinct locations or keep a larger allocation alive, and the allocator can reserve additional memory.
A contiguous tensor has the layout required by the specified memory format. In the ordinary row-major format, the last axis changes fastest. This describes the order of stored values, not whether two tensors share an allocation.
Example: Independent values and expanded rows
A separately allocated contiguous tensor with shape \([2,3,4]\) has \(2(3)(4)=24\) values. With float32, its value payload is \(24(4)=96\) bytes, excluding overhead.
In a separate example, three float32 values occupy 12 bytes. Expanding their row from [1,3] to [2,3] can produce six logical entries sharing those same three values. Its logical payload is 24 bytes, but its underlying value storage is still 12 bytes.
Conclusion: Logical count and distinct stored values answer different questions. Summing the logical sizes of several views can count the same allocation repeatedly.
The runnable example explicitly chooses float32 for a random [4,8,16] tensor and leaves it on the default CPU device.
Code example: PyTorch tensors carry dimensions, dtype, and device metadata
import torch
x = torch.randn(4, 8, 16, dtype=torch.float32)
print(x.shape) # batch=4, sequence=8, hidden=16
print(x.dtype) # float32
print(x.device) # cpu unless moved to cudaIt prints shape [4,8,16], dtype torch.float32, and device cpu. Calling .size() returns the same axis lengths as .shape; .dim() returns their count, here three. There are 512 logical elements, or 2048 payload bytes for this independent allocation. No gradient tracking is requested. The random values do not affect these counts. Section 23.3 adds the layout information needed to identify sharing.
A dtype such as torch.float32 describes tensor storage. It is not a Python class for constructing individual numeric objects. A zero-dimensional indexed tensor still has tensor metadata. Its .item() method returns the corresponding Python scalar, which can incur synchronization when the value resides on a device. torch.get_default_dtype() reports the floating dtype used by constructors that rely on that default. An explicit dtype or one inherited from a NumPy array can differ from it.
The axes also determine whether an operation combines the intended values. A dimensionally legal broadcast can still compare the wrong examples.
23.2 Matrix contraction and trailing-axis broadcasting
Applying one bias vector to many rows should reuse its class coordinates without mixing examples. Matrix multiplication and broadcasting provide different rules for that batched computation.
The affine map \(Y=XW+b\) from §4.3 contracts the shared input-feature axis. With shapes \([B,D_{\mathrm{in}}]\) and \([D_{\mathrm{in}},D_{\mathrm{out}}]\), the product has shape \([B,D_{\mathrm{out}}]\). A two-by-three matrix times a three-by-four matrix therefore gives a two-by-four matrix.
Section 5.3 establishes the orientation convention: mathematical \(W\) has shape \([D_{\mathrm{in}},D_{\mathrm{out}}]\), while nn.Linear stores its transpose with shape \([D_{\mathrm{out}},D_{\mathrm{in}}]\). These are two conventions for the same map, not an instruction to transpose an already compatible mathematical matrix.
Broadcasting, introduced for biases in §4.3, aligns axes from the right. Two aligned lengths are compatible when they agree or one equals one. A missing leading axis acts as length one. A size-one axis takes the other length, while equal lengths stay unchanged. A length-one input reuses its value along that result axis.1
For a score array [[1,2,3],[4,5,6]] and bias [10,20,30], shapes [2,3] and [3] align as [2,3] and [1,3]. Their sum is [[11,22,33],[14,25,36]]. A bias of shape [2] would fail because the trailing lengths three and two disagree.
A column shape [2,1] is also compatible with [2,3]. It adds one value per row across three columns. This is a different meaning from adding a length-three feature bias, despite both results having shape [2,3].
Example: A legal broadcast comparing the wrong pairs
Predictions [[0.8],[0.2]] have shape [2,1], while labels [1,0] have shape [2]. Subtraction aligns them as [2,1] and [1,2], producing shape [2,2]:
[[-0.2,0.8],[-0.8,0.2]].
Each prediction has been compared with both labels. Pairwise errors would be [-0.2,0.2], using two matching [2] arrays, or the equivalent two [2,1] arrays.
Conclusion: Broadcasting succeeded but changed the comparison from two aligned pairs to four cross-pairs. Matching the target and prediction contracts prevents a silent objective change.
An expanded view can reuse storage. An arithmetic result such as the sum above normally needs its own values. Vectorization can remove Python loops while still allocating a large intermediate, so it does not make every expression cheaper.
The axes must retain their meanings after reshaping. In the equal-width attention setup, heads times head width equals model width. §17.2 explains why splitting features and permuting axes are separate operations.
23.3 Strides, offsets, aliases, and copies
A transpose can change which values are neighbors in logical reading order without moving them in memory. A view shares storage while describing its own shape, strides, and starting offset. A copy has separately allocated storage containing copied values. Shared-storage tensors are aliases, so changing one can change what another reads.
A storage stride is the number of storage elements traversed when one logical index increases by one. The storage offset is the index of the tensor’s first logical element relative to the storage start. For two axes,
\[ \operatorname{offset}(i,j)=o+i s_0+j s_1 \tag{23.3}\]
Here \(o\) is the storage offset, and \(s_0,s_1\) are strides. The offset and strides count elements. Multiplying the resulting offset by bytes per element gives a byte displacement from the storage start.
With strides [3,1], storage offset zero, and coordinate (1,2), the element offset is \(0+1(3)+2(1)=5\). For float32, this is 20 bytes from the storage start.
Now let base contain 0 through 11 in shape [4,3]. The slice base[1:3] has shape [2,3], strides [3,1], and storage offset 3. Its coordinate (1,2) reaches element \(3+3+2=8\), or 32 bytes from the underlying start. Adding only the strides would incorrectly select element 5.
For an independent x=[[0,1,2],[3,4,5]], transposition gives shape [3,2] and strides [1,3]. Its coordinate (2,1) still reaches element 5. Assigning x.T[0,1]=30 changes x[1,0] to 30 because both refer to the same location.
An expanded axis can instead have stride zero: increasing its logical index reads the same storage element. The repeated row in §23.1 uses this layout to share three stored values across six logical entries.
view requires a new shape compatible with the existing strides. It need not require every input to be contiguous. reshape and flatten may return a view or allocate a copy, depending on layout. Their names alone do not guarantee sharing.2
Example: Two reshapes of one stepped slice
Let a contain the integers zero through 23 in shape [3,8], with strides [8,1]. Its 24 values also fit a.view(4, -1): the inferred axis has length six, and the view shares storage.
Now take b = a[::2, ::2]. It selects rows zero and two and every other column, giving [[0,2,4,6],[16,18,20,22]]. Its shape is [2,4] and its strides are [16,2]. It is noncontiguous in the ordinary row-major format.
b.view(2, 2, 2) succeeds with strides [16,4,2]: it splits the existing length-four axis without joining the two rows. The values and storage are unchanged. In contrast, b.view(4, 2) fails. Moving from the last selected value in the first row, 6, to the first selected value in the second row, 16, breaks the regular spacing needed to join those axes. b.reshape(4, 2) copies the values into [[0,2],[4,6],[16,18],[20,22]], with strides [2,1] and separate storage.3
Compare the tensors’ untyped_storage().data_ptr() values to check storage sharing in this example. The compatible views match a’s storage pointer. The copied reshape differs while all tensors remain alive.
Conclusion: Noncontiguity alone does not decide whether a view is legal. The requested shape must preserve the spacing described by the original strides.
contiguous() returns the existing tensor when it already satisfies the requested memory format from §23.1. Otherwise it copies into compatible storage. Copying x.T therefore separates future mutations, at the cost of allocation and copying. Contiguity alone does not prove a later kernel runs faster.
detach() removes the tensor’s connection to its earlier autograd history while sharing storage. It is not an independent data copy. Changing the detached values also changes the original storage, and can invalidate saved values needed for backward. detach().clone() creates independent values when both separation of history and storage are required.4
Conversion to or from NumPy can preserve storage sharing or introduce a copy. The result depends on the operation, device, numerical type, layout and gradient state.5
| Operation and required input | Storage and numerical type | Gradient or transfer condition |
|---|---|---|
torch.from_numpy(array), for a compatible CPU array |
Shares the array’s storage and preserves its dtype | No dtype conversion is implied |
torch.tensor(array, dtype=torch.float32) |
Creates separate float32 storage | The constructor copies the supplied values |
x.numpy(), for a supported CPU dtype and layout |
Can share the tensor’s storage | With gradient recording enabled, x must not require gradients |
x.detach().numpy(), for such a CPU tensor |
Can still share x’s values | Detaching removes the earlier gradient history; array mutation can change x |
x.detach().cpu().numpy(), for a CUDA tensor |
The CPU transfer creates host storage | Array mutation does not modify the original CUDA allocation |
A typical random NumPy array uses float64. Sharing that storage does not convert it to float32. Detaching changes gradient history, while a device transfer or an explicit copying constructor changes storage.
An in-place operation, commonly marked by a trailing underscore such as add_, changes existing values. Its out-of-place counterpart returns a separate result. Boolean comparisons such as a > 10 produce Boolean values. An in-place comparison such as a.gt_(10) instead writes zero or one into the original tensor’s dtype. Aliases observe those changed values, so the distinction affects both correctness and storage.
These operations return tensors, so calls can be chained. With independent integer tensors a = torch.tensor([1, 2]) and b = torch.tensor([3, 4]), a.add_(b).mul_(2) returns a itself and changes its values to [8,12]. Resetting a to [1,2] and calculating (a + b).mul(2) instead returns [8,12] while a stays [1,2]. Equal final values do not establish equal mutation behavior. This example uses tensors without gradient tracking.
A call to .to(device) can return the original tensor if device and dtype already match. Changing device requires a transfer. The nonscalar input and weight operands of the matrix operation in §23.5 must have compatible placement. Checking only the input’s device leaves the weight’s location unresolved.
23.4 Tensor roles and state lifetime
A training step can keep several arrays with the same shape alive for different reasons. Releasing one does not release all the others. Their roles determine when they are created, retained, and reused.
Parameters persist across steps, including frozen parameters whose values remain unchanged during updates. An activation is an intermediate value produced by the forward calculation. Autograd retains the values required by its backward rules while the graph needs them. A vocabulary embedding table is a parameter, while its looked-up vectors are activations. A torch.nn.Module registers parameters and child layers so their state can be discovered, moved, saved, and updated together. Defining a module does not fit its parameters.
A gradient buffer accumulates a parameter’s derivatives in the same shape and orientation, as §5.3 specifies. It can be absent before backward or when the parameter has no contributing path. Non-leaf derivatives can propagate without a retained .grad, as §11.1 explains.
The optimizer state introduced with Adam contains persistent information used by an update rule. The following relation gives the dimensions of its first and second moment arrays for each participating parameter:
\[ \operatorname{moments}_{\mathrm{Adam}}(\theta)=\{m_{\theta},v_{\theta}\},\quad\operatorname{shape}(m_{\theta})=\operatorname{shape}(v_{\theta})=\operatorname{shape}(\theta) \tag{23.4}\]
Here \(m_\theta\) and \(v_\theta\) have the parameter’s shape. Adam also keeps an update counter for bias correction. The arrays’ numerical type is part of the memory calculation. These arrays belong to the optimizer’s state mapping, not to every tensor’s metadata.
optimizer.step() changes parameters but does not automatically clear .grad. Resetting gradients leaves Adam moments available for later steps. An ordinary backward can release saved graph data that it no longer needs. Keeping references or retaining the graph can extend those lifetimes.
Parameter, activation, gradient, optimizer and serving-state lifetimes can overlap. The embedding table is part of parameter storage. Counting it again would duplicate the same values. A tokenizer’s vocabulary can remain outside GPU memory.
Example: Four arrays for one million parameters
One million float32 parameter values use 4,000,000 bytes, or 4 decimal MB. A same-type gradient adds 4 MB. Two same-type Adam moments add 8 MB, giving 16 MB for these four arrays.
This is 16,000,000 bytes, about 0.01490 GiB, using \(2^{30}\) bytes per GiB. Saved activations, other optimizer scalars, optional master parameter copies, temporary buffers, and allocation overhead are excluded.
Conclusion: Two moment arrays alone occupy twice the parameter payload under these assumptions. The 16 MB sum is not the complete training peak.
Calling model.eval() changes behavior of layers such as dropout. It does not disable autograd, erase existing gradients, or free an optimizer. With trainable inputs or parameters, a forward call in evaluation mode can still record a graph. torch.no_grad() disables recording within its scope. Inference mode additionally removes tracking overhead under stricter reuse constraints.6
For example, a linear layer in evaluation mode still gives an output with requires_grad=True when its parameters require gradients. Running that call inside torch.no_grad() gives an output without that recorded graph. The weight values and any existing optimizer state remain separate from this recording choice.
Ordinary inference needs parameters and temporary forward values. Autoregressive serving also retains the KV state from Chapter 21 for the request or a compatible cached prefix. Training gradients and moments can be absent in a dedicated inference process. They do not disappear merely because a training process switches to evaluation mode.
23.5 Compatible devices and optional distributed execution
A CPU model cannot multiply its weights with a nonscalar batch moved alone to a GPU. Both operands need compatible placement. A CUDA device is a supported GPU memory and execution target selected by PyTorch, such as cuda:0.
The runnable example selects CUDA when available and otherwise CPU. It moves both the layer and its input to that device, then checks the output placement.
Code example: Moving a model and its batch to one device
import torch
import torch.nn as nn
# The same code runs on a GPU when CUDA is available and on CPU otherwise.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nn.Linear(3, 2).to(device)
features = torch.tensor([[1.0, 0.0, -1.0]], device=device)
logits = model(features)
assert logits.device == features.device == next(model.parameters()).device
print(device, logits.shape)The output shape is [1,2]. Random layer weights make its values variable. A CPU fallback verifies this placement logic. It is not a GPU execution or performance result. Even on a GPU, a successful device assertion supplies no timing evidence.
For a supported operation, PyTorch selects an implementation according to device, dtype and properties such as layout. This selection is called dispatch. A kernel is the implementation that executes the operation on that device. Missing support for a device or dtype can produce an error.
Floating-point implementations can change rounding through operation order and numerical formats. Checking a transferred calculation therefore needs numerical tolerances and task results, rather than a requirement that every output bit match.
Data parallelism uses model replicas to process different data portions, then combines their gradients before consistent updates. An all-reduce operation aggregates arrays across the participating devices and returns the aggregate to each replica.7
If two devices each compute a mean over four equally weighted examples, averaging their gradients gives the combined eight-example mean. If their included counts are two and six, their means need weights \(2/8\) and \(6/8\). Equal averaging would give the smaller group too much influence. Masked-token objectives require the same count-aware treatment.
The global batch can stay fixed while its portions shrink across more devices. It grows only if the chosen local batches and device count imply growth. Communication, synchronization, and small local work can offset any computation savings. Greater throughput requires measurement rather than following from the device count.
Mixed precision uses different floating-point formats for different operations or stored state. Supported low-precision matrix operations can reduce traffic or use specialized hardware. Sensitive reductions and optimizer state may retain higher precision. The exact choices must be recorded.8
Loss scaling can enlarge small gradients during backward to reduce underflow in some formats. Unscaling must precede clipping or a comparison with ordinary gradients. An overflow policy may skip an update. These are real state changes, not merely a storage conversion. Loss, gradient, and held-out behavior need checks under the selected precision.
A GPU timing comparison additionally needs warmup, a declared timed region, and synchronization around asynchronous device work. Host transfers must be consistently included or excluded. Device, driver, library versions, tensor sizes, and repetitions belong with the timing.
23.6 Following shared storage into a differentiated result
A storage copy can preserve a gradient path, while a detached tensor can still share values. Following a transpose, a copy, and a detachment into one derivative distinguishes those properties. Let x contain [[0,1,2],[3,4,5]] on CPU in float32, with gradients enabled as a leaf.
It has shape [2,3], stride [3,1], storage offset zero, and grad_fn=None. The transpose t=x.T has shape [3,2] and stride [1,3]. It shares the same six values. Its grad_fn is PermuteBackward0, the autograd record created by this axis permutation.
Calling c=t.contiguous() copies those values into row-major [3,2] storage with stride [2,1]. It remains connected to the original leaf through the copy operation. Its independent storage does not imply independent gradient history.
The two-axis rule in §23.3 gives the view’s element offset from the start of its underlying storage. Here \(o=0\) and the transpose’s strides are [1,3]. Every term counts storage elements. If \(p_0\) is the underlying storage’s byte address and \(s\) the element size, the byte address is \(p_0+s\,\operatorname{offset}(i,j)\). A pointer already at the view’s first element must not add its storage offset again.
For a separate matrix product, §23.2’s contraction rule maps \([B,F]\) times \([F,D_{\mathrm{out}}]\) to \([B,D_{\mathrm{out}}]\). A changed stride does not change this mathematical shape rule. A compatible kernel can multiply a noncontiguous tensor correctly. Whether it copies internally or runs faster with contiguous input depends on the implementation and workload.
The runnable trace creates the specified leaf directly, checks sharing and copying, then differentiates the copied tensor’s squared sum. It also makes a detached alias without mutating it.
Code example: A shared transpose, independent copy, detached alias, and leaf gradient
import torch
x = torch.tensor([[0., 1., 2.], [3., 4., 5.]], requires_grad=True)
t = x.T
c = t.contiguous()
alias = x.detach()
assert t.untyped_storage().data_ptr() == x.untyped_storage().data_ptr()
assert c.untyped_storage().data_ptr() != x.untyped_storage().data_ptr()
assert alias.untyped_storage().data_ptr() == x.untyped_storage().data_ptr()
assert not alias.requires_grad
for name, value in [("x", x), ("t", t), ("c", c)]:
print(name, value.shape, value.stride(), value.storage_offset(),
value.requires_grad, type(value.grad_fn).__name__)
loss = c.square().sum()
loss.backward()
assert torch.allclose(x.grad, 2 * x.detach())
print(loss.item(), x.grad)The loss is \(0^2+1^2+2^2+3^2+4^2+5^2=55\). Backward gives x.grad=[[0,2,4],[6,8,10]]. It reaches the original coordinates through the copy and transpose. The detached alias shares storage but has no tracked history.
Conclusion: Layout, storage sharing, and differentiation are independent properties. The copy changes storage without cutting the gradient path. Detachment cuts that path without copying values. Chapter 24 adds the dataset and held-out evidence needed beyond these local checks.
Chapter checkpoint
Does [2,3] always require six distinct stored values? Does eval() clear Adam state? Does a legal broadcast guarantee the intended target pairing?
Answer: An expanded view can reuse three stored values for six logical positions. Evaluation mode changes layer behavior but neither disables gradients nor clears optimizer state. Broadcasting [B,1] against [B] can form [B,B], so pairwise target alignment requires its own shape check.
PyTorch Contributors. (2026). Broadcasting semantics. PyTorch 2.12 documentation.↩︎
PyTorch Contributors. (2026). Tensor views. PyTorch 2.12 documentation.↩︎
PyTorch Contributors. (2026). Tensor views. PyTorch 2.12 documentation.↩︎
PyTorch Contributors. (2026). Tensor.detach. PyTorch 2.12 documentation.↩︎
PyTorch Contributors. (2026). Tensor.numpy. PyTorch 2.12 documentation. With gradient recording enabled, ordinary conversion requires a supported CPU tensor that does not require gradients.↩︎
PyTorch Contributors. (2026). Autograd mechanics: Evaluation mode. PyTorch 2.12 documentation.↩︎
PyTorch Contributors. (2026). DistributedDataParallel. PyTorch 2.12 documentation. The example explains weighting needed for a combined mean, not a complete distributed program.↩︎
PyTorch Contributors. (2026). Automatic mixed precision examples. PyTorch 2.12 documentation.↩︎