Tenant Isolation Is Verified, Not Assumed

Tenant isolation is verified, not assumed

A container gets a writable root disk. It writes four kilobytes into it. Later, a raw read of that same disk returns sixty kilobytes the container never wrote, and some of those bytes belong to whoever used that block before.

That is the bug Cloudflare disclosed on 4 September. Oren Yomtov, a researcher at Accomplish, reported it through the bug bounty program. Cloudflare Containers and Cloudflare Sandboxes both ran on the affected storage path, and the company says it fixed the issue with no customer action required.

The mechanism is worth understanding because it is not a Cloudflare mistake. It is a storage default that any multi-tenant platform can inherit.

What thin provisioning does to a deleted disk

Cloudflare Containers give each container a writable root disk built on Linux device mapper thin provisioning. Each container runs inside its own Firecracker virtual machine, and the disk appears to the guest as /dev/vdc.

Thin provisioning allocates physical storage only when a virtual disk writes to a region it has not used before. The affected pools used a 64 KiB block size. When a container's root disk was deleted, its physical blocks went back into a pool shared across customer accounts.

The pool had one option set: skip_block_zeroing.

That option does what its name says. When a previously used 64 KiB block is reassigned, a full-block write replaces everything in it. A smaller write does not. Write 4 KiB and the remaining 60 KiB keeps whatever the block's previous owner left behind.

The kernel documentation describes the same tradeoff. If you are not zeroing newly allocated data, it suggests a much larger block size, in the region of 128 MiB, because the cost of clearing blocks is the reason the option exists at all.

The exploit is small

Reading an unmapped region of a fresh thin disk returns zeroes, so the first attempt reveals nothing. The trick is to force allocation.

The proof of concept found 64 KiB-aligned regions that corresponded to free space in the guest's ext4 filesystem, then wrote one aligned 4 KiB block into each region. Each write triggered allocation of a shared 64 KiB block. Because zeroing was off, the untouched 60 KiB could still hold another container's bytes. A raw read of /dev/vdc then surfaced them.

The researchers used ext4 directory block checksums to tell their own test filesystem's blocks apart from blocks that came from somewhere else. Across six production placements they examined 5,614 testable directory blocks. None belonged to their filesystem. Checksum analysis attributed 2,700 distinct foreign directory inodes.

Residual data showed up on 18 of 24 placements and 20 of 22 underlying nodes, across four continents. The recovered block types included directory structures, database pages and structurally complete SQLite databases.

What it was not

The disclosure is careful about the limits, and they matter. The technique could not target a specific customer, workload, host or data. Residual bytes were not guaranteed to be present. An attacker could not reach an actively attached disk or modify another customer's live data, and Cloudflare found no evidence anyone else used the vector. The researchers deleted the recovered data after submitting it.

Those caveats make the bug less dangerous than the headline suggests. They do not make it less instructive. A cross-tenant read is a cross-tenant read, and the boundary that failed was a configuration flag, not an exotic exploit.

The fix had two halves

Cloudflare's first move was to remove skip_block_zeroing from the thin pool configuration across the fleet. That restored the default behavior of clearing blocks before exposing them, and the researchers confirmed their proof of concept stopped working.

That was not enough. Zeroing new allocations does not sanitize blocks already mapped into existing thin devices. Those mappings lived in running container disks and in each host's cache of prepared snapshots for OCI image layers. A new container could inherit a mapping from a cached layer without allocating the block again, and the residual bytes in its unused regions stayed readable.

So Cloudflare retired every running container disk and removed every cached image snapshot created before the fix. They drained hosts off-peak, restarted the virtual machines on each host, and cleared each host's image cache so that disks and layers were recreated with zeroed allocations.

The lesson generalizes. A data-isolation fix has two halves. The allocation path is the obvious one. The already-materialized state is the one that gets missed, and it is the half that stays exploitable after you think you have shipped the patch.

The timeline is the other thing worth copying. Report at 15:26 UTC, incident opened at 18:45, runtime fix merged at 21:27, pool changes merged at 22:03, rollout started at 23:15. Roughly eight hours from disclosure to rollout. The full cleanup of cached snapshots finished on 19 September.

Why builders should care

Every AI product that runs untrusted code is now a multi-tenant storage problem. Agent sandboxes, code interpreters, browser tools and evaluation harnesses all execute someone else's bytes on infrastructure someone else owns. Cloudflare Sandboxes, the product built on Containers, is exactly that shape, and it was in scope for this bug.

The practical rule is that isolation is a property you verify, not one you inherit by picking a provider. Three habits follow from that.

Do not treat "our platform isolates tenants" as a fact about your system. It is a claim about a configuration, and configurations change between versions, regions and instance types.

Encrypt tenant data at rest with keys the storage layer cannot read. Residue you cannot decrypt is residue that does not leak, and it is the one control that survives a bug you did not find.

Read the raw device in your own test suite. The proof of concept is not exotic. A container that writes aligned small blocks and then reads back the surrounding bytes will tell you whether the platform zeroes its allocations.

None of this is an argument against multi-tenant platforms. The economics of running one tenant per host are brutal, and Cloudflare's response was fast and transparent. It is an argument against trusting a boundary you have never tested. The block that leaked was 60 KiB of storage the new tenant never wrote and never asked for.