rmcast.sys AdvanceWindow integer overflow

PGM (Pragmatic General Multicast, RFC 3208) is a reliable one-to-many multicast transport layered on IP multicast. A sender pushes original data packets (ODATA) to a multicast group; receivers report losses back to the sender with negative acknowledgements (NAKs); the sender retransmits the missing data as repair packets (RDATA). The protocol is deliberately asymmetric: the sender keeps the authoritative state, and a receiver’s NAK tells the sender which byte ranges still need to be repairable.

To bound how long a receiver may ask for a repair, the sender maintains a send window — a sliding byte range covering the data it is still willing to retransmit. Bytes age out of the window once they are older than the configured repair window; the window is described on the sender by a ring-buffer offset (sender->WindowOffset), a per-message byte size (sender->BytesPerMessage), the first sequence number still in the window (sender->FirstSeq), and a running byte total (sender->TotalBytesReceived). When the window advances, the byte cost of the retired batch is count × BytesPerMessage. The sender then uses ((seq - FirstSeq) × BytesPerMessage + WindowOffset) mod BufferSize to convert a sequence number from a NAK into an offset into its ring buffer, and copies that range into the RDATA packet.

The retirement count and the per-message stride are both sender-side state — the count is how many messages aged out in this advance, the stride is configured. Nothing in that multiply is bounded against 2^32. Anything in that arithmetic that overflows or truncates corrupts the sender’s geometry silently, and the next repair read — the one a receiver’s NAK triggers — walks that corrupted geometry.

The window advance that wraps

AdvanceWindow computed the byte cost of each retirement batch in 32-bit and added the truncated value to the running totals and the window offset:

/* AdvanceWindow — vulnerable build */
uint32_t stride  = sender->BytesPerMessage;               /* +0x6c */
uint32_t advance = stride * messages;                     /* 32-bit mul, wraps past 2^32 */

sender->MessagesReceived   += messages;                   /* +0x74 */
sender->TotalBytesReceived += (uint64_t)advance;          /* +0x78 */

uint64_t next = sender->WindowOffset + (uint64_t)advance; /* +0x88 */
sender->WindowOffset = next;                              /* +0x88 */
if (next >= sender->BufferSize)                           /* +0x58 */
    sender->WindowOffset = next - sender->BufferSize;     /* ring base slid by the wrapped delta */

sender->FirstSeq += messages;                             /* +0xa0, slid by the full count */
return messages;                                          /* no error path: the wrap is invisible */

The product messages × BytesPerMessage is computed in ULONG (32-bit). Once it exceeds 2^32, the high half is dropped. The truncated advance is then written into TotalBytesReceived and used to slide WindowOffset. The function returns the message count — nothing in the path detects the wrap — so every subsequent operation on this sender walks its buffers through a corrupted coordinate system.

The corruption is silent but total. WindowOffset no longer reflects how far the window has actually advanced; TotalBytesReceived no longer reflects how many bytes the sender has actually streamed. Worse, FirstSeq advances by the full message count while WindowOffset advances by the truncated byte cost, so the sequence base and the byte-offset anchor desynchronise. Anything that later reads those fields to compute “where in the ring buffer is sequence N” — and the repair path is the obvious reader — converts a NAK’s sequence number into a slot the window no longer owns.

How it lands out of bounds

The consumer of WindowOffset is the repair path. A NAK arrives asking for sequence seq; the sender computes offset = ((seq - FirstSeq) × BytesPerMessage + WindowOffset) mod BufferSize, then reads BufferBase + offset to fill the RDATA payload. The ring buffer is built for single-shot copies — a message never straddles the tail when the geometry is correct. With WindowOffset set from the truncated delta, that invariant breaks: a sequence whose offset resolves near the wrapped base lands the fixed-size copy across the buffer’s edge, reading past the allocation into whatever the pool parked next to it.

The sender kernel is the one doing that read, on behalf of a multicast it is already sending. The corruption is a sender-side precondition — a window wide enough that a single retirement batch’s byte cost crosses 2^32 — but the read that turns it into a leak is triggered by an ordinary NAK from any group member whose target sequence resolves against the corrupted offset. The read bytes go into the RDATA packet, which is multicast back to the group, so the same attacker receives them as payload. The OOB is on the read side: information disclosure rather than pool corruption. A long enough overrun walks neighbouring pool allocations; whatever kernel memory was parked next to the ring buffer is leaked onto the wire.

There is also a reliable DoS angle independent of the leak: the overrun can cross from the allocation’s last mapped page into an unmapped one, taking the sender kernel into a bugcheck on the copy.

The reason this is a remote primitive, not a local one, is PGM’s trust model. The sender is the one running the vulnerable code, but the read that consumes the corrupted offset is triggered by a NAK — and a NAK is just a packet from a receiver. Any host that can join the multicast group and emit a NAK turns the sender’s pre-corrupted geometry into an on-the-wire leak.

The patch

The multiply is now computed as a full 64-bit product, and an explicit overflow guard gates whether the byte-total accumulator advances:

/* AdvanceWindow — patched build */
uint32_t stride = sender->BytesPerMessage;                 /* +0x6c */
uint64_t full   = (uint64_t)stride * (uint64_t)messages;   /* 64-bit product, no truncation */

if ((full >> 32) == 0)                                      /* overflow guard: high dword zero */
    sender->TotalBytesReceived += full;                     /* +0x78 */

sender->MessagesReceived += messages;                       /* +0x74 */
uint64_t next = sender->WindowOffset + full;                /* +0x88, advances by the exact byte cost */
sender->WindowOffset = next;
if (next >= sender->BufferSize)                             /* +0x58 */
    sender->WindowOffset = next - sender->BufferSize;

sender->FirstSeq += messages;                               /* +0xa0 */
return messages;

WindowOffset now consumes the full 64-bit product, so the byte cost of a retirement batch can no longer wrap and desynchronise the ring anchor from FirstSeq. The high-dword check is defensive — both operands arrive as 32-bit values, so the guard cannot trip on the inputs the caller controls today — but it states the intent plainly: a count × stride product that overflows its width is a fault condition, not a silent wrap, and the byte total stays put rather than absorbing a corrupted delta.

The principle is multiply in the width the consumer needs: any count × stride that lands in a pointer or a running offset must be computed at a width that cannot wrap before it is added. The repair path in this same module has its own multiply into a pointer — a separate bug, fixed separately this month.

Attack path

flowchart TD
    A["PGM sender with a wide window streams enough traffic to retire over 2^32 bytes in one batch"] --> B["AdvanceWindow computes the retirement byte cost in 32-bit and the multiply wraps"]
    B --> C["sender writes the truncated delta into WindowOffset and TotalBytesReceived"]
    C --> D["a group member sends a NAK whose target sequence resolves against the corrupted offset"]
    D --> E["the single-shot repair copy overruns the ring buffer tail, leaking adjacent pool into the RDATA packet on the wire"]

PGM is configured explicitly (it is not enabled by default), and the corruption needs a window wide enough to retire 2^32 bytes of messages in one batch. But once that precondition holds, every group member is inside the trust boundary for the repair read — so the leak is triggered by a single NAK from any receiver.

A byte offset computed as count × element-size must use the full width before it is added to a pointer. messages × BytesPerMessage in 32-bit silently drops the carry, and the truncated value then slides a window offset that subsequent repair reads consume as a pointer. The window machinery keeps running but on a corrupted coordinate system, and the first reader of the corrupted geometry — here, the repair path — is the one that turns it into an OOB. Anywhere a count and a stride are multiplied to produce a pointer offset, ask what width the multiply is and what happens on overflow; the fix is the 64-bit product plus a high-bits-non-zero reject, every time.