Node.js Streams Silently Drop Data When You Forget Backpressure

The symptom

A reporting job read rows from Postgres and wrote each one to a CSV on disk. The job reported success. The file was short. Not corrupted, not truncated mid line, just missing thousands of rows from the middle and the end.

No error was thrown. No promise rejected. The exit code was zero.

The cause

This was the code, and the bug is the return value nobody was reading:

for await (const row of cursor) {
    file.write(toCsvLine(row))
}

file.end()

write() returns a boolean. true means the data went into the buffer, false means the internal buffer is already over its high water mark and you are supposed to stop and wait for the drain event.

Ignoring that return value does not usually lose data on its own, because Node keeps queueing writes in memory. The loss came from what followed. file.end() was called as soon as the loop finished iterating, while a large queue of buffered writes was still waiting for the disk. The stream closed, the pending writes were discarded, and nothing in that sequence is treated as an error.

The fix

Let the runtime handle backpressure by connecting the source to the destination with a pipeline instead of a manual loop:

import { pipeline } from 'node:stream/promises'
import { Readable } from 'node:stream'

await pipeline(
    Readable.from(cursor),
    async function* (source) {
        for await (const row of source) yield toCsvLine(row)
    },
    file
)

pipeline waits for drain, propagates errors in both directions, destroys the streams on failure, and only resolves once everything has actually been flushed. Memory use stayed flat at a few megabytes instead of climbing past a gigabyte on the largest report.

If you must write manually, honour the signal:

if (!file.write(line)) {
    await once(file, 'drain')
}

And always await the close, because end() is a request, not a guarantee:

file.end()
await once(file, 'finish')

How to catch this in review

• Any bare stream.write() whose return value is discarded deserves a question.
• Any end() that is not awaited in some form deserves a question.
• A job that reports success but produces output of varying size is a backpressure bug until proven otherwise.

Silent data loss is worse than a crash. A crash gets fixed the same day, while a short file gets discovered a quarter later by someone reconciling numbers.