Alexander Stepaniuk Alexander Stepaniuk - 1 year ago 254
C++ Question

boost::asio fails to read more than 65536 bytes from file

I'm failing to read more than

bytes into a buffer from a file using

Starting from
th byte the buffer contains the the data from the very beginning of the file, rather than the expected data.

Here is a code example, which reproduces the issue:

auto handle = ::CreateFile(L"BigFile.xml", GENERIC_READ, FILE_SHARE_READ, nullptr, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, nullptr);
boost::asio::io_service ios;

boost::asio::windows::stream_handle streamHandle(ios, handle);

const auto to_read_bytes = 100000;
char buffer[to_read_bytes];

boost::asio::async_read(streamHandle, boost::asio::buffer(buffer, to_read_bytes), [](auto &ec, auto read) {
std::cout << "Bytes read: " << read << std::endl;

auto bufferBegin = std::string(buffer, 38);
auto bufferCorrupted = std::string(buffer + 65536, 38); // <- it contains bytes from the beginning of the file

std::cout << "offset 0: " << bufferBegin << std::endl;
std::cout << "offset 65536: " << bufferCorrupted << std::endl;


That code produces an output:

> Bytes read: 100000
> offset 0: <?xml version="1.0" encoding="UTF-8"?>
> offset 65536: <?xml version="1.0" encoding="UTF-8"?>

The source file is bigger than 65536.

This is reproducible with boost 1.61 + VS2015. Also that issue was in boost 1.55 + VS2010.

Operating systems are: Windows 7 and Windows Server 2008R2.

My questions are:

1. Is that the known limitation in
or in

2. If it is the known limitation, what would be the safe size of the buffer to read data? Is it safe to have a buffer of size 65536, or it should be smaller?

Answer Source

I've delved into the deeps of asio and I'll have to call this a bug on Windows. async_read() is basically this loop in asio/impl/read.hpp:

for (;;)
    stream_.async_read_some(buffers_, ASIO_MOVE_CAST(read_op)(*this));

    total_transferred_ += bytes_transferred;

    if (!ec && bytes_transferred == 0)

The actual maximum number of bytes that will be read in one call comes from completion_condition.hpp:

enum default_max_transfer_size_t { default_max_transfer_size = 65536 };

The problem is the async_read_some() call above. You'll notice that there's no offset to tell it where to start reading. Because asio is using asynchronous reads (also called "overlapped" on Windows), an offset would have to be specified for every read.

This is where it ends up, in asio/detail/impl/win_iocp_handle_service.ipp:

DWORD bytes_transferred = 0;
op->Offset = offset & 0xFFFFFFFF;
op->OffsetHigh = (offset >> 32) & 0xFFFFFFFF;
BOOL ok = ::ReadFile(impl.handle_,,
    &bytes_transferred, op);

op->Offset and op->OffsetHigh are always 0. The pointer inside your buffer will advance correctly, but every chunk will be read from the start of the file.

There's an async_read_some_at() that's available, so I'm not sure why async_read() isn't using that. I think it's using just regular non-blocking IO on POSIX, where file pointers are moved around automatically on every read, so an explicit offset isn't needed. It look like reads over 64k were never tested on Windows (which is a bit scary).

The documentation for the OVERLAPPED structure says this:

The Offset and OffsetHigh members together represent a 64-bit file position. It is a byte offset from the start of the file or file-like device, and it is specified by the user; the system will not modify these values. The calling process must set this member before passing the OVERLAPPED structure to functions that use an offset, such as the ReadFile or WriteFile (and related) functions.

There's also this part in Synchronous and Asynchronous I/O:

The system does not maintain the file pointer on asynchronous handles to files and devices that support file pointers (that is, seeking devices), therefore the file position must be passed to the read and write functions in the related offset data members of the OVERLAPPED structure. For more information, see WriteFile and ReadFile.

As for a workaround, I'm not sure. I don't think you can use async_read() at all on Windows if your file is more than 64k. You might be stuck using async_read_at() with a maximum of detail::default_max_transfer_size bytes and track the number of bytes read yourself. I'd also file a bug report.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download