TalBeno TalBeno - 1 year ago 185
Scala Question

How to Increase (AKKA-HTTP Client) singleRequest response chuck size

I am reading an HTTP Response stream from a URL.
The response runForeach returns a stream of small chunks.
I want to increase the chunk size.

I have an application.conf file where I experimented with the following (increasing default values), but nothing changed. Wasn't sure which config param is the one I need. I must be doing something wrong.



my code:

var config: Config = ConfigFactory.load()

implicit val system = ActorSystem("test", config)
import system.dispatcher
implicit val materializer = ActorMaterializer()

val responseFuture: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = source))

responseFuture.flatMap { response =>
val responseData = response.entity.dataBytes
responseData.runForeach(chunk => {

Thank you.

Answer Source

The max-chunk-size is a protection against filling memory, it will fail the request if the server would respond with larger chunks, it does not dictate the size of the chunks in any other way.

The chunk size is dictated by things such as OS network buffers, network packet size, the server responding with http chunks, etc. you cannot really say "I want 50m chunks".

What you can do is to concatenate the incoming ByteStrings, into 50mb chunks, concat is a zero copy operation so the overhead should be fairly low.

There is an example in the Akka Streams docs of how to create a custom graph stage that does "rechunking" that might be useful here: http://doc.akka.io/docs/akka/2.4/scala/stream/stream-cookbook.html#chunking-up-a-stream-of-bytestrings-into-limited-size-bytestrings

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download