asj177 asj177 - 3 months ago 403
Java Question

Download Large file from server using REST template Java Spring MVC

I have a REST service which sends me a large ISO file ,there are no issues in the REST service .
Now I have written a Web application which calls the rest service to get the file ,on the client(web app) side I receive a Out Of memory Exception.Below is my code

HttpHeaders headers = new HttpHeaders();//1 Line

headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
headers.set("Content-Type","application/json");//3 Line
headers.set("Cookie", "session=abc");//4 Line
HttpEntity statusEntity=new HttpEntity(headers);//5 Line
String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line

ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line


I receive out of memory exception at 7 line ,I guess i will have to buffer and get in parts ,but dont know how can i get this file from the server ,the size of the file is around 500 to 700 MB .
Can anyone please assist .

Exception Stack:

org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause

java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:3236)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)


My Server Side REST Service Code which is working fine is

@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{



byte[] reportBytes = null;
File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);

if(result.exists()){
InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName);
String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
response.setHeader("Content-Type",type);

reportBytes=new byte[100];//New change
OutputStream os=response.getOutputStream();//New change
int read=0;
while((read=inputStream.read(reportBytes))!=-1){
os.write(reportBytes,0,read);
}
os.flush();
os.close();






}

Answer

Here is how I do it. Based on hints from this Spring Jira issue.

RestTemplate restTemplate // = ...;

// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

From the aforementioned Jira issue:

Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.

Comments