Avik Das Avik Das - 9 months ago 42
Javascript Question

How to use AJAX to upload large CSV file?

We have some CSV files with large number of records. Need to upload this files into a MySQL database using web interface. These files are collected from different field works and uploaded into the server through a web application. The web application is developed using PHP.

Initially the number of records was very small. But, now it's became very large and users unable to upload the file into the server.

Problems -

  1. We have to wait for upload the entire file to start processing.

  2. We can see the data validation errors after uploading the entire file.

  3. Uploading large files are not supported by default.

  4. PHP execution time is limited.

So, I am looking for a solution to upload CSV file content as chunk. It will help me to process a small amount of data per request.

My idea is to read data from CSV file using JavaScript. Then POST this data using AJAX.

Want to see the data validation errors against small amount of data and stop execution, if required.

Answer Source

After doing some research on web I have found HTML5 JavaScript File API is very useful to read content from local file system. I have written a small script and it's perfectly working.

For this example, I have created two files -

  1. index.htm
  2. service.php


<!DOCTYPE html>
		<meta charset="UTF-8">
		<title>Upload large CSV file</title>
			body{font-size:8pt; color:#333}
			#wrap{width:500px; margin:5px auto}
			#responce{height:200px; overflow-y:scroll; border:1px solid #ddd;}

		<div id="wrap">
			<ul id="responce">
			</ul><!-- // response -->
			<input type="file" id="fileSelected"/>
			<button id="btnUpload">Upload</button>
		</div><!-- // wrap -->
			var reader_offset = 0;		//current reader offset position
			var buffer_size = 1024;		//
			var pending_content = '';
			* Read a chunk of data from current offset.
			function readAndSendChunk()
				var reader = new FileReader();
				var file = $('#fileSelected')[0].files[0];
				reader.onloadend = function(evt){
					//check for end of file
					if(evt.loaded == 0) return;
					//increse offset value
					reader_offset += evt.loaded;
					//check for only complete line
					var last_line_end = evt.target.result.lastIndexOf('\n');
					var content = pending_content + evt.target.result.substring(0, last_line_end);
					pending_content = evt.target.result.substring(last_line_end+1, evt.target.result.length);
					//upload data
				var blob = file.slice(reader_offset, reader_offset + buffer_size);
			* Send data to server using AJAX
			function send(data_chank)
					url: "service.php",
					method: 'POST',
					data: data_chank
				}).done(function(response) {
					//show responce message received from server
					$( '#responce' ).append( '<li>' + response + '</li>');
					//try for next chank
			* On page load
					reader_offset = 0;
					pending_content = '';


$content = file_get_contents('php://input');
$lines = explode("\n", $content);
foreach($lines as $line){
    $csv_row = str_getcsv($line);
    //save data into database