Avik Das Avik Das - 1 month ago 6
Javascript Question

How to use AJAX to upload large CSV file?

We have some CSV files with large number of records. Need to upload this files into a MySQL database using web interface. These files are collected from different field works and uploaded into the server through a web application. The web application is developed using PHP.

Initially the number of records was very small. But, now it's became very large and users unable to upload the file into the server.

Problems -


  1. We have to wait for upload the entire file to start processing.

  2. We can see the data validation errors after uploading the entire file.

  3. Uploading large files are not supported by default.

  4. PHP execution time is limited.



So, I am looking for a solution to upload CSV file content as chunk. It will help me to process a small amount of data per request.

My idea is to read data from CSV file using JavaScript. Then POST this data using AJAX.

Want to see the data validation errors against small amount of data and stop execution, if required.

Answer

After doing some research on web I have found HTML5 JavaScript File API is very useful to read content from local file system. I have written a small script and it's perfectly working.

For this example, I have created two files -

  1. index.htm
  2. service.php

index.htm

<!DOCTYPE html>
<html>
	<head>
		<meta charset="UTF-8">
		<title>Upload large CSV file</title>
		<style>
			body{font-size:8pt; color:#333}
			#wrap{width:500px; margin:5px auto}
			#responce{height:200px; overflow-y:scroll; border:1px solid #ddd;}
		</style>
	</head>

	<body>
		<div id="wrap">
			<ul id="responce">
				
			</ul><!-- // response -->
			
			<input type="file" id="fileSelected"/>
			<button id="btnUpload">Upload</button>
		</div><!-- // wrap -->
		
		<script
			  src="https://code.jquery.com/jquery-3.1.1.min.js"
			  integrity="sha256-hVVnYaiADRTO2PzUGmuLJr8BLUSjGIZsDYGmIJLv2b8="
			  crossorigin="anonymous"></script>
		<script>
			var reader_offset = 0;		//current reader offset position
			var buffer_size = 1024;		//
			var pending_content = '';
			
			/**
			* Read a chunk of data from current offset.
			*/
			function readAndSendChunk()
			{
				var reader = new FileReader();
				var file = $('#fileSelected')[0].files[0];
				reader.onloadend = function(evt){
					
					//check for end of file
					if(evt.loaded == 0) return;
					
					//increse offset value
					reader_offset += evt.loaded;
					
					//check for only complete line
					var last_line_end = evt.target.result.lastIndexOf('\n');
					var content = pending_content + evt.target.result.substring(0, last_line_end);
					
					pending_content = evt.target.result.substring(last_line_end+1, evt.target.result.length);
					
					//upload data
					send(content);
				};
				var blob = file.slice(reader_offset, reader_offset + buffer_size);
				reader.readAsText(blob);
			}
			
			/**
			* Send data to server using AJAX
			*/
			function send(data_chank)
			{
				$.ajax({
					url: "service.php",
					method: 'POST',
					data: data_chank
				}).done(function(response) {
				
					//show responce message received from server
					$( '#responce' ).append( '<li>' + response + '</li>');
					
					//try for next chank
					readAndSendChunk();
				});
			}
			
			/**
			* On page load
			*/
			$(function(){
				$('#btnUpload').click(function(){
					reader_offset = 0;
					pending_content = '';
					readAndSendChunk();
				});
			});
		</script>
	</body>
</html>

service.php

<?php
$content = file_get_contents('php://input');
$lines = explode("\n", $content);
foreach($lines as $line){
    $csv_row = str_getcsv($line);
    //save data into database
    //----
}