Yassine BHS Yassine BHS - 6 months ago 120
Java Question

Inserting data with UCanAccess from big text files is very slow

I'm trying to read text files .txt with more than 10.000 lines per file, splitting them and inserting the data in Access database using Java and UCanAccess. The problem is that it becomes slower and slower every time (as the database gets bigger).

Now after reading 7 text files and inserting them into database, it would take the project more than 20 minutes to read another file.

I tried to do just the reading and it works fine, so the problem is the actual inserting into database.

N.B: This is my first time using UCanAccess with Java because I found that the JDBC-ODBC Bridge is no longer available. Any suggestions for an alternative solution would also be appreciated.

Answer

If your current task is simply to import a large amount of data from text files straight into the database, and it does not require any sophisticated SQL manipulations, then you might consider using the Jackcess API directly. For example, to import a CSV file you could do something like this:

String csvFileSpec = "C:/Users/Gord/Desktop/BookData.csv";
String dbFileSpec = "C:/Users/Public/JackcessTest.accdb";
String tableName = "Book";

try (Database db = new DatabaseBuilder()
        .setFile(new File(dbFileSpec))
        .setAutoSync(false)
        .open()) {

    new ImportUtil.Builder(db, tableName)
            .setDelimiter(",")
            .setUseExistingTable(true)
            .setHeader(false)
            .importFile(new File(csvFileSpec));

    // this is a try-with-resources block, 
    //     so db.close() happens automatically
}

Or, if you need to manually parse each line of input, insert a row, and retrieve the AutoNumber value for the new row, then the code would be more like this:

String dbFileSpec = "C:/Users/Public/JackcessTest.accdb";
String tableName = "Book";
try (Database db = new DatabaseBuilder()
        .setFile(new File(dbFileSpec))
        .setAutoSync(false)
        .open()) {

    // sample data (e.g., from parsing of an input line)
    String title = "So, Anyway";
    String author = "Cleese, John";

    Table tbl = db.getTable(tableName);
    Object[] rowData = tbl.addRow(Column.AUTO_NUMBER, title, author);
    int newId = (int)rowData[0];  // retrieve generated AutoNumber
    System.out.printf("row inserted with ID = %d%n", newId);

    // this is a try-with-resources block, 
    //     so db.close() happens automatically
}

To update an existing row based on its primary key would be

Table tbl = db.getTable(tableName);
Row row = CursorBuilder.findRowByPrimaryKey(tbl, 3);  // i.e., ID = 3
if (row != null) {
    // Note: column names are case-sensitive
    row.put("Title", "The New Title For This Book");
    tbl.updateRow(row);
}

Note that for maximum speed I used .setAutoSync(false) when opening the Database, but bear in mind that disabling AutoSync does increase the chance of leaving the Access database file in a damaged (and possibly unusable) state if the application terminates abnormally while performing the updates.