Viru reddy Viru reddy - 4 years ago 161
MySQL Question

Special characters in mysql sqoop import as Text and Parquet file

I have 1000 tables with more than 100000 records in each table in mysql. The tables have 300-500 columns.
Some of tables have columns with special characters like .(dot) and space in the column names.

Now I want to do sqoop import and create a hive table in HDFS in a single shot query like below as textfile

sqoop import --connect ${domain}:${port}/$(database) --username ${username} --password ${password} --table $(table) -m 1 --hive-import --hive-database ${hivedatabase} --hive-table $(table) --create-hive-table --target-dir /user/hive/warehouse/${hivedatabase}.db/$(table)


After this the hive table is created but when I do select * query on the table it shows error as

This error output is a sample output.

Error while compiling statement: FAILED: RuntimeException java.lang.RuntimeException: cannot find field emp from [0:emp.id, 1:emp.name, 2:emp.salary, 3:emp.dno]


However when I do the import as parquet file the table is created and when I do select query the query gives me expected results. Even the .(dot) are replaced with _(underscore) automatically

Why is this happening?

Answer Source

The reason for the above is due to a limitation in Hive. In hive they have disabled creation of hive tables with dot/colon in the column names.

They have a Jira for this at https://issues.apache.org/jira/browse/HIVE-10120

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download