Jiangbo Jiangbo - 3 months ago 7x
SQL Question

load multi tables into hdfs one time using sqoop

I want to load data of multi tables in mysql into hdfs, the tables' name like a_0_0, a_0_1, a_0_2. how to Load these tables data into hdfs one time using Sqoop, can I use union ?


There are many ways that you can achieve this.

  1. if you want to import all tables in mysql db, you can use: import-all-tables

  2. if you want to import some data from some tables (meaningful data), you can use: Free-form Query Imports

  3. if you want to import number of tables, you can right shell script:

    while [ ${i} -le 5 ]
        echo "importing table a_0_${i}"
        #here write your full sqoop command, this is just an example  
        #sqoop import --connect  --table a_0_${i}
      i=$(( i + 1 ))

Now run the shell script: sqoop command will run 6 times as per the logic and import 6 tables.

$ ./importAll.sh
importing table a_0_0
importing table a_0_1
importing table a_0_2
importing table a_0_3
importing table a_0_4
importing table a_0_5

NOTE: you will have to modify logic in shell script according to your need. Solutions I am proposing is based on the details provided in question.