Loading data from one Hive table to another with partition

In some cases you may need to set hive.exec.dynamic.partition.mode=nonstrict to be able to insert data into partitioned table, for example,

CREATE TABLE hivePartitionedTable
(
          c1    int
        , c2    int
        , c3    string
)
PARTITIONED BY (year  int)
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY ','
STORED AS SEQUENCEFILE
;

then this INSERT will work:

set hive.exec.dynamic.partition.mode=nonstrict;

INSERT INTO hivePartitionedTable PARTITION (year)
VALUES (1,2,'3', 1999);

Create a copy of your table

CREATE TABLE my_table_backup LIKE my_table;

Enable dynamic partitioning

SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
SET hive.mapred.mode = nonstrict;

Copy the table

INSERT OVERWRITE TABLE my_table_backup PARTITION (ds)
SELECT * FROM my_table 
WHERE ds = ds;

The where clause is needed if you use strict mode.


this is because you are missing the partition info in your insert query

  insert overwrite table reg_logs_org PARTITION (utc_date)
  select id, region_code, sum(count), utc_date
  from 
  reg_logs
  group by 
  utc_date, id, region_code

Tags:

Hadoop

Hive