问题:运行ods_to_dwd_db_init.sh将数据从ods层处理后到dwd层进行动态分区时出错:FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.
解决:
增加
set hive.exec.max.dynamic.partitions=2000;
set hive.exec.max.dynamic.partitions.pernode=2000;
方法:
dwd_resource_views="
set hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions=2000;
set hive.exec.max.dynamic.partitions.pernode=2000;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
insert overwrite table ${APP}.dwd_resource_views partition(dt)
select
id,
create_time,
creator_id,
resource_type,
resource_id,
equip,
date_format(create_time, 'yyyy-MM-dd')
from ${APP}.ods_sgx_teares_resource_views
where dt='$do_date';
"