FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

本文共有974个字,关键词:

问题:运行ods_to_dwd_db_init.sh将数据从ods层处理后到dwd层进行动态分区时出错:FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.

解决:
增加

set hive.exec.max.dynamic.partitions=2000;
set hive.exec.max.dynamic.partitions.pernode=2000;

方法:

dwd_resource_views="
set hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions=2000;
set hive.exec.max.dynamic.partitions.pernode=2000;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
insert overwrite table ${APP}.dwd_resource_views partition(dt)
select
    id,
    create_time,
    creator_id,
    resource_type,
    resource_id,
    equip,
    date_format(create_time, 'yyyy-MM-dd')
from ${APP}.ods_sgx_teares_resource_views
where dt='$do_date';
"
版权声明:本文为作者原创,如需转载须联系作者本人同意,未经作者本人同意不得擅自转载。
添加新评论
暂无评论