Journal of Shanghai Jiao Tong University(Science) ›› 2020, Vol. 25 ›› Issue (5): 569-577.doi: 10.1007/s12204-020-2177-0

• • 上一篇    下一篇

Judging the Normativity of PAF Based on TFN and NAN

LI Zhiqiang (李志强), BAO Jinsong (鲍劲松), LIU Tianyuan (刘天元), WANG Jiacheng (王佳铖)     

  1. (College of Mechanical Engineering, Donghua University, Shanghai 201600, China)
  • 出版日期:2020-10-28 发布日期:2020-09-11
  • 通讯作者: BAO Jinsong (鲍劲松) E-mail:bao@dhu.edu.cn

Judging the Normativity of PAF Based on TFN and NAN

LI Zhiqiang (李志强), BAO Jinsong (鲍劲松), LIU Tianyuan (刘天元), WANG Jiacheng (王佳铖)     

  1. (College of Mechanical Engineering, Donghua University, Shanghai 201600, China)
  • Online:2020-10-28 Published:2020-09-11
  • Contact: BAO Jinsong (鲍劲松) E-mail:bao@dhu.edu.cn

摘要: The normativity of workers’ actions during producing has a great impact on the quality of the products
and the safety of the operation process. Previous studies mainly focused on the normativity of each single producing
action instead of considering the normativity of continuous producing actions, which is defined as producing action
flow (PAF) in this paper, during operation process. For this issue, a normativity judging method based on two-
LSTM fusion network (TFN) and normativity-aware attention network (NAN) is proposed. First, TFN is designed
to detect and recognize the producing actions based on skeleton sequences of a worker during complete operation
process, and PAF data in sequential form are obtained. Then, NAN is built to allocate different levels of attention
to each producing action within the sequence of PAF, and by this means, an efficient normativity judging is
conducted. The combustor surface cleaning (CSC) process of rocket engine is taken as the experimental case, and
the CSC-Action2D dataset is established for evaluation. Experiment results show the high performance of TFN
and NAN, demonstrating the effectiveness of the proposed method for PAF normativity judging.

关键词: producing action normativity, sequential model, attention mechanism, deep learning

Abstract: The normativity of workers’ actions during producing has a great impact on the quality of the products
and the safety of the operation process. Previous studies mainly focused on the normativity of each single producing
action instead of considering the normativity of continuous producing actions, which is defined as producing action
flow (PAF) in this paper, during operation process. For this issue, a normativity judging method based on two-
LSTM fusion network (TFN) and normativity-aware attention network (NAN) is proposed. First, TFN is designed
to detect and recognize the producing actions based on skeleton sequences of a worker during complete operation
process, and PAF data in sequential form are obtained. Then, NAN is built to allocate different levels of attention
to each producing action within the sequence of PAF, and by this means, an efficient normativity judging is
conducted. The combustor surface cleaning (CSC) process of rocket engine is taken as the experimental case, and
the CSC-Action2D dataset is established for evaluation. Experiment results show the high performance of TFN
and NAN, demonstrating the effectiveness of the proposed method for PAF normativity judging.

Key words: producing action normativity, sequential model, attention mechanism, deep learning

中图分类号: