Towards Human Activity Recognition and Objective Performance Assessment in Human Patient Simulation: A Case Study

In this paper, we present an exploratory work towards the recognition of activities and performing real-time objective assessment in human patient simulation (HPS). Although HPS has been pervasively used in medical and nursing programs in developed countries, there is a huge need in providing consistent and objective assessment on student performance during HPS. Current methods all depend on instructor subjective observation, which not only could lead to inconsistency in evaluation across different students and different instructors, but also are very time and resource intensive. Recognizing complex human activities in the context of HPS is very challenging because it involves the recognition of human actions, gestures, as well as human-object and human-mannequin interactions. Hence, we study the feasibility of developing such a system for a particular simulation where a student is required to first identify the patient and then place a neck brace on the patient's neck. The system we that we have developed identifies the actions and activities in the simulation and provides qualitative assessment on the student performance using computer vision, OpenPose, and TensorFlow. The system also consists of a debriefing mobile app that the student and instructor could use to view an automatically generated report with supporting key frames captured and annotated by our system.