Research Article

Journal of Construction Engineering and Project Management. December 2019. 16-33
https://doi.org/10.6106/JCEPM.2019.9.4.016


ABSTRACT


MAIN

  • INTRODUCTION

  • LITUREATURE REVIEW

  •   Conventional Construction Safety Training and Assessment Methods

  •   Gaming Technologies for Construction Safety Training and Assessment

  •   Immersive AR and VR for safety training

  • RESEARCH GOAL AND OBJECTIVES

  • AR-BASED HAZARD RECOGNITION SKILL ASSESSMENT MODULE

  • EVALUATION OF AP-BASED SAFETY SKILL ASSESSMENT

  •   Pre-Assessment Survey

  •   Paper-Based Assessment

  •   AR-Based Assessment

  •   Debrief

  • DISCUSSSION AND FUTURE RESEARCH

INTRODUCTION

The construction industry has one of the highest accident rates of any occupation, with onsite personnel being constantly exposed to the risk of fatal accidents [20]. According to the U.S. Occupational Safety and Health Administration (OSHA), the construction industry is responsible for more than 20 percent of all job-related fatalities in the nation’s private sector [24]. Falls from heights, being struck by objects, electrocutions, and caught-in-between accidents are construction’s “Fatal Four” leading causes, responsible for 64.2 percent of construction worker deaths in 2015 [24]. To deal with these safety hazards, it is critical to equip workers and onsite managers with the knowledge and cognitive capabilities required to identify these potential safety hazards on construction sites.

Existing paper or computer-based safety training programs (e.g. OSHA 10-hour and 30-hour training modules) cover fundamental knowledge on common job-related safety and health hazards in a classroom environment. However, a more effective way to acquire the cognitive capability to recognize potential safety hazards comes only through intensive practice in a real-world setting beyond the classroom. There is inevitably a significant difference between a classroom training environment and a real construction worksite due to realistic limitations and constraints. Unstructured, unique, and complex construction environments cannot be reconstructed in conventional classroom settings or described in textbooks. Therefore, despite mandatory safety training, 33 percent to 57 percent of safety hazards are simply not recognized by workers, and the rate of hazard recognition becomes even lower when inexperienced workers are exposed to multiple hazards at the same time [18, 28, 29]. Furthermore, the current paper-based safety training lacks any systematic way of quantitatively evaluating a trainee’s ability to recognize hazards either before or after the training, making it difficult to identify those needing additional education. The current paper-based/one-way lecturing approach is simply not an effective way of enhancing and evaluating workers’ cognitive capabilities.

To overcome this representational gap, scenario or case-based gaming technologies have been applied for construction safety training. Various visualization strategies and human-computer interaction methods have been used to realistically simulate real-world scenarios and interactions in typical construction environments. A number of interactive safety lectures [20] and hazard identification games have been developed [1]. Nguyen et al. [22] proposed aframework to explore, design, develop, and evaluate (EDDE) technology-assisted instruction for construction engineering and management. The crane operator safety training scenario developed by Fang and Teizer [5] involves a virtual crane operation practice environment for dangerous blind lifting operations. The social virtual reality (SVR) for safety education such as that proposed by Le et al. [13] provides an online SVR setting that allows multiple students to interact in an immersive virtual environment as they conduct safety inspections. These studies demonstrate the technical promise of immersive technologies and gaming technologies for construction safety training. In particular, safety gaming integrated with immersive technologies such as Augmented Reality (AR) or Virtual Reality (VR) have the potential to make a significant breakthrough in the area of construction safety training and assessment by allowing the trainees to view, analyze, and interact under realistic construction site conditions.

Despite these benefits, there has been little research into the strengths and weaknesses of immersive technologies as a platform to evaluate trainees’ safety knowledge and skills. For instance, a VR tool created by Sacks et al. [30] has been used only to improve training. Similarly, although Le et al. [14] used AR for training on various safety issues, their study did not seek to evaluate the impact of a truly immersive experience on safety training since they used 2-dimensional (2D) visualization devices instead of fully immersive 3-dimensional (3D) tools. Therefore, in this study we sought to explore the potential of immersive technologies as a platform for the assessment of hazard recognition skills by developing an AR-based assessment tool to evaluate trainees’ hazard recognition skills in an immersive construction environment. The tool created in this project was used for evaluations of undergraduate students who had learned relevant safety topics as part of a construction safety course. The effectiveness of AR for safety training assessment was then evaluated both quantitatively and qualitatively based on feedback from the participants.

This paper is organized as follows. The related work section reviews current safety training practices and the existing studies on gaming and immersive technologies used for safety training. The objective and scope of this study are then presented, followed by the methodology section, which describes the AR-based safety evaluation platform and the content of the safety assessment created for this research. The evaluation section then discusses the results obtained when utilizing the new platform to evaluate the safety recognition skills of undergraduate students in an existing construction safety class. The final section concludes the summary of this paper including the discussion and potential future research.

LITUREATURE REVIEW

Conventional Construction Safety Training and Assessment Methods

Construction safety training is conventionally achieved through traditional educational methods, such as safety Toolbox-training, e-learning, a proactive training system, and onsite training [8, 10, 15, 32]. The tools normally used to teach construction safety include lectures, conferences, films, and discussion sessions, among others [23]. OSHA 10-hour and OSHA 30-hour modules are commonly used training programs that teach basic knowledge to enable new workers to identify construction safety and health hazards. However, these traditional methods and programs suffer from a number of serious shortcomings. Modifications to paper-based materials involve excessive cost and time, even though the content of the training often becomes out of date very quickly. Trainees attending the paper-based training programs also find the contents inadequate, ineffective, and less engaging [35]. Interviews with construction personnel from 49 projects revealed that traditional training programs are poorly designed and are not effective in transferring essential knowledge to trainees [36]. To maximize the learning outcomes, Jeelani et al. [9] stressed the importance of providing customized training experiences, which is not easy in conventional construction safety training. Shelton et al. [33] agreed, reporting that conventional safety training programs are not effective in either engaging students or in scaffolding their learning. As noted earlier, existing construction safety teaching methods are primarily limited to paper-based learning, one-way lecturing, and OSHA training lectures. These teaching methods are less effective in training students to deal with complex real-world construction situations.

Gaming Technologies for Construction Safety Training and Assessment

One way to address the shortcomings of current construction safety training and assessment methods could be to utilize gaming technologies to boost the achievement of trainees attending construction safety training sessions. Lin et al. [18] proposed a game called Safety Inspector, where students play the role of a safety inspector, exploring a virtual construction site to identify potential safety hazards. The researchers found that the realistic visualizations in Safety Inspector increased student motivation, participation rate, and interest in learning about the construction hazards found on a construction site. Nguyen et al. [22] proposed a framework to explore, design, develop, and evaluate (EDDE) technology-assisted instruction that provides useful guidance for developing and testing educational games. Based on their proposed EDDE framework, they developed a software application, EDDEaid, that adopted appropriate gaming strategies such as role playing or a quiz-show, targeted Bloom’s Taxonomy level, and took into account their students’ backgrounds to derive effective game-based education designs. Pedro et al. [25] proposed a framework to incorporate safety content in other construction engineering courses in an attempt to provide practical experience to support safety knowledge acquisition, and Le et al. [16] developed a VR and AR construction safety education framework that consisted of three modules: Safety Knowledge Dissemination (SKD), Safety Knowledge Reflection (SKR), and Safety Knowledge Assessment (SKA). McKinney et al. [19] suggested that learning strategies based on the use of visual simulation and games yielded a better performance in student learning. Nguyen et al. [23] concurred, noting that students preferred to have visual simulations included in the teaching process, exhibiting a positive attitude toward the use of video and animation technology in a safety training course.

Immersive AR and VR for safety training

Innovative technologies such as VR and AR are likely to be more effective in delivering comprehensive educational materials compared to the safety lectures traditionally used in the construction industry [31]. Utilizing immersive technologies is thus a very promising direction because of its effective visualization and manipulation capabilities. Pham et al. [26] developed an energy-efficient learning system utilizing web-based panoramic VR technology for interactive construction safety education and the crane operator safety training module was developed by Fang et al. [6]. The SVR safety education scenario proposed by Le et al. [13] created an online SVR that allowed multiple students to interact in an immersive gaming environment to conduct safety inspections or play games where they conduct predefined tasks like window installation. Sacks et al. [30] created a VR tool for safety training related to concrete construction. Their test results showed that their new VR tool was more effective than conventional safety training in making students aware of the potential hazards involved.

AR demonstrates distinctive features that can benefit effective construction safety training and assessment, creating an environment where computer-generated virtual information overlays the user’s view of a real-world scene. Recent technical advances have led to the widespread use of AR tools in various areas, including education, training and the development of new techniques, a project commissioning, and so on [3, 7]. This growing tendency to utilize AR technologies in practical education applications highlights its significant potential for improving existing technologies and providing a better quality of training, especially in the disciplines of engineering education, heavy construction equipment training, fire protection, human behavior and cognitive processes [2, 4, 11, 17, 27, 34]. Nguyen [21] has forecast that by 2018, 50 percent of the current AR solution providers will have been acquired by large consumer platforms, moving technology and innovation to resource-rich firms that will enhance their access to a wider range of end users for marketing, advertising, gaming, education and other forms of consumer engagement. Li et al. [16] conducted a structured literature review of applications of AR in hazards recognition and identification, safety training and education, safety instruction and inspection, among other applications, reporting that a near reality and multi-scenario environment and knowledge database is the likely next step in the development of these technologies. Kim et al. [12] proposed an integration of computer vision and AR that proactively warns workers about potential hazardous situations during construction, enabling workers to identify, assess and avoid potentially dangerous situations before accidents occur by displaying augmented hazard information on a wearable device. Other researchers have suggested an AR-based learning system for architecture students based on a formative assessment mechanism, which assists trainees to find answers on their own by providing indirect suggestions when they fail to correctly answer questions. An AR application that improves the evaluation of post-disaster damage and safety has also been proposed, and various references or pre-disaster information (virtual data) for actual post-disaster building data (real buildings) have been provided with a mobile AR application.

This literature review shows obvious evidence to suggest that gaming technologies will enhance construction safety training by allowing the trainees to interact virtually with a realistic construction site environment. Particularly, AR or VR have significant potential to improve the effectiveness of construction safety training and assessment by placing the trainees in a vivid immersive environment where they can view and analyze authentic construction site conditions. However, compared to the extensive application of immersive technologies for safety training, few studies have examined their effectiveness as a tool to evaluate hazard recognition skills.

RESEARCH GOAL AND OBJECTIVES

Compared to traditional paper-based assessment methods, immersive assessment has considerable potential as a method to properly evaluate hazard recognition skills by placing trainees in a realistic construction environment. Since trainees are required to identify unsafe situations under the complex and dynamic conditions present on a construction site, assessing their safety knowledge and hazard recognition skills using immersive platforms will test whether the trainees can apply the knowledge learned in the classroom on actual construction sites. Therefore, the goal of this research is to evaluate the effectiveness of an immersive safety training assessment platform compared to traditional paper-based assessment methods. Although several researchers have utilized AR or VR tools to teach safety training, as yet few researchers have sought to determine the effect of AR or VR-based safety training by performing an appropriate assessment exercise. Therefore, the research question guiding this study is: “What are the benefits of adopting immersive AR/VR as a platform for the assessment of construction safety training?” The study’s objectives are as follows:

•To measure each student’s achievement using both a standard paper-based assessment and an AR-basedassessment.
•To investigate the statistical correlation between each student’s achievement levels on the paper-based assessment and the AR-based assessment.
•To classify the level of difficulty and unfamiliarity involved in using an AR-based assessment.
•To identify students’ preferred assessment method for evaluating their academic achievement for the safety training received in a higher education institute.

AR-BASED HAZARD RECOGNITION SKILL ASSESSMENT MODULE

A new AR-based tool to assess a trainee’s ability to recognize unsafe situations from an immersive construction environment was developed for this study. The construction site conditions, interactions between workers and equipment, and unsafe situations in the environment were designed based on five rounds of in-depth interviews with a professional construction safety training provider. Working with the expert in construction safety training and assessment, the research team created a virtual construction site and a list of potential safety hazards. Among these unsafe situations, some were considered relatively straightforward for trainees to recognize after receiving traditional safety education. Examples of this type of hazard include walking/working near open edges, improper uses of ladders, and not wearing PPEs. Other unsafe situations are more difficult to recognize, even after education, because of the need to analyze interactions between multiple objects. Example here include potential safety hazards such as being struck by reversing trucks, working under a live load, or caught in/ between, all of which require observation of the dynamic interactions among workers, equipment, and structures. Based on the list of safety hazards and details created, shown in Table I, an AR-based safety skill assessment tool was developed using a mixed reality smart goggle, Microsoft Hololens. Figure I shows a typical scene of a commercial building under structural construction. In this scene, workers and equipment interact around the nine-story building.

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F1.jpg
FIGURE I.

VIRTUAL CONSTRUCTION SITE

TABLE I. DESCRIPTION OF UNSAFE SITUATIONS DESIGNED IN THE ASSESSMENT MODULE

No. Unsafe Situations Description of Situation Possible Corrections Scenes
1 Not wearing PPE
properly
Workers are not wearing one
of the following PPEs
(e.g. hard hat, safety glasses,
harness) or wearing
incorrectly colored vests.
Employers should provide
adequate PPE for all the
operations and enforce their
use.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-1.jpg
2 Not wearing PPE
properly for specific
operations
Worker is welding without a
welding mask.
Employers should provide
adequate PPE in all the
operations and enforce their use.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-2.jpg
3 Falling from height Worker is standing near the
slab without guardrails or
personal fall arrest systems.
Employees should be protected
by using of
guardrails, nets, or personal fall arrest systems.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-3.jpg
4 Standing on unstable
objects
Worker on scaffolding is
standing on a box of
materials as a working
platform.
Unstable objects should not be
used as working platforms on
scaffolding.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-4.jpg
5 Use of top step of a
portable ladder
Worker is sitting or standing
on the top step of portable
ladders.
Top step/rung of a portable
ladder should not be used
unless it is specifically
designed for such purposes.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-5.jpg
6 Being in the path of
moving equipment
Worker is walking across the
path of a backing truck
without a flagger controlling
the movement of the
equipment.
Flaggers should be deployed
when there are insufficient
signs, signals, and barricades
available.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-6.jpg
7 Being close to a
predefined unsafe
location
Worker is inside a predefined
unsafe location inside a
guardrail.
A danger sign board should be
placed to prevent access to an
unsafe location, and workers
need to be instructed not to
enter those unsafe locations.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-7.jpg
8 Equipment is
carrying loose
materials that are not
firmly strapped.
A forklift is moving a
collection of pipes that are
not strapped firmly to the
equipment.
Rigging equipment for material
handling should be inspected
before each use.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-8.jpg
9 Caught by rotating
equipment
Worker is working or
walking between a rotating
crane and a fence that can
lead to a caught-in-between
incident.
A danger sign board and
guardrails should be placed to
prevent workers from walking
or working near rotating
equipment.
http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_T1-9.jpg

Safe and unsafe situations change randomly in each assessment, shifting the locations of objects (workers, equipment, etc.) and changing the states of objects (e.g. wearing or not wearing PPEs). A rule-based system was developed to allow the random generation of safe and unsafe conditions at the beginning of each assessment. For each rule shown in Table II, either a true or false state is randomly assigned, and the situations impacted by the rules become safe or unsafe accordingly. As a result, different participants will be experiencing different possible scenarios. Even when a participant repeats the evaluation, he/she will be experiencing different states of safe and unsafe conditions. For instance, a worker might be sitting on the top rung of a ladder or a signal person can be in the path of a reversing truck (Figure II and III) depending on the states. For each new trial, the state will be chosen randomly to show either a safe or unsafe state. Each interacting object has one or more rules attached to it.

TABLE II. SAFE AND UNSAFE SITUATION ASSIGNMENT RULES

Object (state) True False
Worker (working, walking) Wearing proper PPE Not wearing proper PPE
Welder (welding) Wearing a welding mask Not wearing a welding mask
Worker (harness, on scaffolding) Wearing a fall harness Not wearing a fall harness
Worker (using a ladder) Properly using the ladder Using the top rung of the ladder
Worker (standing near guardrails) Standing outside the guardrails Standing inside the guardrails
Worker (walking) Passing safe locations Passing unsafe locations
Worker (standing near unsafe zones) Working in safe locations Working in unsafe locations
Signal person (signaling) Providing correct signals to a truck Providing incorrect signals to a truck
Signal person (standing) Standing in a safe location Standing in the path of a truck
Forklift (carrying pipes) Carrying a properly strapped load of pipes Carrying a load of pipes that is not
properly strapped
Dump truck (moving) Following signals Not following signals
Crane (moving load) Not moving load above workers Moving the load above workers

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F2.jpg
FIGURE II.

A WORKER SITTING ON THE TOP THE LADDER

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F3.jpg
FIGURE III.

A SIGNAL PERSON UNAWARE OF A BACKING TRUCK

EVALUATION OF AP-BASED SAFETY SKILL ASSESSMENT

The AR tool developed for this study was used to evaluate the hazard recognition skills of students who had learned relevant topics in a construction safety course. The study consisted of 18 undergraduate students that were taking the OSHA Construction Safety course. Students completed the evaluation process comprising of both traditional paper-based and AR-based tests. The evaluation procedure was the following five steps:

(1)Pre-Assessment Survey.
(2)Paper-based Assessment.
(3)AR-based Assessment.
(4)Post-Assessment Survey.
(5)Debrief.

During the study, four sets of Microsoft HoloLens goggles were used simultaneously. The trial was conducted after the fifth lecture to ensure that the topics and material in all steps of the test had already been covered in the course.

Pre-Assessment Survey

Before the assessments of their safety knowledge and hazard recognition skills, participants completed a survey to collect information on their demographics, familiarity with AR tools, and experience with safety and construction projects. The questions posed and answers received are as below:

Demographics

•What is your personal identifier?

To protect participants’ personal information, a personal identifier was assigned to each one of them at the beginning of the evaluation. These personal identifiers were used to collect data from different steps of the assessment and automatically linked the associated results for the AR-based and paper-based assessments.

•What is your age?

18 participants had an average age of 25 with a standard deviation of 4.58 (Figure IV).

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F4.jpg
FIGURE IV.

AGES OF PARTICIPANTS

•What is your classification? (freshman, sophomore, junior, senior, or other)Eleven participants were seniors, and seven participants were juniors.

•Are you a US or an International Student? (US, International Student)All but one of the participants were U.S. students.

Familiarity with AR tools

•How many times have you used Augmented or Virtual Reality? The answer of this question was shown in Table III.

TABLE III. PARTICIPANTS’ RESPONSES

Never 1 - 5 times 5 - 15 times More than 15 times
12 5 1 0

Safety and Construction Project Experience

•Is this your first-time receiving OSHA certification? (Yes or No)Four responded “No”, and 14 of the participants responded “Yes”.

•How many times have you worked on a professional construction site? Table IV presented the answer.

TABLE IV. PARTICIPANTS’ RESPONSES

Never 1 - 5 times 5 - 15 times More than 15 times
8 3 0 7

•Have you had any construction management experience outside educational coursework? Table V showed the participamt’s responses.

TABLE V. PARTICIPANTS’ RESPONSES

No experience Some experiences A lot of experiences
8 3 7

Paper-Based Assessment

Paper assessment is the traditional approach for giving tests. The participants were given 20 minutes to complete the paper assessment. The paper assessment had 14 questions that were created by the instructor teaching the Construction Safety course to ensure that participants had learned accurate and relevant safety course materials before taking the paper-based assessment. The paper-based assessment covered the following seven OSHA topics:

(1)Signs, signals, and barricades.
(2)Personal protective and life-saving equipment.
(3)Motor vehicles and mechanized equipment.
(4)Cranes.
(5)Fall protection.
(6)Ladders.
(7)Scaffolds.

AR-Based Assessment

In the AR-based assessment, participants were asked to explore the immersive construction site and select objects (e.g., workers, equipment) that they considered represented unsafe conditions (Figure V).

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F5.jpg
FIGURE V.

AGES OF PARTICIPANTS

To support the participants’ observations, the tool allowed participants to rotate and scale the size of the construction site so that they could either inspect the entire construction site or zoom in to look at specific situations (Figure I, Figure II, and Figure III). After selecting an object, participants were asked to select reasons why they thought that the situation was unsafe (Figure VI and Figure VII).

Grading Method for the AR-Based Assessment

This interactive assessment tool does not follow a conventional grading approach. The conventional testing approach is to ask sets of questions and expect the same sets of answers. If a problem was left blank, then points will be subtracted. The interactive assessment assumes that all conditions are safe and expect the participant to only mark unsafe situations. The reason behind not following the conventional approach is to make the assessment as close to the real-world situation as possible. Safety inspectors should be searching for unsafe conditions by examining all cases without anyone pointing out specific sets of circumstances, then report/correct any violations if found. By removing any visible indicator, the assessment becomes closer to what a real safety inspector will be performing in a real construction site.

Participants were not allowed to modify their answers. As shown above in Figure VI and VII, they could confirm a dialog asking whether to mark the item unsafe or cancel and were required to select one or more reasons for the situation being unsafe. If participants were allowed to change selections from unsafe to safe, then they might use the list of hazardous reasons to see if any items in the list.

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F6.jpg
FIGURE VI.

UNSAFE SELECTION CONFIRMATION DIALOG

http://static.apub.kr/journalsite/sites/jcepm/2019-009-04/N042090402/images/jcepm_09_04_02_F7.jpg
FIGURE VII.

REASONS FOR MARKING THE SITUATION UNSAFE

Test scores

Out of 18 test results in Table VI, only 16 were usable because two files in the interactive assessment were corrupted. The 16 usable results had an average of 65 percent, a median of 68 percent, and a standard deviation of 14 percent for the paper assessment and an average of 49 percent, a median of 49 percent, and a standard deviation of 12 percent for the assessment tool.

TABLE VI. PARTICIPANTS’ SCORES

ID Paper Assessment HoloLens Assessment ID Paper Assessment HoloLens Assessment
1 64 47 10 36 47
2 71 49 11 64 49
3 Corrupted 12 79 46
4 64 39 13 71 55
5 57 65 14 79 42
6 Corrupted 15 86 59
7 71 48 16 57 45
8 57 53 17 36 55
9 71 16 18 79 61

As expected, participants performed better in the paper assessment than the interactive one. Some of the factors for these anticipated results are as follow:

•Participants had clear sets of questions to answer in the paper assessment, but unknown in the interactive one. They had to examine the whole scene in search of hazardous situations; the number of variables is high compared to the paper assessment. The amount of information given to participants to examine the interactive evaluation is far overwhelming than the paper one.
•Participants were not allowed to change their unsafe selection in the interactive assessment, but they were able to change their answers in the paper one, which made the participants think twice before attempting to mark a situation unsafe.
•Participants had years of practice to take tests using the conventional paper method, but the majority never used such technology before. Figuring out and adapting to new technology in a short time is not trivial.

Post-Assessment Survey

The post-assessment survey asked two questions about the participants’ feelings during the assessment and the comparative effectiveness of the AR-based and paper-based assessments. As shown in Table VII, most of the participants felt the HoloLens Trial was an excellent method to test their safety skills and knowledge with three participants saying it was a perfect method. When compared with the paper-based pre-test, as shown in Table VIII, 13 participants considered HoloLens simulation is more accurate than the paper-based test, and all 18 participants preferred HoloLens simulation.

Participants’ feelings during the assessment

TABLE VII. PARTICIPANTS' RESPONSES

Poor Fair Good Excellent Perfect
How well do you think you did on the pre-assessment? 0 5 9 3 1
How well do you think you did on the HoloLens simulation? 1 3 8 6 0
How well do you feel the pre-assessment tested your skill and knowledge of
OSHA certification?
0 2 7 9 0
How well do you feel the HoloLens Trial tested your skill and knowledge of
OSHA certification?
0 1 6 8 3

Comparing the paper and interactive assessments for preference and effectiveness

Table VIII. PARTICIPANTS' RESPONSES

Pre-assessment HoloLens simulation Both the same
Which assessment do you think more accurate for your
skills and knowledge of OSHA certification?
0 13 5
Which did you prefer? 0 18 0
Which test do you believe was easier? 5 12 1

Debrief

The purpose of the debrief is to encourage participants to provide more feedback for the free-form questions orally rather than asking for written responses because people tend to skip free-form written questions. Debrief was done in a group setting to encourage participants to share more thoughts because usually when someone says something, others are more inclined to agree or disagree or comment by relating to the initially shared feedback compared to the one-on-one setting. The group was divided into two to five participants for each group, and then the following questions have been asked:

•How did you feel about using the HoloLens before the HoloLens Simulation began? Few responded nervous or worried, but the majority responded exited, and only one was unsure.
•How did you feel about using the HoloLens after the HoloLens Simulation was over? Few participants had negative feedback such as "hoped it would be easier to control," "had zoom issues.", and "eyes hurt." However, the majority provided positive feedback such as "exciting," "wow" and expressed a preference in taking exams using Microsoft HoloLens rather paper-based exams. One participant and another agreed reported that HoloLens assessment was hard than the paper assessment because with the Hololens "you needed to perform many actions," but with paper-based assessment, "you only need to memorize."
•Did you feel like you were in control of the experience? Did you feel like the augmented world responded well when you tried to do something? The majority reported issues such as problems with hand gestures, poor battery life, disappearing of the pointer, leading restricted views and devices not fitting their head correctly, and others. However, a quarter of responses reported no issues after the initial adjustment period.
•If you have used AR/VR before, how did this experience compare to previous experiences?Only two participants reported using AR/VR. However, only one of them expressed a preference for AR, and the other one did not report a preference for VR or AR.
•How would you compare the paper-based assessment to the HoloLens simulation? Which one did you prefer?Majority of participants reported positive feedback toward HoloLens simulation due to the visualization part. However, few felt that paper-based assessment would be better with testing facts knowledge. There was a universal agreement that HoloLens provided applied knowledge, and one participant recommended combining both HoloLens and paper-based assessments.
•Anything else you want to say?Majority of responses were positive, but the majority of participants did not provide any more comments. Few responses suggested that students would be more inclined to show up to lectures if HoloLens were widely used. One participant thought that the tool is more appropriate to be used in a company setting. Several participants criticized the tool regarding some of the geometrical aspects, zooming out, and vague menu options, and expressed the needs for further improvements.

DISCUSSSION AND FUTURE RESEARCH

In this study, we compared two different assessment methods to determine which of the two the participants undergoing OSHA training preferred for assessing their knowledge and skill. Almost all the participants expressed their preference for our proposed AR-based interactive assessment based on Microsoft HoloLens over the traditional paper-based assessment. Although a group of participants mentioned issues with the user interface and control methods, there was still a clear majority of responses preferring the visualization and “fun” factor aspects of using the assessment tool. Further work is needed to refine the user experience for participants, after which the evaluation should be repeated to confirm that the necessary improvements have been implemented and the issues raised minimized.

When comparing the paper-based assessment results with those obtained using the AR-based assessment, the participants returned better results on the paper-based assessment than the other assessment when grading on a 100 percent scale, even though they thought the AR simulations were easier than the paper-based one. There was a 16 percent difference between the averages for the two assessment methods. The number of possible choices is taken into consideration when weighing these results, however; the paper-based assessment contained 14 multiple choice questions, with 4 choices each, while the AR-based assessment had 23 interactable objects linked to 4 safe/unsafe questions, leading to 92 safe/unsafe decisions depending on the randomization of that particular run of the assessment tool.

The findings of this study show that the interactive assessment provided a unique advantage due to its ability to measure the students’ knowledge and ability to identify unsafe conditions on a construction site by creating conditions and visualizations that are not feasible using traditional approaches. This case study demonstrates the importance and need for assessment methods to bridge the chasm between applied and theoretical knowledge. Exposing the participants to contextually dependent safety hazards in a more natural setting than is possible given the inherent limitations of a word problem enabled us to explore and begin to test aspects of the hazard recognition and OSHA safety curriculum in ways that were not previously possible.

We demonstrated that the assessment developed for this study has strong potential to provide enhanced assessments and become an important educational tool, especially for teaching students to identify safety issue in complex and potentially hazardous environments such as construction sites. One important component that needs to be added and in the future is to develop better evaluation methods for these assessments that allow the participants to discuss their results with experts to develop a deeper understanding of the scenarios presented. It would also be helpful to explore ways to incorporate more deterministic methods in the scene to investigate whether this innovative methodology outperforms the existing subjective methods.

References

1

Albert, A., Matthews, H. R., and Kleiner, B. M., "Enhancing Construction Hazard Recognition and Communication with Energy-Based Cognitive Mnemonics and Safety Meeting Maturity Model: Multiple Baseline Study.", Journal of Construcgtion Engineering and Management, vol .140, no. 2, pp. 04013042-1-12, 2013.

10.1061/(ASCE)CO.1943-7862.0000790
2

Chen, Y.-C., Chi, H. -L, Hung, W.-H, and Kang, S.-C., "Use of Tangible and Augmented Reality Models in Engineering Graphics Courses.", Journal of Professional Issues in Engineering Education, vol. 137, no. 4, pp. 267-276, 2011.

10.1061/(ASCE)EI.1943-5541.0000078
3

Chu, M., Matthews, J., and Love, P. E. D. "Integrated Mobile Building Information Modeling and Augmented Reality Systems: An Experimental Study.", Journal of Automation in Construction, vol. 85, no. 2018, pp. 305-316, 2018.

10.1016/j.autcon.2017.10.032
4

Dünser, A., Steinbügl, K., Kaufmann, H., and Glück, J., "Virtual and Augmented Reality as Spatial Ability Training Tools.", 7th ACM SIGCHI New Zealand Chapter'S International Conference on Computer-Human Interaction: Design Centered HCI., Christchurch, New Zealand, pp. 125-132, 2006.

10.1145/1152760.1152776
5

Fang, Y. and Teizer, J., "A Multi-user Virtual 3D Training Environment to Advance Collaboration Among Crane Operator and Ground Personnel in Blind Lifts.", 2014 International Conference Computing in Civil and Building Engineering, Orlando, FL, pp. 2071-2078, 2014.

10.1061/9780784413616.257
6

Fang, Y., Teizer, J., and Marks, E., "A Framework for Developing an As-Built Virtual Environment to Advance Training of Crane Operators.", Constr. Research Congress 2014, Atlanta, GA, 2014.

10.1061/9780784413517.004
7

Golparvar-Fard, M., Pena-Mora, F., and Savarese, S., "Integrated Sequential As-Built and As-Planned Representation with D4ARToolsinSupportofDecision-MakingTasksintheAEC/FMIndustry.", Journal of Construction Engineering and Management, vol. 137, no. 12, pp. 1099-1116, 2011.

10.1061/(ASCE)CO.1943-7862.0000371
8

Ho, C.-L. and Dzeng, R.-J., "Construction Safety Training Via E-Learning: Learning Effectiveness and user Satisfaction.", Computing and Education, vol. 55, no. 2, pp. 858-867, 2010.

10.1016/j.compedu.2010.03.017
9

Jeelani, I., Han, K., and Albert, A., "Development of Immersive Personalized Training Environment for Construction Workers.", Computing in Civil Engineering, Seattle, WA, 2017.

10.1061/9780784480830.050
10

Jeschke, K. C., Kines, P., Rasmussen, L., Anderseen, L. P. S., Dyreborg, J., Ajslev, J., Kabel, A., Jensen, E., and Ansdersen, L. L., " Process Evaluation of a Toolbox-Training Program for Construction Foremen in Denmark.", Journal of Safety Science, vol. 94, pp. 152-160, 2017.

10.1016/j.ssci.2017.01.010
11

Kaufmann, H., "Collaborative Augmented Reality in Education.", Institute of Software Technology and Interactive Systems Vienna University of Technology, Vienna, Austria, 2013.

12

Kim, K., Kim, H., and Kim, H., "Image-Based Construction Hazard Avoidance System using Augmented Reality in Wearable Device.", Automation in Construction, vol. 83, pp. 390-403, 2017.

10.1016/j.autcon.2017.06.014
13

Le, Q. T., Pedro, A., Lim, C. R., Park, H. T., Park, C. S., and Kim, H. K., "A Framework for using Mobile Based Virtual Reality and Augmented Reality for Experiential Construction Safety Education.", International Journal of Engineering Education, vol. 31, no. 3, pp. 713-725, 2015.

14

Le, Q. T., Pedro, A., and Park, C. S., "A Social Virtual Reality Based Construction Safety Education System for Experiential Learning.", Journal of Intelligent and Robotic System, vol. 79, no. 3-4, pp. 487-506, 2014.

10.1007/s10846-014-0112-z
15

Li, H., Lu, M., Chan, G., and Skitmore, M., "Proactive Training System for Safe and Efficient Precast Installation.", Automation in Construction, vol. 49, no. A, pp. 163-174, 2015.

10.1016/j.autcon.2014.10.010
16

Li, X., Yi, W., Chi, H.-L., Wang, X., and Chan, A. P. C., "A Critical Review of Virtual and Augmented reality (VR/AR) Applications in Construction Safety.", Automation in Construction, vol. 86, pp. 150-162, 2018.

10.1016/j.autcon.2017.11.003
17

Liarokapis, F., Mourkoussis, N., White, M., Darcy, J., Sifniotis, M., Petridis, P., Basu, A., and Lister, P. F., "Web3D and Augmented Reality to Support Engineering Education.", World Transaction on Engineering And Technology Education, vol. 3, no. 1, pp. 11-14, 2004.

18

Lin, K.-Y., Son, J. W., and Rojas, E. M. "A Pilot Study of a 3D Game Environment for Construction Safety Education.", Journal of Information Technology in Construction, vol. 16, pp. 69-84, 2011.

19

McKinney, K., "Personal Reflection: Lessons from My Students and Other Reflections on SoTL.", International Journal of Scholarship of Teaching and Learning, vol. 3, no. 2, pp. 31, 2009.

10.20429/ijsotl.2009.030231
20

Namian, M., Albert, A., Zuluaga, C. M., Behm, M., "Role of Safety Training: Impact on Hazard Recognition and Safety Risk Perception.", Journal of Construction Engineering and Management, vol. 142, no. 12, 2016.

10.1061/(ASCE)CO.1943-7862.0001198
21

Nguyen, T., "Market Guide for Augmented Reality.", Gartner Research, Stamford, CT, 2018.

22

Nguyen, T., Solis, F. A. M., and O;Brien, W. J., "A Framework to Explore, Design, Develop and Evaluate Technology-Assisted Instruction for Construction Engineering and Management.", Journal of Information Technology in Construction, vol. 17, pp. 434-464, 2012.

23

Nguyen, T.-H., and Khoo, I-H., "Learning and Teaching Engineering Courses with Visualizations.", The World Congress on Engineering and Computer Science, San Francisco, CA, 2009.

24

OSHA., Commonly used Statistics, 2014.

25

Pedro, A., Le, Q. T., and Park, C. S., "Framework for Integrating Safety into Construction Methods Education Through Interactive Virtual Reality.", Journal of Professional Issues in Engineering Education and Practice, vol. 142, no. 2, 2016.

10.1061/(ASCE)EI.1943-5541.0000261
26

Pham, H. C., Dao, N.-N., Kim, J.-U., Cho, S., and Park, C. S. "Energy-Efficient Learning System using Web-Based Panoramic Virtual Photoreality for Interactive Construction Safety Education.", Sustainability, vol. 10, no. 7, pp. 1-17, 2018.

10.3390/su10072262
27

Phan, V. T., and Choo, S. Y., "Augmented Reality-Based Education and Fire Protection for Traditional Korean Buildings.", International Journal of Archtectural Computing, vol. 8, no. 1, pp. 75-91, 2010.

10.1260/1478-0771.8.1.75
28

Ringen, K., Dong, X. S., Goldenhar, L. M., and Cain, C. T., "Construction Safety and Health in the USA: Lessons From a Decade of Turmoil.", Annals of Work Expo and Health, vol. 62, Supplement 1, pp. 25-S33, 2018.

10.1093/annweh/wxy06930212888
29

Ruttenberg, R., "The Economic and Social Benefits of OSHA-10 Training in the Building and Construction Trades.", The Center for Construction Resarch and Training, Silver Spring, MD, 2013.

30

Sacks, R., Perlman, A., and Barak, R., "Construction Safety Training using Immersive Virtual Reality.", Construction Management and Economics, vol. 31, no. 9, pp. 1005-1017, 2012.

10.1080/01446193.2013.828844
31

Saltan, F., and Arslan, Ö., "The use of Augmented Reality in Formal Education: A Scoping Review.", Eurasia Journal of Mathematics, Science and Technology Education, vol. 13, no. 2, pp. 503-520, 2017.

10.12973/eurasia.2017.00628a
32

Shamsuddin, K., Ani, M., Ismail, A., and Ibrahim, M., "Investigation the Safety, Health and Environment (SHE) Protection in Construction Area.", International Journal of Engineering and Technology, vol 2, no. 6, pp. 624-636, 2015.

33

Shelton, C. C., Warren, A. E., and Archambault, L. M.,"Exploring the use of Interactive Digital Storytelling Video: Promoting Student Engagement and Learning in a University Hybrid Course.", TechTrends, vol. 60, no. 5, pp. 465-474, 2016.

10.1007/s11528-016-0082-z
34

Wang, X., Dunston, P. S., and Skiniewski, M., "Mixed Reality Technology Applications in Construction Equipment Operator Training.", 21st International Symposium on Automation and Robotics in Constr uction, Jeju, South Korea, 2004.

10.22260/ISARC2004/0069
35

Wyman, J. R., "Safety and the Security Professional: a Guide to Occupational Safety and Health Strategies.", Butterworth-Heinemann, Woburn, MA, 2000.

36

Zuluaga, C. M., Namian, M., and Albert, A., "Impact of Training Methods on Hazard Recognition and Risk Perception in Construction.", Construction Research Congress 2016, San Juan, Puerto Rico, 2016.

10.1061/9780784479827.285
페이지 상단으로 이동하기