Abstract Details

Serious Games In Corporate Learning

This study evaluates the ability of serious games through the use of a computer-based game called Datacenter Defense as a driver for engagement, and to what extent this enables them to be effective as learning mechanisms. In this instance, the learning scope was to build from product-specific knowledge to solution-driven understanding. This particular scope is the outcome of a training need identified within information security company Trend Micro, where knowledge among support engineers is often kept product-specific due to the compartmentalization of their workload. However, an understanding of the solution rather than the individual products is desirable since Trend Micro customers often deploy more than one product as part of a security strategy. This is progressively accentuated as the integration between these products becomes more prevalent, making broader understanding more critical. Knowledge was measured using multiple selection surveys before and after playing the game, while engagement was measured through the use of multiple selection surveys and focus group discussions. The results after this study showed a significant improvement of knowledge relating to the learning objectives scoped for the game, as well as the majority of the participants reported being engaged by the game. These findings extend previous studies in an area where more literature is needed (Bodnar et al., 2016), showing that games can help enhance learning and the attitude of learners towards the learning subject.

Introduction

This study employed a computer-based game called Datacenter Defense, created for the purpose of this study, to answer the following question: To what extent can learning be achieved by employing serious games to engage specialized technical professionals to acquire technical knowledge outside their main area of expertise?

Games and gamification elements can be used in any educational medium, such as during classroom activities, but this project will specifically focus on implementing this as a form of video game. Zyda defined video games as “a mental contest, played with a computer according to certain rules for amusement, recreation, or winning a stake” (2005, p. 25). The video game in this research project was designed as a serious game. It will be considered a game, fitting the definition coined by Ibrahim et al. (2011) where the game will contain challenges, goals, feedback and a game story. Likewise, it will also fit the definition for a serious game proposed by Ritterfeld et al. (2009), where a game can involve fun, but also present educational benefits while retaining other game elements, like being engaging, impactful, purposeful and meaningful.

 

This comes from the training needs analysis of technical professionals in the field of digital information security in the company Trend Micro, where there is an almost ubiquitous extrinsic driver to specialize in single products or product silos rather than in the broader solution strategy, since day to day engagements require that in-depth specific knowledge to solve product-specific technical issues. With serious games possessing the ability to be entertaining as well as instructional (Zyda, 2005), and potentially being able to drive higher engagement to play (Apostol et al., 2013), achieving the desired learning outcome is possible if the participants are driven to play the game for intrinsic reasons. Thus, for the purpose of this study, engagement is defined by the desire for players to go through the game because they enjoy playing it (Apostol et al., 2013). Knowledge in this study is measured in a declarative form (ten Berge, T. and van Hezewijk, R., 1999). That is, as the ability to successfully apply the information acquired in the game in practical exercises, as well as showing an understanding of the concepts exposed.

 

Accordingly, the participants in this study were technical professionals employed by the information security company Trend Micro. The game offered a platform for the participants to understand the relative value, valid use cases and relative protection capacity of different products within the company portfolio.

Precedent and Going Forward

A similar initiative has been tried within Trend Micro before (Trend Micro, 2017), but in a more superficial manner that doesn’t explore as many products or as much depth in their technical scope and their place within an information system network. The existing game, offered to an external audience as well rather than being limited only to internal employees, offers an opportunity to test the player’s knowledge. However, due to the marketing focus of the application, the game is very limited in what it teaches. It is also very brief and narrow in scope to that of a single particular situation. The lower production costs of the game made for this study, which eschews the use of professional actors and live filming, permits the design of more content to represent more situations where knowledge of the Trend Micro suites can be tested. Likewise, it is easier to broaden the scope or update the game as required, for the same reasons stated.

While this rationale may seem grounded in a very specific training need context, serious games have been used effectively in multiple instances and subjects as an educational tool (Wang et al., 2016) and prior studies have observed a connection between game-based learning engagement and affective engagement with games (Ke et al., 2016). Thus, findings from this thesis can be posited to any other situation where there is a requirement of expanding knowledge horizontally onto other related topics with an audience that possesses specialized knowledge. This degree of specialization is very often seen in technology professionals (Malone et al., 2011), where only a few individuals at the top reach a truly holistic understanding of complete integrated systems rather than single solutions in a contextual void. This level of in-depth knowledge required by professionals was found to be a factor for that impacted the drive of technical personnel in Trend Micro to broaden their understanding of the larger portfolio and integration between products.

Rationale and Hypotheses

As the research question implied, this study tried to determine the extent of the success of the game as an educational tool. The European management team in Trend Micro has presently expressed interest in retaining the game as part of their educational curriculum for new staff, so it is the hope that as this study shows a clear and significant improvement in knowledge with high engagement and good feedback from participants, it will justify the continuation of the game’s development or the creation of similar applications to teach other similar topics. Further development of the game used in this study could be in the form of more scenarios or products added to the existing program. Other gamification elements could also be integrated into the game, such as leaderboards or achievements, which are often cited as behaviourist motivators that drive engagement (Raczkowski, 2013).

Based on this context, there were two hypotheses for this study. The first hypothesis is that it is predicted that the participants will be engaged while playing the game. The second, that it is predicted that the participants will demonstrate having developed knowledge in line with the learning objectives of the game. The findings from this study showed that both hypotheses were met, and a definite positive answer to the research question was found. This paper will explore the background and methodology employed throughout the study, as well as show and analyse the data gathered to support these findings.

A way to expand on these findings would be to look into the motivation shift in participants during a similar study, by using artefacts to measure motivation like the Intrinsic Motivation Inventory (IMI), which has been used on previous studies to measure this change (Ryan, 1982). While a positive shift in intrinsic motivation could have been used to form a third hypothesis, due to time limitations while conducting this study, only engagement and learning were measured and analysed.

Gameplay Implementation

The game is available to play online using a web browser, and upon beginning it presents the players with an introduction and tutorial on how to play. After that, a Trend Micro security product is introduced, exploring what the software does as well as how to use it in the representation that the game includes. The game allows the players to drag and drop icon representations of these products onto a map that matches a computer network topology, only allowing the product to be deployed in ways that are congruent with how the software is used in reality. There is an economy of resources in the game, in which deploying software uses budget and increases a protection score. The main objective of each level is to achieve a certain protection score using the allocated budget, as well as meet certain secondary goals (protect email traffic, for instance). Meeting these goals determine whether the player was successful. Once a successful solution to the level is provided, another product is introduced and the player is allowed to use it as part of the next level’s solution.

A version of this game is available in this location: https://datacenterdefense.000webhostapp.com

Research Methods

To see to what extent can learning be achieved by employing serious games to engage specialized technical professionals to acquire technical knowledge outside their main area of expertise three methods were employed to collect data:

  1. Knowledge test (pre-play and post-play)
  2. Participant feedback (Self-report questionnaire)
  3. Focus group discussions

The pre-play knowledge test was done over one week before the participants played the game, and it was done to assess the knowledge of the participants on the topic before they played. The post-play knowledge tests were done over one week after, to assess the knowledge of the participants after playing. The participant feedback questionnaire was done in parallel with the post-play knowledge test (participants received both questionnaires at the same time). The focus group discussions were done over a week after this. Both the participant feedback questionnaire and the focus groups discussions were conducted to assess whether the participants were engaged or not, as well as whether they saw a value for this game to be used as an educational tool. These data points led, after the analysis of results, to draw conclusions and test the study’s hypotheses.

Knowledge Tests

The knowledge test measured the participant’s general knowledge of what the purpose was for nine specific products from the Trend Micro portfolio. Since all participants were recruited from the technical staff of the company, they already possessed in-depth knowledge of one or more solutions, but cursory or non-existent knowledge of the rest.

 

This test was performed twice by the testing group, once before playing the game (pre-play) and again after playing the game (post-play). This allowed for a knowledge baseline to be established, and to compare it against the results from participants after they played the game.

 

The test presented the participants 10 questions with 4 answers each in a multiple selection format, from which only 1 answer was correct. Each correct answer was worth 1 point. Thus, the best possible score was 10 and the lowest 0. The same questions were asked pre and post-play, which may have impacted the validity of the results. However, this was deemed as the preferable alternative, rather than having a potential offset in difficulty from the pool of questions on both tests. A recommendation for a more reliable data gathering method for this stage would be to have a larger pool of questions from which a smaller subset are assigned randomly to each participant.

Participant Feedback

The self-report questionnaire sought to elicit data on the topics of engagement. For example, it was asked to the participants whether they were satisfied with the game and whether they found it engaging. Both the participant groups were given the same participant feedback questions, but the results were kept separate for analysis. The questions each had statements regarding their enjoyment of the game or how engaged were they with it and could be answered with yes or no, with an option for additional comments in a text field.

Some concerns on the validity of data obtained through this method regard to known problems of using self-report mechanisms (Austin et al., 1998). A more accurate measure of engagement would be through observation of actual gameplay. However, due to scheduling and regional constraints, this approach wasn’t possible during this study.

Focus Group Discussions

The focus group discussions were conducted after the study in informal conversations in small groups or on occasion as one-to-one, due to scheduling concerns and due to the different regional constraints from different participants. Some were held in person, in the Trend Micro office, while some were held via online chat. They focused on areas of concern and feedback that were raised in the participant feedback form comments. A focus group meeting was held prior to the beginning of the work on the program, to pitch the idea to a group of experts and managers within the company. During this event, the main gameplay elements were introduced and feedback was elicited from the attendees regarding the perceived value of such a product and which products and components would need to be included in the game.

Study Implementation

In the same context, the participants were sought among the Technical Support Department, Professional Services, Escalation Management and the Sales Engineering teams in Trend Micro. These are large groups of specialized professionals, composed of adult males predominantly (25-45 years old), from different cultural backgrounds and nationalities, located across different countries in Europe.

 

Using games to teach individuals in an Engineering context has been done successfully in other studies (Bodnar et al., 2015), setting some precedent to this study in regard of the knowledge context from the participants involved.

This study and the parallel creation of the game that was used was done in seven phases:

  1. Recruitment
  2. Grouping
  3. Pre-game testing
  4. Playing and Learning
  5. Post-game testing and feedback
  6. Focus group discussions
  7. Data analysis

 

The pre- and post-game knowledge tests and the participant feedback questionnaires were implemented using the platform provided by SurveyMonkey (https://www.surveymonkey.com). For the focus group discussions, when it wasn’t possible to meet in person, the Zoom communicator platform was used (https://zoom.us/).

Results:

Knowledge tests conducted after the study confirm that learning was acquired. An increase of 67% to 87% points average out of 100% points maximum (20% improvement) in the results across participants confirm that the core learning objectives of the game were transferred successfully to the players through gameplay. This is reinforced by the participant’s perception regarding the educational value of the game. On this, 20 out of 20 participants (100%) found the game informative, and thus able to educate staff on the basics of the products introduced. Likewise, there was a clear perceived value by both participant groups to use the game as an educational tool. Here 20 out of 20 participants (100%) thought it would be worthwhile to publish the game internally to offer it as an option for onboarding and initial learning of new employees. There was also a perceived worth on continuing to develop the game, with 19 out of 20 participants (95%) agreeing that it would be worth to expand the current game.

Thus, based on these results, the findings provide a positive answer to the stated research question and expand on the conclusions drawn from previous studies (Bodnar et al., 2015), showing through the feedback provided that to a great extent technical professionals can find serious games for learning to be engaging to play. This drive to play had the extent of offering clear and significant learning, showed in this study by a 20% knowledge score improvement between pre and post-game knowledge test results.

 

References:

Back to list of abstracts