We tend to view abstract thinking and language as ‘higher cognitive function’ and procedural, motor learning as, well, less higher. As abstract thinking and language seems to be the basis of advanced education, we value higher cognitive function over, say, skills at ping-pong. A tendency to liken video game skills more to ping-pong more than calculus may preclude recognizing their full potential as an educational tool. However, the distinction between higher cognitive function and ping-pong may not be as clear as we think.
Our brains have evolved to act: to respond to on-going stimuli, learn from experience and adapt to our environment in order to survive. Vast tracts of our brain are dedicated to putting sensory information together with potential actions, selecting and executing one response among many and subsequently learning from the outcome. Language and abstract thinking can be thought of as an extension of this action system, a mental tool. Though we usually feel like our actions arise after a thinking process– think before you act– in a sense, thinking itself is an action. Educators already treat thought as action; wrong answer = wrong thought. In many ways, we learn to think the same way we learn everything else– through trial and error and reinforcement. And much like motor learning, such as ping-pong, our thinking can become as automatic and habitual as our backhand. Again, this is precisely what education often tries to achieve– an automaticity in thought tools, such as how to solve integrals or recognizing chemical structures. In this sense, thought does not precede action, thought is action.
Video games offer the opportunity to map thought-as-action onto literal actions in a virtual world. Through this dual mapping, we facilitate abstract learning by yoking it to procedural, action learning to harness the millions of years of evolution that has designed brains, first and foremost, to act not think. In more psychological terms, we are dually mapping learning onto an explicit, declarative learning system and, simultaneously onto an implicit, procedural learning system allowing the two to strengthen and reinforce each other.
What sort of actions in a video game can map onto cognitive skills and functions? The key to understanding dual mapping is to separate the literal and the virtual, which corresponds to sensorimotor processing and cognitive representation, respectively. A video game, at its most fundamental requires the player take in sensory stimuli– ie., the screen– and respond with a motor output, pressing a key or moving the joystick. In Mario Bros, when a player ‘jumps’, they are really just pushing a button. The same with stabbing a dragon. And steering a race car around a curve– all these actions are literally simple motor movements of the hand and fingers that cause things on the screen to change which, through a combination of the programmers art and our cognitive apparatus, we come to think of as jumping, stabbing and steering. The same holds true for the sensory stimuli. Faced with the evil gate-keeping dragon, we cognitively perceive a dragon appearing in our path. But the dragon could be replaced with a giant lizard, or a big red block or an enormous, teeth baring beach ball and our required response– stab it!– could be identical. In the game, the characteristics of the sensory stimuli– like a dragon versus a beach ball– has nothing to do with the programmed requirement that the player tap the left button in response. All the jumping, stabbing and steering we do in response to all the cars, dragons and growling beach balls is not intrinsic to the mechanics of the game, but an overlaid cognitive layer that assigns meaning to both sensory stimuli and our motor responses. And these meanings can be anything. The opportunity, then, is to take this simple sensorimotor learning– which our brain is very good at– and to map it onto a virtual world of cognitive representation that teaches chemistry or math instead of dragons and beach balls.
For the moment, I am going to set aside the mapping and look at just the literal, sensorimotor part of the learning. It is in this stripped down view that the learning opportunities become more apparent. For discussion, I have listed some basic, or elemental types of actions in video games in the table below.
The table is not intended as a strict typology but just as some examples. By elemental, I refer to actions that do not involve any tools, like a sword or potions. I’ll address tool-based actions in a subsequent post. Even simple actions like moving require both a motor and cognitive component: figuring out where to move, when and how fast, or in what manner, such as walk or jump. Other basic actions include positioning and targeting, where the cognitive component can range from the simple, like positioning oneself under a falling banana, to the complex, like identifying the weak point in a configuration in Angry Birds. Sequencing also can range from simple motor sequences, such as learning to hop across a series of ledges in a particular order, to obviously cognitive things like learning the order in which a series of doors need to be opened in order to solve a puzzle.
All of these elemental actions engage a fundamental process of adapting to an environment. First the player has to learn which stimuli in the video game environment are actionable: which things can I interact with and cause something to happen? Second, the player has to learn the sort of responses can be applied to these different stimuli. Finally, the player has to learn the various consequences associated with each action applied to different stimuli. In doing this, the player shapes attention processes– what should I pay attention to? They begin to recognize patterns, discern cause and effect and make predictions– when this occurs, that is likely to happen. As their learning deepens, they begin to make discriminations– only the red dragon is dangerous, the green ones give gifts– as well as learn categories– grouping together stimuli that share properties in common. As the player acquires a deeper cognitive understanding of the video game environment and starts to learn the effects of their actions, they can begin to acquire sequences of behavior and through prediction, begin to plan series of actions, reflecting a well-developed mental representation of the virtual world. Finally, with this cognitive framework in place, they can begin to develop decision-making skills, assessing risk, uncertainty, and the relative cost-benefit ratio of different actions and choices.
We might be inclined to think of things like chemistry and calculus as representing ‘real’ cognitive learning while dismissing the cognitive component in video games as trivial. From the point of view of neural processing, however, I’m not convinced of the difference. In fact, one might make a case that the video game represents more complex, multi-dimensional learning. Before discussing how to leverage this learning to map onto things like chemistry and math, in the next post I will talk about tool-based actions in video games.
SynaptoGaming 