39
Intelligent textbooks Challenges, barriers, limitations While intelligent textbooks are a reality, there is much work to be done to bring them into the educational landscape. A major challenge is their integration in the current teaching practice in ways that are reflective of possible biases and meet existing learning objectives and curricula. For example, students should become aware of concerns related to textbooks such as the limited number of languages available, constraining the development of intelligent textbooks in under-resourced languages, and biases in the human data used to train models behind textbooks. An example is AI bias toward the writing of non-native speakers (NNS) of English. Because intelligent textbooks often focus on having readers generate knowledge, ideas produced by NNS may receive differential feedback from that of native speakers. That feedback may be biased or less accurate and provide lower scores to NNS. Furthermore, student privacy is an issue if intelligent textbooks rely on industrial LLMs like ChatGPT, where student data is processed and shared by commercial entities. Another concern is the amount of data needed to train LLMs to provide accurate feedback to users. As an example, the summary feedback models in iTELL were trained with over 15,000 summaries from around 50 source texts that had been hand-scored by experts. The constructed response model in iTELL was trained with over 25,000 questions and answers from around 450 sources. More specialised intelligent textbooks will require specific datasets to train models, which may be cost-prohibitive. Context-specific adaptation may also prove difficult in some areas. For instance, current LLMs are not efficient at processing maths word problems (maths problems described in words) or interpreting figures, graphs, and tables. This makes adapting a mathematics textbook into an intelligent framework difficult. remembering improves when ideas are generated by a person’s mind rather than simply reading about them
Another important issue is the cost of developing intelligent textbooks and their maintenance. These costs include for example, developing material (unless the material is open-source), constructing an interface, training AI models, and integrating content. The cost incurred can be substantial and will likely require support from commercial enterprises. This probably means that many intelligent textbook frameworks will be proprietary, making analysis of the systems difficult, including the reliability of their educational features and the validation of their AI interactions and learning potential. Conclusions The potential to make reading material interactive and enhance learning through the use of strategies known to increase text comprehension and skill development will lead many organisations to adopt intelligent textbooks. As the use of AI is changing how people work, intelligent textbooks that integrate AI will become important tools to support learning at school as well as learning beyond schooling. Learners will need to develop new skills – reskill or upskill – throughout their professional lives when they complete formal education. Intelligent textbooks can play a key role in making education and training more accessible, affordable, efficient, and adaptable.. 

Scene 1

39
Intelligent textbooks 
Challenges, barriers, limitations 
While intelligent textbooks are a reality, 
there is much work to be done to bring 
them into the educational landscape. A 
major challenge is their integration in the 
current teaching practice in ways that 
are reflective of possible biases and meet 
existing learning objectives and curricula. 
For example, students should become aware 
of concerns related to textbooks such as 
the limited number of languages available, 
constraining the development of intelligent 
textbooks in under-resourced languages, 
and biases in the human data used to train 
models behind textbooks. An example is 
AI bias toward the writing of non-native 
speakers (NNS) of English. Because intelligent 
textbooks often focus on having readers 
generate knowledge, ideas produced by 
NNS may receive differential feedback from 
that of native speakers. That feedback may 
be biased or less accurate and provide 
lower scores to NNS. Furthermore, student 
privacy is an issue if intelligent textbooks 
rely on industrial LLMs like ChatGPT, where 
student data is processed and shared by 
commercial entities. 
Another concern is the amount of data 
needed to train LLMs to provide accurate 
feedback to users. As an example, the 
summary feedback models in iTELL were 
trained with over 15,000 summaries from 
around 50 source texts that had been 
hand-scored by experts. The constructed 
response model in iTELL was trained with over 
25,000 questions and answers from around 
450 sources. More specialised intelligent 
textbooks will require specific datasets to 
train models, which may be cost-prohibitive. 
Context-specific adaptation may also prove 
difficult in some areas. For instance, current 
LLMs are not efficient at processing maths 
word problems (maths problems described 
in words) or interpreting figures, graphs, and 
tables. This makes adapting a mathematics 
textbook into an intelligent framework 
difficult. 
remembering improves when ideas are generated by  
a person’s mind rather than simply reading about them
Another important issue is the cost of 
developing intelligent textbooks and their 
maintenance. These costs include for 
example, developing material (unless the 
material is open-source), constructing 
an interface, training AI models, and 
integrating content. The cost incurred can 
be substantial and will likely require support 
from commercial enterprises. This probably 
means that many intelligent textbook 
frameworks will be proprietary, making 
analysis of the systems difficult, including 
the reliability of their educational features 
and the validation of their AI interactions and 
learning potential. 
Conclusions 
The potential to make reading material 
interactive and enhance learning through 
the use of strategies known to increase text 
comprehension and skill development will 
lead many organisations to adopt intelligent 
textbooks. As the use of AI is changing 
how people work, intelligent textbooks that 
integrate AI will become important tools 
to support learning at school as well as 
learning beyond schooling. Learners will 
need to develop new skills – reskill or upskill 
– throughout their professional lives when 
they complete formal education. Intelligent 
textbooks can play a key role in making 
education and training more accessible, 
affordable, efficient, and adaptable.

40
Innovating Pedagogy 2024 References
1. A short description of the history of intelligent books in the AI magazine: Brusilovsky, P., Sosnovsky, S., and Thaker, K. (2022) ‘The Return of Intelligent Textbooks’, AI Magazine, 43, pp. 337–340. Available at: https://doi.
org/10.1002/aaai.12061 (Accessed 27 June 2024). 2. A paper presenting the potential for interactivity in digital texts:
Clinton-Lisell, V., Seipel, B., Gilpin, S., and Litzinger, C. (2021) ‘Interactive features of E-texts’ effects on learning: a systematic review and meta-analysis’, Interactive Learning Environments, pp. 1–16. 3. An overview of the effect COVID had on the adoption of digital texts from a consultancy firm: Seaman, J. E., and Seaman, J. (2020) ‘Digital Texts in the Time of COVID: Educational Resources in U.S. Higher Education’, Bay View Analytics. Available at: https://eric.ed.gov/?q=higher+education+chang
es+in+the+u.s.&ff1=dtysince_2019&id=ED616838 (Accessed 27 June 2024).
4. A study introducing generation effects: Slamecka, N. J., and Graf, P. (1978) ‘The generation effect: Delineation of a phenomenon’, Journal of Experimental Psychology: Human Learning and Memory, 4(6), pp. 592–604. Available at: https://doi.org/10.1037/0278-7393.4.6.592 (Accessed 27 June 2024).
5. A study discussing generation effects in learning: Chen, O., Kalyuga, S., and Sweller, J. (2015) ‘The worked example effect, the generation effect, and element interactivity’, Journal of Educational Psychology, 107(3), pp. 689–704. Available at: https://doi.org/10.1037/edu0000018 (Accessed 27 June 2024).
6. A paper that compares generation to simple reading: McCurdy, M. P., Viechtbauer, W., Sklenar, A. M., Frankenstein, A. N., and Leshikar, E. D. (2020) ‘Theories of the generation effect and the impact of generation constraint: A meta-analytic review’, Psychonomic Bulletin & Review, 27(6), pp. 1139–1165. Available at: https://doi.org/10.3758/s13423-020-
01762-3 (Accessed 27 June 2024). 7. An article introducing iTELL and summarisation interactivity: Morris, W., Crossley, S., Holmes, L., Ou, Chaohua, Dascalu, M., and McNamara, D. (2024) ‘Formative Feedback on Student-Authored Summaries in Intelligent Textbooks using Large Language Models’, Journal of Artificial Intelligence in Education. Available at: https://doi.org/10.1007/s40593-024-
00395-0 (Accessed 27 June 2024).
8. A case study of using intelligent textbooks with university students in the US: Sun, Q., Norman, T. J., and Abdourazakou, Y. (2018) ‘Perceived value of interactive digital textbook and adaptive learning: Implications on student learning effectiveness’, Journal of Education for Business, 93(7), pp. 323–331.
9. A case study examining student perspectives using an intelligent textbook in the US: Morris, W., Choi, J., Holmes, L., Gupta, V., and Crossley, S. A. (in press). Automatic Question Generation and Constructed Response Scoring in Intelligent Texts. Proceedings of the 17th International Conference on Educational Data Mining (EDM). Atlanta, GA.
10. An experimental study comparing student achievements when using an intelligent textbook: Crossley, S. A., Choi, J. S., Morris, W., Holmes, L., Joyner, D. and Gupta, V. (in press). Using Intelligent Texts in A Computer Science Classroom: Findings from an iTELL Deployment. Proceedings of the 17th International Conference on Educational Data Mining (EDM). Atlanta, GA.
Resources • A short video about iTELL: Intelligent Textbooks for Enhanced Lifelong Learning produced by the AI institute for adult learning and online education: iTELL Demo. Available at: https://www.youtube.com/
watch?v=YZXVQjSDZtI (Accessed 27 June 2024).
• A short video of an example of how intelligent textbooks have been developed and used at the Rice University in Houston, Texas: Revolutionizing learning with free “intelligent” textbooks. Available at: https://www.youtube.com/
watch?v=XqTiS8VmYQg (Accessed 27 June 2024).. 

Scene 2

40
Innovating Pedagogy 2024
 References
1. A short description of the history of intelligent 
books in the AI magazine:  
Brusilovsky, P., Sosnovsky, S., and Thaker, K. (2022) 
‘The Return of Intelligent Textbooks’, AI Magazine, 
43, pp. 337–340. Available at: https://doi.
org/10.1002/aaai.12061 (Accessed 27 June 2024). 
2. A paper presenting the potential for interactivity in 
digital texts:
Clinton-Lisell, V., Seipel, B., Gilpin, S., and Litzinger, 
C. (2021) ‘Interactive features of E-texts’ effects on 
learning: a systematic review and meta-analysis’, 
Interactive Learning Environments, pp. 1–16. 
3. An overview of the effect COVID had on the 
adoption of digital texts from a consultancy firm: 
Seaman, J. E., and Seaman, J. (2020) ‘Digital Texts 
in the Time of COVID: Educational Resources in U.S. 
Higher Education’, Bay View Analytics. Available at: 
https://eric.ed.gov/?q=higher+education+chang
es+in+the+u.s.&ff1=dtysince_2019&id=ED616838 
(Accessed 27 June 2024).
4. A study introducing generation effects: 
Slamecka, N. J., and Graf, P. (1978) ‘The generation 
effect: Delineation of a phenomenon’, Journal of 
Experimental Psychology: Human Learning and 
Memory, 4(6), pp. 592–604. Available at:  
https://doi.org/10.1037/0278-7393.4.6.592 
(Accessed 27 June 2024).
5. A study discussing generation effects in learning: 
Chen, O., Kalyuga, S., and Sweller, J. (2015) ‘The 
worked example effect, the generation effect, 
and element interactivity’, Journal of Educational 
Psychology, 107(3), pp. 689–704. Available at: 
https://doi.org/10.1037/edu0000018 (Accessed 27 
June 2024).
6. A paper that compares generation to simple reading: 
McCurdy, M. P., Viechtbauer, W., Sklenar, A. M., 
Frankenstein, A. N., and Leshikar, E. D. (2020) 
‘Theories of the generation effect and the impact 
of generation constraint: A meta-analytic review’, 
Psychonomic Bulletin & Review, 27(6), pp. 1139–1165. 
Available at: https://doi.org/10.3758/s13423-020-
01762-3 (Accessed 27 June 2024). 
7. An article introducing iTELL and summarisation 
interactivity: 
Morris, W., Crossley, S., Holmes, L., Ou, Chaohua, 
Dascalu, M., and McNamara, D. (2024) ‘Formative 
Feedback on Student-Authored Summaries in 
Intelligent Textbooks using Large Language Models’, 
Journal of Artificial Intelligence in Education. 
Available at: https://doi.org/10.1007/s40593-024-
00395-0 (Accessed 27 June 2024).
8. A case study of using intelligent textbooks with 
university students in the US: 
Sun, Q., Norman, T. J., and Abdourazakou, Y. (2018) 
‘Perceived value of interactive digital textbook and 
adaptive learning: Implications on student learning 
effectiveness’, Journal of Education for Business, 
93(7), pp. 323–331.
9. A case study examining student perspectives using 
an intelligent textbook in the US: 
Morris, W., Choi, J., Holmes, L., Gupta, V., and Crossley, 
S. A. (in press). Automatic Question Generation and 
Constructed Response Scoring in Intelligent Texts. 
Proceedings of the 17th International Conference on 
Educational Data Mining (EDM). Atlanta, GA.
10. An experimental study comparing student 
achievements when using an intelligent textbook: 
Crossley, S. A., Choi, J. S., Morris, W., Holmes, L., Joyner, 
D. and Gupta, V. (in press). Using Intelligent Texts in A 
Computer Science Classroom: Findings from an iTELL 
Deployment. Proceedings of the 17th International 
Conference on Educational Data Mining (EDM). 
Atlanta, GA.
Resources 
• 
A short video about iTELL: Intelligent Textbooks for 
Enhanced Lifelong Learning produced by the AI 
institute for adult learning and online education: 
iTELL Demo. Available at: https://www.youtube.com/
watch?v=YZXVQjSDZtI (Accessed 27 June 2024).
• 
A short video of an example of how intelligent 
textbooks have been developed and used at the 
Rice University in Houston, Texas: 
Revolutionizing learning with free “intelligent” 
textbooks. Available at: https://www.youtube.com/
watch?v=XqTiS8VmYQg (Accessed 27 June 2024).

41
Assessments through extended reality Assessments through extended reality Harnessing immersion to demonstrate and develop skills
Introduction Simulation-based learning replicates aspects of the real world, requiring learners to take actions and making the consequences of these actions visible. This can allow practical skills to be developed and demonstrated without real-world constraints or risks. Simulation is already an important part of training in areas such as healthcare and aviation. Extended reality (XR) – encompassing techniques such as virtual reality (VR) and augmented reality (AR) – increases opportunities for immersion, which is understood as the combination of an increased sense of presence (the feeling of ‘being there’) and of agency (having control over our actions and being able to manipulate the environment). In diverse industries, companies such as Hilton (hospitality), DHL (logistics), Volkswagen (automobile manufacture) and Rolls Royce (aerospace) are making use of XR simulations as part of workplace training. As greater immersion enhances the sense of agency, immersive simulations are particularly suited to testing and gaining feedback on procedural knowledge – how to do a particular task well, such as flying a plane, responding to an emergency or performing surgery.1
Assessment is key to learning, and assessing procedural competences is a core goal of many learning activities. It is increasingly recognised that XR could fulfil the growing desire for authentic assessments that test what someone can do and how well they can do it. This offers a radical alternative to traditional written assignments2. It can address difficulties with other ways of assessing performance in the real world, such as a lack of repeatability or availability of a particular scenario on demand, or the need for a specific physical environment, equipment or people, which are all limited resources. Extended reality creates new forms of interactive activities which generate rich data on procedural knowledge and learners’ abilities. 

Scene 3

41
Assessments through extended reality 
Assessments through extended reality 
Harnessing immersion to demonstrate and develop skills
Introduction 
Simulation-based learning replicates 
aspects of the real world, requiring 
learners to take actions and making the 
consequences of these actions visible. This 
can allow practical skills to be developed 
and demonstrated without real-world 
constraints or risks. Simulation is already an 
important part of training in areas such as 
healthcare and aviation. 
Extended reality (XR) – encompassing 
techniques such as virtual reality (VR) 
and augmented reality (AR) – increases 
opportunities for immersion, which is 
understood as the combination of an 
increased sense of presence (the feeling 
of ‘being there’) and of agency (having 
control over our actions and being able to 
manipulate the environment). In diverse 
industries, companies such as Hilton 
(hospitality), DHL (logistics), Volkswagen 
(automobile manufacture) and Rolls 
Royce (aerospace) are making use of XR 
simulations as part of workplace training. 
As greater immersion enhances the sense 
of agency, immersive simulations are 
particularly suited to testing and gaining 
feedback on procedural knowledge – how 
to do a particular task well, such as flying 
a plane, responding to an emergency or 
performing surgery.1
Assessment is key to learning, and assessing 
procedural competences is a core goal of 
many learning activities. It is increasingly 
recognised that XR could fulfil the growing 
desire for authentic assessments that test 
what someone can do and how well they 
can do it. This offers a radical alternative 
to traditional written assignments2. It can 
address difficulties with other ways of 
assessing performance in the real world, 
such as a lack of repeatability or availability 
of a particular scenario on demand, or the 
need for a specific physical environment, 
equipment or people, which are all limited 
resources. 
Extended reality creates new forms of interactive activities which generate rich data on procedural 
knowledge and learners’ abilities

42
Innovating Pedagogy 2024
Simulation provides important potential for assessment of competences where there may be serious consequences for real-world failure, or situations which are difficult to find or create for the purposes of assessment. In a simulation, formative and summative assessment can occur without these consequences, but emphasising the related learning points, testing skill levels and decision-making abilities.
The sense of presence that comes from increased immersion can allow assessment of whether someone can perform in a facsimile of a real situation, where environmental or affective elements, such as navigating a complex physical space, time pressure, or managing interpersonal interactions can be made more realistic. For example, activities to assess and develop soft skills in hospitality can be undertaken in a realistic virtual simulation of the actual hotel environment, complete with guests that present challenging situations which need to be responded to by working with colleagues. Where is this useful?
Examples of assessments through XR show reasons to use immersive approaches and provide insights into appropriate design. Medical teaching has been an early adopter of VR and AR, with demonstrations of this being used to assess proficiency and differentiate levels of skill in some procedures. Researchers have shown for example that VR can be used to distinguish novice and expert skill in neurosurgeons by capturing data on how they complete complex motor skills-based tasks in a virtual environment3.
There has been substantial interest in VR-based training for health and safety. It is recognised, for example, that many construction workers are not able to identify and manage hazards, with potentially serious consequences. VR offers the opportunity to practise and assess these skills away from the dangers of a real-world construction site but with the benefits of feeling as if you are present in a real setting3. Simulations can capture data on whether a person can correctly identify hazards in a simulacrum (representation) of a workplace, such as a chemical laboratory or factory, and if the person can act appropriately to resolve or avoid these hazards. Other examples involve capturing and assessing data on the operation of complicated and potentially dangerous machinery, such as cranes or lathes4. A whole range of further vocational subjects, where a simulation of the work environment can be produced including relevant objects, actors and decision points, offer opportunities for students to be assessed on their ability to respond to situations. The Sheffield College in the UK chose Animal Care, Catering, and Carpentry and Joinery courses as subjects in which to trial VR-based assessments. In each case, a virtual resource was built based around an actual environment using 360 degree photos. For example, in the animal care resource, students virtually navigate around the room at the college that contains animal enclosures and a walk-in fridge area, interact with enclosures and other objects, and answer questions to test their knowledge5.
assessing procedural competences is a core goal of many learning activities
By reducing reliance on reading and writing, some assessments in XR can also be designed to be completed regardless of the language proficiency of the student. The Augmented Assessment project (see Resources) is exploring how these assessments can be more inclusive of migrants studying in schools when compared to assessments with written questions and answers. For example, the students’ mathematical or scientific understanding can be demonstrated through interaction with virtual objects.
Designing assessments with XR While further research and development is needed, some initial principles for the design of assessment using XR could include:
• Use multiple assessment methods and data sources to understand different dimensions of learning and performance: Many forms of analytical data can be identified, logged and analysed to assess skills and procedural knowledge through. 

Scene 4

42
Innovating Pedagogy 2024
Simulation provides important potential 
for assessment of competences where 
there may be serious consequences for 
real-world failure, or situations which are 
difficult to find or create for the purposes of 
assessment. In a simulation, formative and 
summative assessment can occur without 
these consequences, but emphasising the 
related learning points, testing skill levels and 
decision-making abilities.
The sense of presence that comes from 
increased immersion can allow assessment 
of whether someone can perform in 
a facsimile of a real situation, where 
environmental or affective elements, such 
as navigating a complex physical space, 
time pressure, or managing interpersonal 
interactions can be made more realistic. For 
example, activities to assess and develop 
soft skills in hospitality can be undertaken 
in a realistic virtual simulation of the actual 
hotel environment, complete with guests that 
present challenging situations which need to 
be responded to by working with colleagues. 
Where is this useful?
Examples of assessments through XR show 
reasons to use immersive approaches 
and provide insights into appropriate 
design. Medical teaching has been an early 
adopter of VR and AR, with demonstrations 
of this being used to assess proficiency 
and differentiate levels of skill in some 
procedures. Researchers have shown for 
example that VR can be used to distinguish 
novice and expert skill in neurosurgeons 
by capturing data on how they complete 
complex motor skills-based tasks in a virtual 
environment3.
There has been substantial interest in 
VR-based training for health and safety. 
It is recognised, for example, that many 
construction workers are not able to identify 
and manage hazards, with potentially 
serious consequences. VR offers the 
opportunity to practise and assess these 
skills away from the dangers of a real-world 
construction site but with the benefits of 
feeling as if you are present in a real setting3. 
Simulations can capture data on whether 
a person can correctly identify hazards in a 
simulacrum (representation) of a workplace, 
such as a chemical laboratory or factory, 
and if the person can act appropriately  
to resolve or avoid these hazards.  
Other examples involve capturing and 
assessing data on the operation of 
complicated and potentially dangerous 
machinery, such as cranes or lathes4. 
A whole range of further vocational 
subjects, where a simulation of the work 
environment can be produced including 
relevant objects, actors and decision 
points, offer opportunities for students to 
be assessed on their ability to respond to 
situations. The Sheffield College in the UK 
chose Animal Care, Catering, and Carpentry 
and Joinery courses as subjects in which 
to trial VR-based assessments. In each 
case, a virtual resource was built based 
around an actual environment using 360 
degree photos. For example, in the animal 
care resource, students virtually navigate 
around the room at the college that contains 
animal enclosures and a walk-in fridge 
area, interact with enclosures and other 
objects, and answer questions to test their 
knowledge5.
assessing procedural competences is 
a core goal of many learning activities
By reducing reliance on reading and 
writing, some assessments in XR can also 
be designed to be completed regardless 
of the language proficiency of the student. 
The Augmented Assessment project 
(see Resources) is exploring how these 
assessments can be more inclusive 
of migrants studying in schools when 
compared to assessments with written 
questions and answers. For example, 
the students’ mathematical or scientific 
understanding can be demonstrated 
through interaction with virtual objects.
Designing assessments with XR 
While further research and development is 
needed, some initial principles for the design 
of assessment using XR could include:
• Use multiple assessment methods and 
data sources to understand different 
dimensions of learning and performance: 
Many forms of analytical data can be 
identified, logged and analysed to assess 
skills and procedural knowledge through

43
Assessments through extended reality activities in XR. Features such as the time taken, amount of work completed successfully, or adherence to modelled behaviours can be logged and analysed. Decisions made during the simulation, and how the person communicates with others, might also be captured and assessed. This can be combined with observations or review by assessors to provide richer understanding and reflection. External to the data collected through the simulation, quizzes can test understanding and post-simulation activities can encourage reflection and identify ways to improve. It can also be valuable to test knowledge at a later date, well after the initial simulation activity, to check for retention4. Through a combination of these data sources the assessment can offer rich understanding that can be hard to gather through other means.
• Start with formative or low stakes assessments: Learners have expressed more positive views of VR in low stakes or formative assessment, where the novelty and challenge supports new forms of feedback but cannot impact negatively on their grades5. If the assessment is summative and carries weight for passing the course, then it will be particularly important to offer inductions and chances to develop experience in advance3.
• Align the assessment purpose and measures: Research suggests that using existing VR-based games to assess competencies such as emotional intelligence or working at heights is problematic7. In any novel assessment it is important to think carefully about the alignment of the assessment tasks and measures with expected learning outcomes.
Barriers and potential solutions for equitable assessment VR experiences using headsets and other hardware can create a strong sense of immersion, but they raise barriers for some people and pose risks for new inequities to emerge when using them in assessment:
• People who are more susceptible to motion sickness and with less experience of VR are more likely to suffer from ‘VR sickness’, which could result in reduced performance or aversion to taking part. However, VR sickness occurs when motions in the virtual space do not match physical movement, such as moving over a large virtual distance by pressing a joystick direction6. Activities in smaller spaces where the user can move naturally are less likely to cause symptoms.
• Some report other forms of discomfort such as eye strain or headaches from extended use of headsets1. Ensuring proper fitting, comfortable equipment, and limiting the length of time a person is expected to wear a headset is therefore recommended.
• The costs of headsets have reduced substantially in recent years with models available for a similar price to a smartphones or laptop. But as an additional device, cost is still a potential barrier and only a small minority own their own headset.
• A minimum clear space of 2 by 2 metres is suggested for each user. Limited space may reduce performance and create risks where people are unaware of their surroundings. Standard computer or mobile XR experiences also exist, so in some cases it may not be necessary to use specialist hardware to take part in an XR-based assessment. These could also be offered as an alternative assessment method where barriers do exist.
Conclusions
Assessments through XR can address the challenge of achieving more authentic assessments by simulating situations that are difficult to achieve reliably or safely in the real world. It is attractive to think that assessments can be based on detailed analytical data and allow in-depth analysis, reflection, and potentially repetition. There remain uncertainties about the extent to which headset-based VR will become mainstream and can support equitable assessment, but where VR is available for learning it will become increasingly important to think about how to embed assessment effectively.. 

Scene 5

43
Assessments through extended reality 
activities in XR. Features such as the 
time taken, amount of work completed 
successfully, or adherence to modelled 
behaviours can be logged and analysed. 
Decisions made during the simulation, 
and how the person communicates 
with others, might also be captured and 
assessed. This can be combined with 
observations or review by assessors 
to provide richer understanding and 
reflection. External to the data collected 
through the simulation, quizzes can test 
understanding and post-simulation 
activities can encourage reflection and 
identify ways to improve. It can also be 
valuable to test knowledge at a later 
date, well after the initial simulation 
activity, to check for retention4. Through 
a combination of these data sources the 
assessment can offer rich understanding 
that can be hard to gather through  
other means.
• Start with formative or low stakes 
assessments: Learners have expressed 
more positive views of VR in low stakes or 
formative assessment, where the novelty 
and challenge supports new forms of 
feedback but cannot impact negatively 
on their grades5. If the assessment is 
summative and carries weight for passing 
the course, then it will be particularly 
important to offer inductions and chances 
to develop experience in advance3.
• Align the assessment purpose and 
measures: Research suggests that 
using existing VR-based games to 
assess competencies such as emotional 
intelligence or working at heights is 
problematic7. In any novel assessment 
it is important to think carefully about 
the alignment of the assessment tasks 
and measures with expected learning 
outcomes.
Barriers and potential solutions 
for equitable assessment 
VR experiences using headsets and other 
hardware can create a strong sense of 
immersion, but they raise barriers for some 
people and pose risks for new inequities to 
emerge when using them in assessment:
• People who are more susceptible to 
motion sickness and with less experience 
of VR are more likely to suffer from ‘VR 
sickness’, which could result in reduced 
performance or aversion to taking part. 
However, VR sickness occurs when 
motions in the virtual space do not match 
physical movement, such as moving 
over a large virtual distance by pressing 
a joystick direction6. Activities in smaller 
spaces where the user can move naturally 
are less likely to cause symptoms.
• Some report other forms of discomfort 
such as eye strain or headaches from 
extended use of headsets1. Ensuring 
proper fitting, comfortable equipment, 
and limiting the length of time a person is 
expected to wear a headset is therefore 
recommended.
• The costs of headsets have reduced 
substantially in recent years with 
models available for a similar price to 
a smartphones or laptop. But as an 
additional device, cost is still a potential 
barrier and only a small minority own their 
own headset.
• A minimum clear space of 2 by 2 metres 
is suggested for each user. Limited space 
may reduce performance and create 
risks where people are unaware of their 
surroundings. 
Standard computer or mobile XR experiences 
also exist, so in some cases it may not be 
necessary to use specialist hardware to 
take part in an XR-based assessment. These 
could also be offered as an alternative 
assessment method where barriers do exist.
Conclusions
Assessments through XR can address the 
challenge of achieving more authentic 
assessments by simulating situations that 
are difficult to achieve reliably or safely in 
the real world. It is attractive to think that 
assessments can be based on detailed 
analytical data and allow in-depth analysis, 
reflection, and potentially repetition. There 
remain uncertainties about the extent 
to which headset-based VR will become 
mainstream and can support equitable 
assessment, but where VR is available 
for learning it will become increasingly 
important to think about how to embed 
assessment effectively.

44
Innovating Pedagogy 2024
References
1. A widely cited framework for immersive learning:
Makransky, G., and Petersen, G. B. (2021) ‘The cognitive affective model of immersive learning (CAMIL): A theoretical research-based model of learning in immersive virtual reality’, Educational Psychology Review, 33(3), pp. 937–958.
2. A paper exploring frameworks for assessments in VR and AR, along with an evaluation of an example in laboratory safety which uses multiple types of data to assess students: Udeozor, C., Chan, P., Russo Abegão, F. and Glassey, J. (2023) ‘Game-based assessment framework for virtual reality, augmented reality and digital game-based learning’, International Journal of Educational Technology in Higher Education, 20(1), pp. 36.
3. A review of research on VR training across subject areas: Abich IV, J., Parker, J., Murphy, J.S. and Eudy, M. (2021) ‘A review of the evidence for training effectiveness with virtual reality technology’, Virtual Reality, 25(4), pp. 919–933.
4. A review of research on VR training in health and safety applications which identifies different ways in which learning was assessed: Toyoda, R., Russo-Abegão, F., and Glassey, J. (2022) ‘VR-based health and safety training in various high-risk engineering industries: a literature review’, International Journal of Educational Technology in Higher Education, 19(1), pp. 42.
5. Report on pilot studies of VR in summative assessments in vocational learning: The Sheffield College, NCFE. (2023) An investigation into the use of virtual reality on assessment. Available at: https://www.ncfe.org.
uk/media/nq3afg0n/final-report-the-sheffield-
college.pdf (Accessed 24 June 2024).
6. Paper reviewing research on the causes of VR sickness: Chang, E., Kim, H.T. and Yoo, B. (2020) ‘Virtual reality sickness: a review of causes and measurements’, International Journal of Human–
Computer Interaction, 36(17), pp. 1658–1682.
7. Study exploring the use of VR games to assess competencies with mixed results, suggesting next steps for valid XR assessments: Sanchez, D.R., Weiner, E. and Van Zelderen, A. (2022) ‘Virtual reality assessments (VRAs): exploring the reliability and validity of evaluations in VR’, International Journal of Selection and Assessment, 30(1), pp. 103–125.
Resources • Summary of current trends around virtual reality and the metaverse for training employees: What does virtual reality and the metaverse mean for training? Blog post by Scott Likens and Andrea Mower from PwC.
Available at: https://www.pwc.com/us/en/tech-
effect/emerging-tech/virtual-reality-study.html (Accessed 24 June 2024).
• A list of examples of how companies use virtual reality in training, describing features such as how learning is assessed and feedback given: Available at: https://www.vrowl.io/the-22-best-
examples-of-how-companies-use-virtual-
reality-for-training/ (Accessed 24 June 2024).
• Augmented Assessment: Resources from a European Union funded project exploring how augmented reality assessments could be beneficial for migrants: Available at: https://augmented-assessment.eu/ (Accessed 24 June 2024).
• Drawing on a UK-based survey with responses from across 110 educational institutions, Jisc’s Extended reality in learning and teaching report 2023/24 describes the growth in applications of XR in tertiary education but also outlines current challenges experienced by educators: Available at: https://repository.jisc.ac.uk/9534/1/
extended-reality-in-learning-and-teaching-
report-2023-24.pdf (Accessed 24 June 2024).
• A set of standards and guidance from the Association for Simulated Practice in Healthcare (ASPiH), guided by evidence on simulation-based learning and emphasising the appropriate use of assessment within this: Available at: https://aspih.org.uk/standards-2/ (Accessed 24 June 2024).. 

Scene 6

44
Innovating Pedagogy 2024
References
1. A widely cited framework for immersive learning:
Makransky, G., and Petersen, G. B. (2021) ‘The 
cognitive affective model of immersive learning 
(CAMIL): A theoretical research-based model of 
learning in immersive virtual reality’, Educational 
Psychology Review, 33(3), pp. 937–958.
2. A paper exploring frameworks for assessments in 
VR and AR, along with an evaluation of an example 
in laboratory safety which uses multiple types of 
data to assess students: 
Udeozor, C., Chan, P., Russo Abegão, F. and Glassey, 
J. (2023) ‘Game-based assessment framework 
for virtual reality, augmented reality and digital 
game-based learning’, International Journal of 
Educational Technology in Higher Education, 20(1), 
pp. 36.
3. A review of research on VR training across subject 
areas: 
Abich IV, J., Parker, J., Murphy, J.S. and Eudy, M. 
(2021) ‘A review of the evidence for training 
effectiveness with virtual reality technology’,  
Virtual Reality, 25(4), pp. 919–933.
4. A review of research on VR training in health and 
safety applications which identifies different ways 
in which learning was assessed: 
Toyoda, R., Russo-Abegão, F., and Glassey, J. (2022) 
‘VR-based health and safety training in various 
high-risk engineering industries: a literature 
review’, International Journal of Educational 
Technology in Higher Education, 19(1), pp. 42.
5. Report on pilot studies of VR in summative 
assessments in vocational learning: 
The Sheffield College, NCFE. (2023) An 
investigation into the use of virtual reality on 
assessment. Available at: https://www.ncfe.org.
uk/media/nq3afg0n/final-report-the-sheffield-
college.pdf (Accessed 24 June 2024).
6. Paper reviewing research on the causes of VR 
sickness: 
Chang, E., Kim, H.T. and Yoo, B. (2020) ‘Virtual 
reality sickness: a review of causes and 
measurements’, International Journal of Human–
Computer Interaction, 36(17), pp. 1658–1682.
7. Study exploring the use of VR games to assess 
competencies with mixed results, suggesting next 
steps for valid XR assessments: 
Sanchez, D.R., Weiner, E. and Van Zelderen, A. 
(2022) ‘Virtual reality assessments (VRAs): 
exploring the reliability and validity of evaluations 
in VR’, International Journal of Selection and 
Assessment, 30(1), pp. 103–125.
Resources 
• 
Summary of current trends around virtual reality 
and the metaverse for training employees: 
What does virtual reality and the metaverse mean 
for training? Blog post by Scott Likens and Andrea 
Mower from PwC.
Available at: https://www.pwc.com/us/en/tech-
effect/emerging-tech/virtual-reality-study.html 
(Accessed 24 June 2024).
• 
A list of examples of how companies use virtual 
reality in training, describing features such as how 
learning is assessed and feedback given: 
Available at: https://www.vrowl.io/the-22-best-
examples-of-how-companies-use-virtual-
reality-for-training/ (Accessed 24 June 2024).
• 
Augmented Assessment: Resources from a 
European Union funded project exploring how 
augmented reality assessments could be 
beneficial for migrants: 
Available at: https://augmented-assessment.eu/ 
(Accessed 24 June 2024).
• 
Drawing on a UK-based survey with responses 
from across 110 educational institutions, Jisc’s 
Extended reality in learning and teaching report 
2023/24 describes the growth in applications of 
XR in tertiary education but also outlines current 
challenges experienced by educators: 
Available at: https://repository.jisc.ac.uk/9534/1/
extended-reality-in-learning-and-teaching-
report-2023-24.pdf (Accessed 24 June 2024).
• 
A set of standards and guidance from the 
Association for Simulated Practice in Healthcare 
(ASPiH), guided by evidence on simulation-based 
learning and emphasising the appropriate use of 
assessment within this: 
Available at: https://aspih.org.uk/standards-2/ 
(Accessed 24 June 2024).

45
Immersive language and culture Immersive language and culture Using games to step back in time for authentic learning experiences
Introduction Students often benefit from opportunities to develop and apply their learning in authentic settings. Language students can progress quickly when they are immersed in a setting where everyone is using their target language. Field trips can also support lessons about historical periods and diverse cultures. However, providing authentic experience of an ancient language or culture is challenging. Immersive language and culture provides game-based opportunities for extended investigation of and roleplay within historic scenarios. This can be done through video-game-based experiences or through structured sessions of live-action role play.
Authentic learning
Immersive language and culture draws on the strengths of authentic learning. This is an approach to learning-by-doing that prompts students to find solutions to complex problems in multi-disciplinary settings. Students who are immersed in an authentic learning activity can be supported to develop the judgment to distinguish between reliable and unreliable information, the ability to recognise patterns in unfamiliar contexts, the flexibility to develop innovative solutions and to work across cultural boundaries1. Some features of authentic learning include sustained investigations, the challenge of drawing on multiple sources and perspectives, and opportunities to seek out multiple interpretations and work towards a variety of outcomes. This can be achieved in real-world settings but, in the case of past languages and cultures, students can benefit from the use of digital immersive video game environments, or an approach known as Reacting to the Past. Video games and immersive learning
As access to digital technology has become more widespread, virtual language learning has become increasingly popular. Propelled by the need to teach in online environments due to COVID lockdowns, virtual learning has become increasingly mainstream2. Language-learning software and apps, from Rosetta Stone and Duolingo to the software included with textbooks, have become an increasingly common part of learning a language. Many language-learning programs do not attempt to provide an authentic cultural environment, but function more like flashcards for the learner. The use of commercial entertainment industry video games to learn languages has provided an informal way (now sometimes formalised in language classrooms) of connecting sensory perceptions with unfamiliar languages, and these opportunities can be provided simply by changing the language in which the commercial game is installed. However, this technique is not possible for ‘dead’ and endangered languages such as Latin or medieval French. Furthermore, to achieve the goal of language learning within a cultural context, the immersive environment must mirror the world of the target language and culture. For example, a modern kitchen table in a US-based game set in the 21st century would not be an appropriate table to include in a medieval court environment. Likewise, the vocabulary used in an off-the-shelf video game like Assassin’s Creed applies only to a specific set of conquest and conflict situations, leaving many gaps for those who will be working with ancient and medieval literature – for example, vocabulary relating to rich religious experiences and interpersonal relationships.. 

Scene 7

45
Immersive language and culture 
Immersive language and culture 
Using games to step back in time for authentic learning experiences
Introduction 
Students often benefit from opportunities 
to develop and apply their learning in 
authentic settings. Language students can 
progress quickly when they are immersed 
in a setting where everyone is using their 
target language. Field trips can also support 
lessons about historical periods and diverse 
cultures. However, providing authentic 
experience of an ancient language or culture 
is challenging. Immersive language and 
culture provides game-based opportunities 
for extended investigation of and roleplay 
within historic scenarios. This can be done 
through video-game-based experiences  
or through structured sessions of live-action 
role play.
Authentic learning
Immersive language and culture draws on 
the strengths of authentic learning. This is  
an approach to learning-by-doing that 
prompts students to find solutions to 
complex problems in multi-disciplinary 
settings. Students who are immersed in an 
authentic learning activity can be supported 
to develop the judgment to distinguish 
between reliable and unreliable information, 
the ability to recognise patterns in unfamiliar 
contexts, the flexibility to develop innovative 
solutions and to work across cultural 
boundaries1. Some features of authentic 
learning include sustained investigations, 
the challenge of drawing on multiple sources 
and perspectives, and opportunities to 
seek out multiple interpretations and work 
towards a variety of outcomes. This can 
be achieved in real-world settings but, in 
the case of past languages and cultures, 
students can benefit from the use of digital 
immersive video game environments, or an 
approach known as Reacting to the Past. 
Video games and immersive 
learning
As access to digital technology has become 
more widespread, virtual language learning 
has become increasingly popular. Propelled 
by the need to teach in online environments 
due to COVID lockdowns, virtual learning 
has become increasingly mainstream2. 
Language-learning software and apps, from 
Rosetta Stone and Duolingo to the software 
included with textbooks, have become 
an increasingly common part of learning 
a language. Many language-learning 
programs do not attempt to provide an 
authentic cultural environment, but function 
more like flashcards for the learner. The use 
of commercial entertainment industry video 
games to learn languages has provided an 
informal way (now sometimes formalised in 
language classrooms) of connecting sensory 
perceptions with unfamiliar languages, and 
these opportunities can be provided simply 
by changing the language in which the 
commercial game is installed. 
However, this technique is not possible for 
‘dead’ and endangered languages such 
as Latin or medieval French. Furthermore, 
to achieve the goal of language learning 
within a cultural context, the immersive 
environment must mirror the world of the 
target language and culture. For example, a 
modern kitchen table in a US-based game 
set in the 21st century would not be an 
appropriate table to include in a medieval 
court environment. Likewise, the vocabulary 
used in an off-the-shelf video game like 
Assassin’s Creed applies only to a specific 
set of conquest and conflict situations, 
leaving many gaps for those who will be 
working with ancient and medieval literature 
– for example, vocabulary relating to rich 
religious experiences and interpersonal 
relationships.

46
Innovating Pedagogy 2024
Game scholars have uncovered and established the powerful links between learning and play3. Using off-the-shelf games can be effective, but these games usually lack linguistic and cultural input that would make them authentic learning environments. In other words, playing a popular game like Fortnite in English may teach some new vocabulary, but it offers little awareness of any anglophone culture. Much more valuable3 are instructor- (or student-) produced games as well as game play systems designed specifically to promote learning the target language and culture. These have been shown to be effective – for example, students who played a 3D-immersive game related to an essay topic subsequently crafted richer second-
language narratives than those who did not use those environments4,5. Brendan’s Voyage is an example of an innovative use of video game design in support of learning and teaching dead languages and assessing the outcomes. The game, an adaptation of the medieval text The Voyage of Saint Brendan the Abbot teaches players to speak, read and write the Anglo-Norman dialect of medieval French. The entire experience is infused with architecture, art and literature appropriate to 12th-century Europe, meaning players also learn about the culture, politics and aesthetics of the period. For instance, a player is coached by a magical codex (an ancient manuscript text in book form) through the language skills needed to interact with a merchant. The codex can be consulted at any point to help the player converse in Anglo-Norman with the merchant. Using a game to provide learners with an immersive experience offers a variety of advantages. Game-based learning can help students to bypass the anxiety and lack of self-confidence that can be experienced when starting to learn a language. Any mistakes can be attributed to the in-game character rather than the student. The game also provides opportunities to communicate in medieval French, rather than being limited to the study of grammar and translation. Interacting in the game world emphasises how the language relates to the activity, context, and culture in which it was developed.
The codex can be consulted at any point to help the player converse in Anglo-Norman with the merchant in the game “Brendan’s Voyage” by Lynn Ramey and Jacob Abel.. 

Scene 8

46
Innovating Pedagogy 2024
Game scholars have uncovered and 
established the powerful links between 
learning and play3. Using off-the-shelf 
games can be effective, but these games 
usually lack linguistic and cultural input 
that would make them authentic learning 
environments. In other words, playing a 
popular game like Fortnite in English may 
teach some new vocabulary, but it offers 
little awareness of any anglophone culture. 
Much more valuable3 are instructor- (or 
student-) produced games as well as 
game play systems designed specifically 
to promote learning the target language 
and culture. These have been shown to be 
effective – for example, students who played 
a 3D-immersive game related to an essay 
topic subsequently crafted richer second-
language narratives than those who did not 
use those environments4,5. 
Brendan’s Voyage is an example of an 
innovative use of video game design in 
support of learning and teaching dead 
languages and assessing the outcomes.  
The game, an adaptation of the medieval 
text The Voyage of Saint Brendan the Abbot 
teaches players to speak, read and write the 
Anglo-Norman dialect of medieval French.  
The entire experience is infused with 
architecture, art and literature appropriate 
to 12th-century Europe, meaning players 
also learn about the culture, politics and 
aesthetics of the period. For instance, a 
player is coached by a magical codex 
(an ancient manuscript text in book form) 
through the language skills needed to 
interact with a merchant. The codex can  
be consulted at any point to help the  
player converse in Anglo-Norman with  
the merchant. 
Using a game to provide learners with an 
immersive experience offers a variety of 
advantages. Game-based learning can help 
students to bypass the anxiety and lack of 
self-confidence that can be experienced 
when starting to learn a language. Any 
mistakes can be attributed to the in-game 
character rather than the student. The game 
also provides opportunities to communicate 
in medieval French, rather than being 
limited to the study of grammar and 
translation. Interacting in the game world 
emphasises how the language relates to the 
activity, context, and culture in which it was 
developed.
The codex can be consulted at any point to help the player converse in Anglo-Norman with the 
merchant in the game “Brendan’s Voyage” by Lynn Ramey and Jacob Abel.

47
Immersive language and culture Immersive language and culture draws on the strengths of authentic learning A similar approach is applied in Operation LAPIS5, a two-year interactive adventure in which Latin learners develop skills in speaking, reading, and translating as they role-play as Romans. The game environment is used to foster engagement – it also provides a way of supporting a gradual shift from instructor-led learning to self-study, with students using the CODEX (a set of web-
based language resources) as a resource to support their decisions about in-game interactions and how to proceed with their adventure. As they do this, the teacher can provide feedback and guidance on areas such as language use, problem-solving techniques and quality of background research.
Unlike Brendan’s Voyage, Operation LAPIS is not set in a single game environment but uses a range of online tools and resources. However, an immersive language and culture approach can also be adopted without the use of technology, using a form of live-action role-playing game known as Reacting to the Past.
Reacting to the Past
Reacting to the Past6 consists of complex games in which students are assigned roles to explore an historic topic. Class sessions are run by learners, with educators advising and guiding them, as well as assessing both oral and written work. The intention is to draw students into the past, to encourage them to engage with big ideas, and to develop their intellectual and academic skills. For example, a class of students might take on the roles of countries’ representatives at the 1919 Paris Peace Conference, working to develop a treaty that establishes the terms of peace after the World War. This requires them to carry out research to establish what their countries’ visions and priorities are, and then negotiate with others in topical sub-committees to recommend a course of action of the leaders who will determine the final treaty. In another class, a group of students might take on roles in ancient Athens, in a time following military defeat and rebellion. The newly restored democracy is unstable, and students will debate topics such a democracy, oligarchy, imperialism, women’s rights and immigration – debates informed by the use of classical texts including Plato’s Republic.
The Reacting Consortium currently has more than 75 games on offer, covering subjects including politics, religion, economics and STEM. Each of these comes with a student gamebook, which describes the historical context, explains the game premise, introduces the central debates and sets out the rules. An instructor manual includes handouts and gives instructions on running the game – how many students it is designed for and how long it will last. Role sheets are supplied for the students, so they can find out what their individual goals are and which strategies they can use. There are also companion texts, including readings from primary sources. Some role-playing games – such as Monumental Consequence (in which students consider whether art is ever worth dying for) – can be played in less than an hour and could provide an introduction to Reacting for both students and educators. Others can play out over weeks or months. Athens 403BCE, described above, is one of the larger scale games and can run for as many as 15 sessions with roles for up to 50 students. Versailles 1919, the Paris Peace Conference game, takes around eight sessions and can run with as few as ten students. Challenges, barriers, limitations
A recent study7 looked at the experience of running a Reacting game three times with classes of undergraduates and graduates. The students found some aspects challenging, particularly giving oral presentations, and engaging in debates. They also found it difficult to understand and summarise research and to critique arguments. Overall, though, they found the experience valuable. They reported that it not only helped them to strengthen skills and knowledge but also to identify areas where they had previously been weak. They also felt it prompted them to learn the material better than they would have done in other circumstances, and they believed that they would retain it better because the experience was so memorable.. 

Scene 9

47
Immersive language and culture 
Immersive language and culture draws 
on the strengths of authentic learning 
A similar approach is applied in Operation 
LAPIS5, a two-year interactive adventure 
in which Latin learners develop skills in 
speaking, reading, and translating as they 
role-play as Romans. The game environment 
is used to foster engagement – it also 
provides a way of supporting a gradual shift 
from instructor-led learning to self-study, 
with students using the CODEX (a set of web-
based language resources) as a resource 
to support their decisions about in-game 
interactions and how to proceed with their 
adventure. As they do this, the teacher can 
provide feedback and guidance on areas 
such as language use, problem-solving 
techniques and quality of background 
research.
Unlike Brendan’s Voyage, Operation LAPIS 
is not set in a single game environment but 
uses a range of online tools and resources. 
However, an immersive language and 
culture approach can also be adopted 
without the use of technology, using a form 
of live-action role-playing game known as 
Reacting to the Past.
Reacting to the Past
Reacting to the Past6 consists of complex 
games in which students are assigned roles 
to explore an historic topic. Class sessions 
are run by learners, with educators advising 
and guiding them, as well as assessing both 
oral and written work. The intention is to draw 
students into the past, to encourage them 
to engage with big ideas, and to develop 
their intellectual and academic skills. For 
example, a class of students might take on 
the roles of countries’ representatives at 
the 1919 Paris Peace Conference, working to 
develop a treaty that establishes the terms 
of peace after the World War. This requires 
them to carry out research to establish what 
their countries’ visions and priorities are, 
and then negotiate with others in topical 
sub-committees to recommend a course 
of action of the leaders who will determine 
the final treaty. In another class, a group 
of students might take on roles in ancient 
Athens, in a time following military defeat 
and rebellion. The newly restored democracy 
is unstable, and students will debate topics 
such a democracy, oligarchy, imperialism, 
women’s rights and immigration – debates 
informed by the use of classical texts 
including Plato’s Republic.
The Reacting Consortium currently has more 
than 75 games on offer, covering subjects 
including politics, religion, economics and 
STEM. Each of these comes with a student 
gamebook, which describes the historical 
context, explains the game premise, 
introduces the central debates and sets 
out the rules. An instructor manual includes 
handouts and gives instructions on running 
the game – how many students it is 
designed for and how long it will last. Role 
sheets are supplied for the students, so they 
can find out what their individual goals are 
and which strategies they can use. There are 
also companion texts, including readings 
from primary sources. 
Some role-playing games – such as 
Monumental Consequence (in which 
students consider whether art is ever worth 
dying for) – can be played in less than an 
hour and could provide an introduction to 
Reacting for both students and educators. 
Others can play out over weeks or months. 
Athens 403BCE, described above, is one of the 
larger scale games and can run for as many 
as 15 sessions with roles for up to 50 students. 
Versailles 1919, the Paris Peace Conference 
game, takes around eight sessions and can 
run with as few as ten students. 
Challenges, barriers, limitations
A recent study7 looked at the experience 
of running a Reacting game three times 
with classes of undergraduates and 
graduates. The students found some 
aspects challenging, particularly giving oral 
presentations, and engaging in debates. 
They also found it difficult to understand 
and summarise research and to critique 
arguments. Overall, though, they found the 
experience valuable. They reported that it 
not only helped them to strengthen skills and 
knowledge but also to identify areas where 
they had previously been weak. They also 
felt it prompted them to learn the material 
better than they would have done in other 
circumstances, and they believed that they 
would retain it better because the experience 
was so memorable.

48
Innovating Pedagogy 2024
Almost a decade ago, Julie Sykes and Jonathon Reinhardt anticipated an explosion of purpose-built games for language learning in authentic contexts, noting that it is ‘critical that researchers and practitioners evaluate and, in many cases, participate in the creation of these games’8. However, immersive language-
learning games created with researchers have not yet become mainstream or widely available. Perhaps due to the complexities of game production, studies have focused on games produced for entertainment (most commonly, multi-player role-playing games) that are co-opted into a classroom setting to be played in a target language.
Conclusions
Immersive and game-based learning methods provide a rich, engaging way for students to learn languages and understand historical and cultural contexts. By integrating digital game environments and live-action role-playing, these educational techniques allow students to dive deep into historical and cultural settings, enhancing their learning experience and retention. Games like Brendan’s Voyage and Operation LAPIS exemplify innovative approaches that not only teach language skills but also imbue students with a deeper appreciation of historical cultures and practices. Despite challenges such as the complexity of creating authentic game-based learning tools, the potential for immersive educational experiences is vast, promising a more dynamic and effective approach to learning languages and exploring cultures. As virtual reality and augmented reality technologies advance, immersive learning environments could become dramatically enhanced, offering an even deeper level of immersion and interaction. Immersive pedagogies are likely to become widely adopted across the curriculum.. 

Scene 10

48
Innovating Pedagogy 2024
Almost a decade ago, Julie Sykes and 
Jonathon Reinhardt anticipated an 
explosion of purpose-built games for 
language learning in authentic contexts, 
noting that it is ‘critical that researchers 
and practitioners evaluate and, in many 
cases, participate in the creation of these 
games’8. However, immersive language-
learning games created with researchers 
have not yet become mainstream or widely 
available. Perhaps due to the complexities 
of game production, studies have focused 
on games produced for entertainment 
(most commonly, multi-player role-playing 
games) that are co-opted into a classroom 
setting to be played in a target language.
Conclusions
Immersive and game-based learning 
methods provide a rich, engaging way for 
students to learn languages and understand 
historical and cultural contexts. By 
integrating digital game environments and 
live-action role-playing, these educational 
techniques allow students to dive deep into 
historical and cultural settings, enhancing 
their learning experience and retention. 
Games like Brendan’s Voyage and Operation 
LAPIS exemplify innovative approaches 
that not only teach language skills but also 
imbue students with a deeper appreciation 
of historical cultures and practices. Despite 
challenges such as the complexity of 
creating authentic game-based learning 
tools, the potential for immersive educational 
experiences is vast, promising a more 
dynamic and effective approach to learning 
languages and exploring cultures. As virtual 
reality and augmented reality technologies 
advance, immersive learning environments 
could become dramatically enhanced, 
offering an even deeper level of immersion 
and interaction. Immersive pedagogies are 
likely to become widely adopted across  
the curriculum.

49
Immersive language and culture References
1. A paper that explains and exemplifies authentic learning:
Lombardi, M.M. and Oblinger, D.G. (2007) ‘Authentic learning for the 21st century: An overview’, Educause Learning Initiative, 1(2007), pp. 1–12. Available at: https://library.educause.
edu/resources/2007/1/authentic-learning-for-
the-21st-century-an-overview (Accessed 27 June 2024).
2. A web page setting out some consequences of the COVID-19 pandemic:
Li, C. and Lalani, F. (2020) ‘The COVID-19 pandemic has changed education forever. This is how’, World Economic Forum, April 29, 2020. Available at: https://www.weforum.org/agenda/2020/04/
coronavirus-education-global-covid19-online-
digital-learning/ (Accessed 27 June 2024). 3. A book about video games in education:
Gee, J.P. (2003) What video games have to teach us about learning and literacy. Palgrave Macmillan, New York.
4. A journal article reporting using an interactive fiction game to teach German vocabulary, reading, and culture to university students:
Neville, D.O., Shelton, B.E., and McInnis, B. (2009) ‘Cybertext redux: using digital game-based learning to teach L2 vocabulary, reading, and culture’, Computer Assisted Language Learning, 22(5), pp.409–424.
5. A magazine article describing the Operation LAPIS game and how it is used in teaching:
Slota, S.T., Ballestrini, K., and Pearsall, M. (2013) ‘Learning through Operation LAPIS: A game-based approach to the language classroom’, The Language Educator, October, pp.36–38.
6. A journal article discussing the pedagogy of Reacting to the Past:
Lamontagne, K. (2020) ‘When the Past is the Classroom: Merging ‘Reacting to the Past’ and Experiential Education’, Impact: The Journal of the Center for Interdisciplinary Teaching & Learning, 9(2), pp.9–17. 7. A journal article reflecting on the value of immersive, experiential active learning:
Richardson, D.S., Bledsoe, R.S., and Manning, K. (2023) ‘Pushing Active Learning to the Extreme: Is It Worth It?’, College Teaching, 71(1), pp. 9–17.
8. A book chapter on digital games in language teaching and learning: Reinhardt, J., and Sykes, J.M. (2012) ‘Conceptualizing digital game-mediated L2 learning and pedagogy: Game-enhanced and game-based research and practice’, in Digital games in language learning and teaching (pp. 32–49). London: Palgrave Macmillan UK.
Resources • A short video introduction to the Brendan’s Voyage video game, designed to teach Anglo-
Norman language and culture:
Brendan’s Voyage. Available at: https://www.
youtube.com/watch?v=-3pVVCzQfP8&ab_
channel=LR (Accessed 27 June 2024).
• Resources on Reacting to the Past:
Reacting to the Past. Available at: https://reacting.barnard.edu/ – browse the games, sign up for an event (they tend to be in the US), join the community and/or sign up for the newsletter (Accessed 27 June 2024).
Reacting Consortium. Resources from a consortium of institutions and instructors ‘united by the belief that imagination, inquiry, and engagement are essential features of teaching and learning’. Available at: https://reactingconsortium.org/ (Accessed 27 June 2024).
• An article by Nicolas Trépanier about teaching history with video games, from the Newsmagazine of the American Historical Association:
The Assassin’s Perspective: Teaching History with Video Games. May 1, 2014. Available at: https://www.historians.org/
perspectives-article/the-assassins-perspective-
teaching-history-with-video-games-may-2014/ (Accessed 27 June 2024). • Barnard and Columbia undergraduates play Defining a Nation: India on the Eve of Independence, 1945 at Barnard College (5-minute video):
Reacting to the past: The student perspective. Available at: https://www.youtube.com/watch?v=_
U6L9ERzw0U&ab_channel=ReactingtothePast (Accessed 27 June 2024).
• Students—with costumes and trumpet in hand—
assume roles in the Athenian Assembly to debate issues as Athens reconstructs after the Peloponnesian War (2-minute video). Reacting to the Past FYS142 Class Visit. Available at: https://www.youtube.
com/watch?v=clO0uvCgwlM&ab_
channel=SmithCollege (Accessed 27 June 2024).. 

Scene 11

49
Immersive language and culture 
References
1. A paper that explains and exemplifies authentic 
learning:
Lombardi, M.M. and Oblinger, D.G. (2007) 
‘Authentic learning for the 21st century: An 
overview’, Educause Learning Initiative, 1(2007), 
pp. 1–12. Available at: https://library.educause.
edu/resources/2007/1/authentic-learning-for-
the-21st-century-an-overview (Accessed 27 June 
2024).
2. A web page setting out some consequences of 
the COVID-19 pandemic:
Li, C. and Lalani, F. (2020) ‘The COVID-19 pandemic 
has changed education forever. This is how’, 
World Economic Forum, April 29, 2020. Available 
at: https://www.weforum.org/agenda/2020/04/
coronavirus-education-global-covid19-online-
digital-learning/ (Accessed 27 June 2024). 
3. A book about video games in education:
Gee, J.P. (2003) What video games have to 
teach us about learning and literacy. Palgrave 
Macmillan, New York.
4. A journal article reporting using an interactive 
fiction game to teach German vocabulary, 
reading, and culture to university students:
Neville, D.O., Shelton, B.E., and McInnis, B. (2009) 
‘Cybertext redux: using digital game-based 
learning to teach L2 vocabulary, reading, and 
culture’, Computer Assisted Language Learning, 
22(5), pp.409–424.
5. A magazine article describing the Operation LAPIS 
game and how it is used in teaching:
Slota, S.T., Ballestrini, K., and Pearsall, M. (2013) 
‘Learning through Operation LAPIS: A game-based 
approach to the language classroom’,  
The Language Educator, October, pp.36–38.
6. A journal article discussing the pedagogy of 
Reacting to the Past:
Lamontagne, K. (2020) ‘When the Past is the 
Classroom: Merging ‘Reacting to the Past’ and 
Experiential Education’, Impact: The Journal of the 
Center for Interdisciplinary Teaching & Learning, 
9(2), pp.9–17. 
7. A journal article reflecting on the value of 
immersive, experiential active learning:
Richardson, D.S., Bledsoe, R.S., and Manning, K. 
(2023) ‘Pushing Active Learning to the Extreme: Is It 
Worth It?’, College Teaching, 71(1), pp. 9–17.
8. A book chapter on digital games in language 
teaching and learning: 
Reinhardt, J., and Sykes, J.M. (2012) 
‘Conceptualizing digital game-mediated L2 
learning and pedagogy: Game-enhanced and 
game-based research and practice’, in Digital 
games in language learning and teaching  
(pp. 32–49). London: Palgrave Macmillan UK.
Resources 
• 
A short video introduction to the Brendan’s 
Voyage video game, designed to teach Anglo-
Norman language and culture:
Brendan’s Voyage. Available at: https://www.
youtube.com/watch?v=-3pVVCzQfP8&ab_
channel=LR (Accessed 27 June 2024).
• 
Resources on Reacting to the Past:
Reacting to the Past. Available at:  
https://reacting.barnard.edu/ – browse the 
games, sign up for an event (they tend to be in 
the US), join the community and/or sign up for the 
newsletter (Accessed 27 June 2024).
Reacting Consortium. Resources from a 
consortium of institutions and instructors  
‘united by the belief that imagination, inquiry,  
and engagement are essential features of 
teaching and learning’. Available at:  
https://reactingconsortium.org/  
(Accessed 27 June 2024).
• 
An article by Nicolas Trépanier about teaching 
history with video games, from the Newsmagazine 
of the American Historical Association:
The Assassin’s Perspective: Teaching History with 
Video Games. May 1, 2014. 
Available at: https://www.historians.org/
perspectives-article/the-assassins-perspective-
teaching-history-with-video-games-may-2014/ 
(Accessed 27 June 2024). 
• 
Barnard and Columbia undergraduates 
play Defining a Nation: India on the Eve of 
Independence, 1945 at Barnard College  
(5-minute video):
Reacting to the past: The student perspective. 
Available at: https://www.youtube.com/watch?v=_
U6L9ERzw0U&ab_channel=ReactingtothePast 
(Accessed 27 June 2024).
• 
Students—with costumes and trumpet in hand—
assume roles in the Athenian Assembly to 
debate issues as Athens reconstructs after the 
Peloponnesian War (2-minute video). 
Reacting to the Past FYS142 Class Visit. 
Available at: https://www.youtube.
com/watch?v=clO0uvCgwlM&ab_
channel=SmithCollege (Accessed 27 June 2024).

50
Innovating Pedagogy 2024
Exploring scientific models from the inside
Rich embodied experiences supported by extended reality and AI
Introduction The value of hands-on activities in science, manipulatives in maths (concrete objects showing concepts), and embodied empathy in literature (experiencing a character’s world) clearly demonstrates how real-
world experiences can enhance learning. An embodied learning approach seeks to expand the repertoire of resources students use to learn from their experiences in the world by incorporating how they move their bodies in space and how they interact with the physical environment around them. This approach is particularly powerful for understanding scientific models, allowing students to experience them first-hand ‘from the inside.’ Pedagogically, such learning is more robust when followed by reflective activities, helping to anchor abstract concepts in tangible experiences. However, the complexity of embodied exploration can hinder effective reflection. Teachers need useful digital data traces to help students focus on key moments of insight and difficulty, facilitating a deeper understanding of the phenomenon being modelled.
Two emerging technical trends are set to enhance student learning of models in classrooms. The first is extended reality (XR) technologies, which include virtual reality (VR), augmented reality (AR), and mixed reality (MR). These technologies blend physical surroundings with digital elements, transforming interactions with technology and learning methods. The second trend is the significant growth in artificial intelligence (AI) capabilities, which can now process extremely large sets of data from multiple modalities. The merging of XR and AI holds promise for evolving and refining educational strategies and tools for model learning by increasing the responsiveness and adaptability of XR-based learning environments and supporting AI-
enhanced interpretation of the full spectrum of human sensory information available in these environments. With guidance and support from teachers, such digitally enhanced experiences can help students move from direct experiences to conceptual understanding of scientific and other models that might otherwise remain ungrounded or misunderstood.
Immersion in models through extended reality
A common theme among XR environments is immersing students in a digitally enhanced experience that can be used as a springboard for more powerful understandings. VR strives for sensory immersion to help users suspend disbelief and engage in the digital world as if it were real. AR overlays digital information onto the physical world, enhancing real-world experiences with interactive and contextual digital elements. MR combines the physical and digital worlds, supporting interactive actions and sensory immersion in dynamic ways. • In MR, physical movements and interactions with real objects are mirrored by their digital counterparts in a simulated environment. For example, to learn about the complex relationships between bees and flowers in pollination, students can move their bodies around in the physical space of a classroom while seeing virtual bees moving where they move in a virtual garden.1 This integration is visually achieved by overlaying digital elements onto a live video feed that captures real-world phenomena. These virtual representations can be viewed on devices ranging from handheld smartphones to large-scale projections on walls and screens. these environments are novel because they provide playful first-
hand explorations of scientific models. 

Scene 12

50
Innovating Pedagogy 2024
Exploring scientific models from the inside
Rich embodied experiences supported by extended reality and AI
Introduction 
The value of hands-on activities in science, 
manipulatives in maths (concrete objects 
showing concepts), and embodied empathy 
in literature (experiencing a character’s 
world) clearly demonstrates how real-
world experiences can enhance learning. 
An embodied learning approach seeks to 
expand the repertoire of resources students 
use to learn from their experiences in the 
world by incorporating how they move their 
bodies in space and how they interact with 
the physical environment around them. 
This approach is particularly powerful for 
understanding scientific models, allowing 
students to experience them first-hand ‘from 
the inside.’ Pedagogically, such learning is 
more robust when followed by reflective 
activities, helping to anchor abstract 
concepts in tangible experiences. However, 
the complexity of embodied exploration can 
hinder effective reflection. Teachers need 
useful digital data traces to help students 
focus on key moments of insight and 
difficulty, facilitating a deeper understanding 
of the phenomenon being modelled.
Two emerging technical trends are set to 
enhance student learning of models in 
classrooms. The first is extended reality (XR) 
technologies, which include virtual reality 
(VR), augmented reality (AR), and mixed 
reality (MR). These technologies blend 
physical surroundings with digital elements, 
transforming interactions with technology and 
learning methods. The second trend is the 
significant growth in artificial intelligence (AI) 
capabilities, which can now process extremely 
large sets of data from multiple modalities.  
The merging of XR and AI holds promise for 
evolving and refining educational strategies 
and tools for model learning by increasing the 
responsiveness and adaptability of XR-based 
learning environments and supporting AI-
enhanced interpretation of the full spectrum 
of human sensory information available 
in these environments. With guidance 
and support from teachers, such digitally 
enhanced experiences can help students 
move from direct experiences to conceptual 
understanding of scientific and other models 
that might otherwise remain ungrounded or 
misunderstood.
Immersion in models through 
extended reality
A common theme among XR environments is 
immersing students in a digitally enhanced 
experience that can be used as a springboard 
for more powerful understandings. VR strives 
for sensory immersion to help users suspend 
disbelief and engage in the digital world as 
if it were real. AR overlays digital information 
onto the physical world, enhancing real-world 
experiences with interactive and contextual 
digital elements. MR combines the physical 
and digital worlds, supporting interactive 
actions and sensory immersion in dynamic 
ways. 
• In MR, physical movements and 
interactions with real objects are mirrored 
by their digital counterparts in a simulated 
environment. For example, to learn about 
the complex relationships between bees 
and flowers in pollination, students can 
move their bodies around in the physical 
space of a classroom while seeing virtual 
bees moving where they move in a virtual 
garden.1 This integration is visually achieved 
by overlaying digital elements onto a 
live video feed that captures real-world 
phenomena. These virtual representations 
can be viewed on devices ranging from 
handheld smartphones to large-scale 
projections on walls and screens. 
these environments are novel 
because they provide playful first-
hand explorations of scientific models

51
Exploring scientific models from the inside
• In more extensive MR setups, the system tracks user movement within a physical space to display digital information in the context of a scenario being enacted and to enable interactions with virtual entities positioned in specific real-world locations. For example, in the pollination scenario, students can see where different kinds of flowers are placed in specific locations in the virtual garden and move their bee to interact with them by moving their body in physical space. Importantly, the system is tracking and responsive to multiple bees’ actions at once, allowing students to experience the complex dynamics of the system and engage in teamwork strategies to optimise pollination and maximise the health of the garden. Room-
sized space-based MR systems have been successful at enabling students to learn about various types of models from the inside, including biological systems such as animal foraging 2, physical science concepts such as states of matter3, and geological concepts related to celestial entities found in outer space4. Pedagogically, these environments are novel because they provide playful first-
hand explorations of scientific models. For example, the STEP (Science through Technology Enhanced Play) project5 has students pretend to become parts of a scientific system they want to study. Digging deeper into the case of pollination, children learn by becoming bees and exploring a digital flower patch together. In one implementation, they enter a gymnasium and see a video feed of themselves walking around the room, but everywhere they go, a bee in the virtual space follows them (based on an array of sensors placed around the room). In certain locations around the room, there are virtual flowers that only appear when students are next to them. The students-as-bees collect nectar, which can vary in quantity and quality. The bees bring this nectar back to the hive and begin to discuss how best to organise to collect the most nectar before winter comes and it is all gone. Eventually, they discover that depending on their foraging activity, certain flowers get pollinated, or not, and therefore live or die.
The most recent version of the STEP environment is programmable by students. They can choose to add new agents (e.g., birds as predators) or explore edge-case interactions (e.g., program the bees to all go to the ‘best’ flower or to distribute themselves to all the flowers). Moving back and forth between being an agent within a model and reflecting on/programming this model balances the ‘diving in’ and ‘stepping out’ that Edith Ackerman describes as powerful for constructing deep understanding of experiences6. Particularly powerful in this case is the way that taking the role of modeler allows students to tell a story about a place that is personally and culturally meaningful. For instance, students in urban areas may explore the effects on the flower-bee system when new building developments are created with and without green rooftops. Such a student-
led approach to modelling is a powerful new form of ‘computational inquiry’ (often learning through simulations) that allows students to control, and see the effects of changing the ‘rules of the game’, rather than simply learning a constrained set of rules that have been pre-programmed into a system. Data visualisation then serves as a critical tool for teachers to recognise and interpret key moments of discovery and/or confusion among students and use these for reflective discussions that can lead to further experimentation with the model.
AI-processing of data from multiple modalities
Embodied learning data, which captures student interactions in a three-dimensional space and time, presents a complex challenge to represent and interpret due to its multimodal, dynamic nature. Unlike traditional educational data, which often focuses primarily on language or digital interactions, embodied learning data encompasses a wide range of both verbal and non-verbal information and movements through space. Strategic sensors placed in XR embodied learning environments collect vast amounts of data in multiple modalities, including streaming video, sound (commonly speech), positioning data, and system logs from the modelling environment.. 

Scene 13

51
Exploring scientific models from the inside
• In more extensive MR setups, the system 
tracks user movement within a physical 
space to display digital information in the 
context of a scenario being enacted and 
to enable interactions with virtual entities 
positioned in specific real-world locations. 
For example, in the pollination scenario, 
students can see where different kinds of 
flowers are placed in specific locations in 
the virtual garden and move their bee to 
interact with them by moving their body 
in physical space. Importantly, the system 
is tracking and responsive to multiple 
bees’ actions at once, allowing students 
to experience the complex dynamics 
of the system and engage in teamwork 
strategies to optimise pollination and 
maximise the health of the garden. Room-
sized space-based MR systems have been 
successful at enabling students to learn 
about various types of models from the 
inside, including biological systems such 
as animal foraging 2, physical science 
concepts such as states of matter3, and 
geological concepts related to celestial 
entities found in outer space4. 
Pedagogically, these environments are 
novel because they provide playful first-
hand explorations of scientific models. 
For example, the STEP (Science through 
Technology Enhanced Play) project5 has 
students pretend to become parts of a 
scientific system they want to study. Digging 
deeper into the case of pollination, children 
learn by becoming bees and exploring 
a digital flower patch together. In one 
implementation, they enter a gymnasium 
and see a video feed of themselves walking 
around the room, but everywhere they go, a 
bee in the virtual space follows them (based 
on an array of sensors placed around the 
room). In certain locations around the 
room, there are virtual flowers that only 
appear when students are next to them. 
The students-as-bees collect nectar, which 
can vary in quantity and quality. The bees 
bring this nectar back to the hive and begin 
to discuss how best to organise to collect 
the most nectar before winter comes and 
it is all gone. Eventually, they discover that 
depending on their foraging activity, certain 
flowers get pollinated, or not, and therefore 
live or die.
The most recent version of the STEP 
environment is programmable by students. 
They can choose to add new agents (e.g., 
birds as predators) or explore edge-case 
interactions (e.g., program the bees to 
all go to the ‘best’ flower or to distribute 
themselves to all the flowers). Moving back 
and forth between being an agent within 
a model and reflecting on/programming 
this model balances the ‘diving in’ and 
‘stepping out’ that Edith Ackerman 
describes as powerful for constructing deep 
understanding of experiences6. Particularly 
powerful in this case is the way that taking 
the role of modeler allows students to tell 
a story about a place that is personally 
and culturally meaningful. For instance, 
students in urban areas may explore the 
effects on the flower-bee system when new 
building developments are created with 
and without green rooftops. Such a student-
led approach to modelling is a powerful 
new form of ‘computational inquiry’ (often 
learning through simulations) that allows 
students to control, and see the effects of 
changing the ‘rules of the game’, rather 
than simply learning a constrained set of 
rules that have been pre-programmed into 
a system. Data visualisation then serves as 
a critical tool for teachers to recognise and 
interpret key moments of discovery and/or 
confusion among students and use these for 
reflective discussions that can lead to further 
experimentation with the model.
AI-processing of data from 
multiple modalities
Embodied learning data, which captures 
student interactions in a three-dimensional 
space and time, presents a complex 
challenge to represent and interpret due 
to its multimodal, dynamic nature. Unlike 
traditional educational data, which often 
focuses primarily on language or digital 
interactions, embodied learning data 
encompasses a wide range of both verbal 
and non-verbal information and movements 
through space. Strategic sensors placed 
in XR embodied learning environments 
collect vast amounts of data in multiple 
modalities, including streaming video, sound 
(commonly speech), positioning data, and 
system logs from the modelling environment.

52
Innovating Pedagogy 2024
This data represents aspects of learners’ movements, attention focus, emotional states, and digital interactions, which are essential for a comprehensive understanding of students’ learning and problem-solving activities. However, managing and analysing this diverse data is complex and requires sophisticated computational tools and approaches.
Interaction Analysis (IA) is a key method used by educational researchers to extract insights from video data of embodied and other learning interactions7. IA examines human interactions in context, revealing the subtle dynamics necessary to understand collaborative, embodied learning environments. However, IA is labour-
intensive and requires significant human effort. An interdisciplinary team of learning sciences and computer science researchers at Vanderbilt and Indiana universities has been exploiting recent advances in artificial intelligence (AI) and multimodal learning analytics (MMLA) to augment – but not supplant – human analysis and help support the generation of useful insights for students, teachers, and researchers. These technologies are being used to help provide a deeper understanding of learner engagement in embodied modelling activities, enabling teachers to gain a richer grasp of their students’ learning process and offer personalised feedback and support for reflection.
For example, a timeline visualisation has been developed that integrates information across multiple data modalities to bring together information about what learners said (from the audio stream), where they were looking (from the video stream), what they were doing (from the system data logs), and how they were feeling (again based on information extracted from the video stream). This timeline can enhance teachers’ understanding of how students interact with each other and their environment over time, allowing them to facilitate better reflective discussions of the students’ embodied modelling experiences.
To effectively analyse cases of embodied model learning, AI-based algorithms document the evolving states in XR environments, tracking and interpreting student engagement with scientific processes both individually and in groups. The classroom teacher actively guides students, helping them relate their activities to the modelled phenomena. The AI-driven timeline visualisation combines system log data on movements and interactions, gaze data to understand attention shifts, and video-derived emotional expressions. This comprehensive visualisation format provides clear, compelling insights for teachers and researchers, highlighting key moments of insight and difficulty. For example, the timeline may show a student’s period of confusion or expression of frustration when they did not get the attention of other students to help them complete a task, offering teachers valuable information to guide further interactions. The visualisation can also serve as a tool for reviewing, recognising and interpreting key moments of discovery among students, which can be linked to their evolving understanding of the scientific content being studied. Teachers use these insights to facilitate meaningful discussions, supporting reflection and advancing the students’ learning processes.
Embodied Model Learning in the Classroom with Timeline supporting Interactivity Analyses. This includes elements such as transcription of each individuals’ contributions, gaze, and emotions.. 

Scene 14

52
Innovating Pedagogy 2024
This data represents aspects of learners’ 
movements, attention focus, emotional 
states, and digital interactions, which are 
essential for a comprehensive understanding 
of students’ learning and problem-solving 
activities. However, managing and analysing 
this diverse data is complex and requires 
sophisticated computational tools and 
approaches.
Interaction Analysis (IA) is a key method 
used by educational researchers to extract 
insights from video data of embodied 
and other learning interactions7. IA 
examines human interactions in context, 
revealing the subtle dynamics necessary 
to understand collaborative, embodied 
learning environments. However, IA is labour-
intensive and requires significant human 
effort. An interdisciplinary team of learning 
sciences and computer science researchers 
at Vanderbilt and Indiana universities 
has been exploiting recent advances in 
artificial intelligence (AI) and multimodal 
learning analytics (MMLA) to augment – but 
not supplant – human analysis and help 
support the generation of useful insights 
for students, teachers, and researchers. 
These technologies are being used to help 
provide a deeper understanding of learner 
engagement in embodied modelling 
activities, enabling teachers to gain a richer 
grasp of their students’ learning process and 
offer personalised feedback and support for 
reflection.
For example, a timeline visualisation has 
been developed that integrates information 
across multiple data modalities to bring 
together information about what learners 
said (from the audio stream), where they 
were looking (from the video stream), what 
they were doing (from the system data logs), 
and how they were feeling (again based 
on information extracted from the video 
stream). This timeline can enhance teachers’ 
understanding of how students interact with 
each other and their environment over time, 
allowing them to facilitate better reflective 
discussions of the students’ embodied 
modelling experiences.
To effectively analyse cases of embodied 
model learning, AI-based algorithms 
document the evolving states in XR 
environments, tracking and interpreting 
student engagement with scientific 
processes both individually and in groups. 
The classroom teacher actively guides 
students, helping them relate their activities 
to the modelled phenomena. The AI-driven 
timeline visualisation combines system log 
data on movements and interactions, gaze 
data to understand attention shifts, and 
video-derived emotional expressions. This 
comprehensive visualisation format provides 
clear, compelling insights for teachers and 
researchers, highlighting key moments 
of insight and difficulty. For example, the 
timeline may show a student’s period of 
confusion or expression of frustration when 
they did not get the attention of other 
students to help them complete a task, 
offering teachers valuable information to 
guide further interactions. The visualisation 
can also serve as a tool for reviewing, 
recognising and interpreting key moments 
of discovery among students, which can be 
linked to their evolving understanding of the 
scientific content being studied. Teachers 
use these insights to facilitate meaningful 
discussions, supporting reflection and 
advancing the students’ learning processes.
Embodied Model Learning in the Classroom with 
Timeline supporting Interactivity Analyses. This 
includes elements such as transcription of each 
individuals’ contributions, gaze, and emotions.

Innovating Pedagogy 2024 part 4