[Audio] Hi, my name is Amy Hsiao. I am going to tell you about our article: Developing a framework to re-design writing assignment assessment for the era of Large Language Models. Many teachers in higher education are concerned about whether students misuse these chatbots to ghostwrite their academic assignments and whether assessment can still effectively test what it intends to assess. This article aims to help teachers address these concerns.. 

Scene 1

Developing a framework to re-design writing assignment assessment for the era of Large Language Models (LLMs)

Ya-Ping (Amy) Hsiao*, Nadia Klijn* and Mei-Shiu Chiu**
  *Tilburg University, Tilburg, The Netherlands **National Chengchi University, Taipei, Taiwan

A logo with a tree and a blue and yellow diamond

Description automatically generated

A black and white logo

Description automatically generated

[Audio] We were under high time pressure to solve a real educational challenge. Therefore, we adopted Van Aken and Berends' Design Science Research (DSR) approach. This means we developed a solution to the problem, that is the framework, and then we used cases to show teachers how to apply this framework in the workshops. We also explored the effectiveness of our framework.. 

Scene 2

Design Science Research (DSR) (Van Aken & Berends, 2018)

This study focuses on … Outcomes (3) developing the solution, which may involve creating a framework, model, or other applicable entities the framework (4) providing a demonstration of the solution’s effectiveness workshops

[Audio] The framework consists of six design dimensions. The most important step for teachers to consider is the course context and learning goals.. 

Scene 3

Re-designing Writing Assignment Assessment for the Era of Large Language Models

 1. Purpose -Course Context & Learning Goals

 2. Function & Focus

 3. Grading criteria

 4. Modes 

 5. Authenticity

 6. Administration

[Audio] Although tools based on Large Language Models (LLMs) have the potential to support student learning, teachers should think about how using these tools in their courses aligns with the overall learning outcomes of their programs. For introductory courses, using these tools could potentially make it more challenging for students to acquire fundamental knowledge and skills.. 

Scene 4

PLO1 PLO2 Knowledge
 PLO6 PLO4 Skills PLO5 PLO3 PLO7
 PLO8 PLO9 Character PLO10

Graduation project (Theses/Internships) Course 1 Course 2 Course 3 Course 4 Course 5 Course 6 Year 1 Year 2 Year 3 Course 1 Course 2 Course 3 Course 4 Course 5 Course 6 Course 1 Course 2 Course 3

[Audio] When including critical thinking in the course's learning goals, it's essential to consider the students' developmental stage in their critical thinking abilities. For example, first-year Bachelor's students are at the beginning stage of critical thinking and need to focus on acquiring information literacy skills. On the other hand, third-year Bachelor's students are more advanced critical thinkers and should be able to carry out complex problem-solving tasks. Once this distinction is made, it becomes more straightforward for teachers to create suitable writing assignments for assessing critical thinking.. 

Scene 5

Levels of Critical Thinking in Course Learning Goals

Beginning  1st year Bachelor’s

 Intermediate 2nd year Bachelor’s

 Advanced 3rd year Bachelor’s

 Proficient Master’s

[Audio] Once the course learning goals and writing assignment types are determined, the next step is to define appropriate grading criteria.. 

Scene 6

[Audio] These criteria must align with the targeted critical thinking skills. At the same time, they should encourage students to critically evaluate the weaknesses and limitations of current tools involving Large Language Models.. 

Scene 7

Information literacy Complex problem solving Transfer to contexts Evaluative judgment

[Audio] To enhance teachers' professional development, we conducted three workshops to explain the design dimensions of the framework and address teachers' learning questions. We used cases that were relevant to the participants' respective disciplines to demonstrate how this framework can be applied to redesign their writing assignments. Our observations indicate that the participating teachers responded positively to this framework. In the future studies, we will assess the effectiveness of teachers' implementation of this framework in their self-redesign process.. 

Scene 8

Question-driven

 Case-based application of the framework

Scene 0

Developing a framework to re-design writing assignment assessment for the era of Large Language Models

Many teachers in higher education are concerned about whether students misuse these chatbots to ghostwrite their academic assignments and whether assessment can still effectively test what it intends to assess

This article aims to help teachers address these concerns.

avatar

We were under high time pressure to solve a real educational challenge

Therefore, we adopted Van Aken and Berends' Design Science Research (DSR) approach

This means we developed a solution to the problem, that is the framework, and then we used cases to show teachers how to apply this framework in the workshops

We also explored the effectiveness of our framework.

The framework consists of six design dimensions

The most important step for teachers to consider is the course context and learning goals.

Although tools based on Large Language Models (LLMs) have the potential to support student learning, teachers should think about how using these tools in their courses aligns with the overall learning outcomes of their programs

For introductory courses, using these tools could potentially make it more challenging for students to acquire fundamental knowledge and skills.

When including critical thinking in the course's learning goals, it's essential to consider the students' developmental stage in their critical thinking abilities

For example, first-year Bachelor's students are at the beginning stage of critical thinking and need to focus on acquiring information literacy skills

On the other hand, third-year Bachelor's students are more advanced critical thinkers and should be able to carry out complex problem-solving tasks

Once this distinction is made, it becomes more straightforward for teachers to create suitable writing assignments for assessing critical thinking.

Once the course learning goals and writing assignment types are determined, the next step is to define appropriate grading criteria.

These criteria must align with the targeted critical thinking skills

At the same time, they should encourage students to critically evaluate the weaknesses and limitations of current tools involving Large Language Models.

To enhance teachers' professional development, we conducted three workshops to explain the design dimensions of the framework and address teachers' learning questions

We used cases that were relevant to the participants' respective disciplines to demonstrate how this framework can be applied to redesign their writing assignments.

Our observations indicate that the participating teachers responded positively to this framework

In the future studies, we will assess the effectiveness of teachers' implementation of this framework in their self-redesign process.