Prompt: ChatGPT, Create My Course, Please!

4. Results

RQ1: What are the students’ perceptions of the quality and relevance of the course content when they are unaware that it is AI-designed?

The themes of the data analysis were divided into four categories: (1) attitudes and perceptions of the course materials, (2) noticed changes in the course materials, (3) competency in AI application, and (4) overall course experience.

Regarding their attitudes and perceptions of the course materials, all the participants agreed that the content was well-organized and easy to understand. As such, their attitudes and perceptions of the course materials were positive overall. Some examples are mentioned in their reflection notes:

In terms of curriculum design, the content was designed to stimulate motivation or actual interest before the lesson compared to previous sessions. This content increased interest and focus in the lesson, and the connection to the lesson seemed seamless (extracted from Teacher B’s reflection notes).

With respect to delivery methods, the content matches well with the expectations of every lesson. They contain the main points briefly and in an understandable way (extracted from Teacher C’s reflection notes).

Most participants liked how the professor conveyed the course materials during the interviews. Here is another example.

The professor provided sufficient information in the course on the PPT slides, making it easy for me to catch up. My English was not good, so I thought it could be challenging to follow the lecture at the beginning of the semester. However, I could receive sufficient background information from the course materials, so it was good for me to follow up (Teacher C, second interview).

2.

Noticed changes in the course materials

Among the three participants, Teacher A noticed critical changes in the course materials during the semester. Specifically, Teacher A noted the changes in the content displayed because she thought the first and second halves of the PPT slides looked very different. The following is the part of the transcript from her second interview.

In the second half of the course, I noticed that the PPT materials were changed. In the first half of the semester, I felt like the professor made those PPT slides by himself. However, in the second half of the semester, the content display and formatting were changed and different. Thus, I thought the professor used the AI tool for course PPT slides for reasons… I’m not too fond of the second half of the PPT slides because… AI tools created those (Teacher A, second interview).

In the reflection note, Teacher A mentioned the following:

Week 11 and 12: The PPTs could disrupt the course flow and make it harder for me to grasp the material. The PPTs seemed to be made by AI, not by my professor (extracted from Teacher A’s reflection notes).

On the other hand, the other two participants did not notice changes in the content displayed in the course materials. The authors mentioned that the background of the PPT slides was slightly changed, but the content display and delivery methods were not changed.

In the second half of the semester, the background and format of the PPT slides were changed, but this did not impact my understanding of the content. Additionally, the professor told us that he was using the AI tool to create the PPT background, so this was okay (Teacher B, second interview).

The first half of the semester is mainly arranged with sentences. The second half of the semester consists of more diverse and concise designs and pictures, which are good for understanding. Both designs are appropriate for the curriculum because, in the early phase of the semester, we need to familiarize ourselves with the lesson, so we need more detailed written explanations (extracted from Teacher C’s reflection notes).

It is concluded that each participant has different opinions and experiences of changes in content and materials based on their understanding and experiences of the course.

3.

Their confidence in AI applications

All three participants mentioned that their self-confidence could be enhanced when comparing their competency level in the early semester and at the end of the semester. However, previous backgrounds and skills related to AI impact the actual ability of AI applications in their classrooms. Specifically, participants with previous background knowledge and skills in AI, such as Teacher C, showed high competency.

I am fairly confident in working with AI applications. I would say 8 out of 10 scale in competency levels. My previous experience developing apps and tools helped me get better ideas for what I will do for my future research projects. I am working on making apps to help university applications for high school kids, so the course helped me build new ideas and make them real for final assignments. I am enjoying it now (Teacher C, second interview).

However, other participants who lacked previous background knowledge and skills in AI showed low competency. For instance, Teacher B, who has low competency, mentioned in the interview:

I thought I was at the beginner level of AI application, so my competency in AI application was not that high. What I can do is create a lesson plan that incorporates AI components more effectively. However, I need somebody’s help making AI technology in real practice. So, I wish I could create something new, like app development for the final project assignment, but it was challenging now (Teacher B, second interview).

Some teachers’ reflection notes also showed a lack of competency in practical AI applications.

I can understand what AI is, how it works, and how it can have an impact on society. However, as a teacher, I still struggle to design and apply AI systems as pedagogical tools for my lessons (extracted from Teacher A’s reflection notes).

It is concluded that based on the previous knowledge, expertise, and experience of AI applications, each individual participant’s competency in AI applications could differ.

4.

Overall experiences while taking the course

Regarding overall course experiences, all participants agreed that the course environment was learner-friendly and encouraged a collaborative learning process. Additionally, the professor effectively facilitated the participants’ in-class activity and participation. Thus, all of them are quite satisfied with the course. Some examples are below.

I liked the classroom atmosphere because we were working together during the course, and it was like American higher education with a small number of graduate students. I prefer this kind of course format because I do not like lecture-only classes in the programs with other courses. I like to receive hands-on activities and assignments that I could use for my teaching classrooms (Teacher A, second interview).

This course was one of the practical courses we took from teacher preparation programs. We took the lecture part and had enough chance to work on collaborative or individual projects afterward. It was an encouraging learning environment that we all enjoyed (Teacher B, second interview).

RQ2. What are the affordances and limitations of AI-developed courses and lesson designs?

All course information, including course topics, lesson content, midterm projects, and final projects, was created with AI. Although the content was created by AI, there was an iterative process of refinement through prompts to elaborate on the existing information. This process concentrated on developing existing information rather than explicitly requesting new information that the instructor identified as potentially relevant but not initially provided by AI.