Challenging chats: Exploring AI Literacy on a Pre-sessional Course

 Introduction

This article was co-written by four international post-graduate students – Natt, Haoyuan, Yen-En, and Liming – and their pre-sessional teacher, Martha. It presents the students’ evolving perspectives on the use of Artificial Intelligence (AI), specifically ChatGPT. They share their journey from casual engagement with AI to recognizing its value as a study aid. The article also highlights the teacher’s research project, which aimed to understand how to develop the students’ AI literacy on their 2023 pre-sessional course.

Students’ perspective

For us, ChatGPT has changed the way we study. At first, we treated ChatGPT more like a pen pal and asked it silly, random questions for fun rather than taking it seriously. But then we found it was a useful study tool. We started using it for research instead of Google since, surprisingly, it was more capable than we initially imagined. What’s more, its responses were immediate, so it was as if we were having a conversation with a real person. This meant that it served as a valuable resource when private tutors or friends were unavailable.

However, some of our undergraduate classmates in our home countries used ChatGPT to generate their entire summative dissertations. In addition to serious issues of over-reliance and plagiarism, what else might choices like these indicate? Our PS class were asked why they have used AI when studying in the past, and almost half chose the option: ‘I did not believe I could do a good job on my assignment’ (Figure 1). Consequently, we want guidance from our teachers on how to use ChatGPT to enhance our study skills so we choose to use it as an empowering tool rather than to escape daunting assignments.

Figure 1: Survey results showing students’ reasons for using AI during past studies.

Teacher’s perspective

While I had presumed my students this summer would have used ChatGPT, I did not know to what extent. Had they used it as a research aid or to write entire essays? Were they aware of its limitations? In short: were they AI literate? To what degree? Would I be able to help them develop this skill?

It felt important to answer these questions. ChatGPT is used by university students worldwide, whether ethically or not, to help them complete assignments (Chan and Hu, 2023). I hoped that by helping my students to use it in class, I could provide them with a study tool that would support their success on their postgraduate courses. Consequently, this article outlines the research project four of my students and I undertook to investigate learners’ needs as regards AI literacy. We believe that while there are legitimate concerns about AI in higher education, embracing its affordances is not only exciting but essential. ChatGPT is not going anywhere (Dergaa et al, 2023), and as the often-cited phrase goes: ‘AI won’t steal your job; someone who uses AI will’ (Bromley, 2023). Students know this, so how can their teachers help them?

The project

Although there are several publicly accessible AI programmes, this project investigates ChatGPT 3.5 only. All data was collected from a group of 14 post-graduate students on a 6-week on-campus pre-sessional course at the University of Bristol in 2023. The group comprised 9 Chinese nationals, 3 Thai and 2 Taiwanese, with IELTS scores averaging 6.5. Data was collected via surveys, completed class worksheets and student-led interviews.

This article reports on the main themes emerging from student-led interviews, before making practical recommendations for two tasks which teachers can use in class to support their students’ AI literacy.

Student-led focus group

In this section, we (the students) report back on interviews we conducted with ten of our peers to find out about their uses and concerns regarding ChatGPT.

Main uses of ChatGPT

It was clear that our peers mainly use ChatGPT as a comprehension aid. For example, they ask it to summarize long academic texts or to explain the text in simpler language. They also use it to check answers to comprehension questions, by comparing its output with their own answers to measure the extent of their understanding. Finally, our peers also find it useful for generating ideas, such as making suggestions for solutions during a Problem-Based Learning task.

Common concerns

One of the main issues worrying our peers was privacy, which is a problem that most of us had not initially considered. Mitigating privacy risks is a key concern for the integration of AI as a study tool in HE (Chan and Hu, 2023). Our peers reported they were not sure how much of what they pasted into ChatGPT would be stored, or how it could be used. Being more informed of this would help us to use the AI tool confidently and responsibly.

Secondly, our peers were also concerned about plagiarism, which often prevents us from using or learning how to use ChatGPT. Since the database of ChatGPT is based on a wide variety of sources, our peers are unsure if or how to attribute it in their work, and which uses of ChatGPT are considered plagiarism. For the same reason, they were also concerned about the reliability and trustworthiness of the information generated by ChatGPT.

Our peers’ final concern, and one of the most frequently mentioned, was how to write effective prompts in order to generate useful output from ChatGPT. They had found that if given ambiguous questions, ChatGPT did not provide the responses they wanted. Through experimenting with prompts in class, our peers had learnt that using detailed questions with clear context and parameters generally led to the most useful output. However, they still wanted to develop their ability and confidence in doing so.

Our interpretation

We consider ChatGPT to be a powerful tool to consolidate our academic work. However, as we become increasingly reliant on this virtual creature, we risk losing a multitude of academic research skills and our ability to think critically. That is why educators could play a significant role in guiding students how to use ChatGPT wisely by shining a light on issues such as data protection, plagiarism and reliability. In addition, the interviews indicate a need for teachers to help us with being able to write useful prompts so we can use ChatGPT as an assistant to improve our academic skills instead of weakening them. To avoid over-reliance, the teacher might impose limitations on the usage of ChatGPT, such as using it only for certain aspects of the research-writing process and not others.

Recommendations

Below are some practical ideas to support your students’ AI literacy, based on the research described above. Both activities were created by teachers using ChatGPT.

  1. Raise awareness of limitations

We recommend raising students’ awareness of ChatGPTs’ potential limitations at the beginning of a course. This is because there are important ethical considerations relating to data protection, bias and inaccurate content being generated (Chan & Hu, 2023). One useful activity is a True/False quiz for students to discuss in groups. This is the one used with this class:

  1. ChatGPT-3.5 creates its answers based on patterns it has learned from its training data. (T)
  2. ChatGPT-3.5 can retrieve famous books from its training data to provide information about them. (F)
  3. ChatGPT-3.5 can be trusted to produce objective responses. (F)
  4. ChatGPT-3.5 can write in different languages because it learned from texts in several languages. (T)
  5. ChatGPT-3.5 always provides correct information. (F)
  6. ChatGPT-3.5 is good at finding academic sources. (F)

Credit to Nicholas Maxwell for creating and sharing this quiz.

 

  1. Writing and evaluating prompts

A key aspect of using ChatGPT is being able to tell it what you want effectively. Termed ‘prompt engineering’, this is now an important academic skill that students need (Wallbank, 2023). During the pre-sessional, the class experimented with using ChatGPT as a reading aid by asking it to summarise long academic texts, then adjusting the output with prompts such as:

– re-write this summary in bullet points

– re-write this summary at IELTS level 6.5

– re-write this summary so it’s easy to understand for a 16 year old

Students then revisited the text with a renewed understanding. ChatGPT helped them to engage in an iterative reading process in which it functioned as a personal ‘virtual tutor’ or stronger peer (Chan & Hu, 2023). In this way, Crompton and Burke (2023) suggest that it can help students work within Vygotsky’s (1978) Zone of Proximal Development (ZPD) because it provides them with personalised support in real-time, potentially leading to faster progress.

Wallbank (2023) suggests the acronym ‘TAP’ (Topic-Action-Parameters) to guide students writing prompts, and ‘TASTE’ (see Wallbank, ibid.) to evaluate and improve the prompts based on ChatGPT’s output. A more concise version we suggest is ‘DRAFT’, described in the table below, which reminds students of key characteristics of effective prompts.

This table could form the basis of various lesson activities, for example: the ‘explanations’ could be removed from the first column for students to match to each example; students could use it to peer review each other’s prompts; or students could use it to improve further examples and then to create their own prompts for an assignment they are working on.

‘DRAFT’ prompt-writing table

Step Explanation Good example Bad example
D: Define the topic Say exactly what you want to find out about. Explain the main causes of climate change, and its consequences for future generations. Tell me about the environment.
R: Request the action Tell the AI what you want it to do using action words eg summarize; explain; evaluate. Evaluate the advantages and disadvantages of social media use on teenagers in the UK. Write something about social media.
A: Add Parameters Give any extra details like the length of format of the answer. Write a discursive essay of 500 words on the advantages and disadvantages of social media use among teenagers in the UK. Include three academic references using these sources: “XYZ”. Write an essay about social media.
F: Fine-Tune and Test Try different ways of asking, make it simple, and check if the answer is correct. Explain the main cultural differences between China and the UK, including one example for each difference. What are the differences between China and Bristol?
T: Think Critically Look closely at the answer, and compare it with what you can find from trusted sources. Compare the answer above with the information here to check for accuracy “copy-paste extract from reliable source”. Is this information correct?

Adapted by Martha Partridge and Nicholas Maxwell from Wallbank (2023)

 

Conclusion

In conclusion, over the course of the Pre-sessional we (the students and the teacher) learnt the value of ChatGPT as a study-support tool. We also experimented with ways that teachers can develop their students’ AI literacy so they use it in a responsible way to enhance rather than diminish their academic skills. Through open conversation, awareness of its limitations and curiosity about its affordances, ChatGPT can be used effectively to help students be more autonomous and effective in their HE studies.

From the students’ perspectives, future experimentation will involve identifying the best uses of ChatGPT on a post-graduate course. From the teacher’s point of view, it will be useful to research the best uses of ChatGPT for students on an International Foundation Programme.

Acknowledgments

All students in class 6.21 on the Pre-sessional 2023 course, who provided their valuable insights into the use of ChatGPT.

In particular, Natt, Haoyuan, Yen-En and Liming for conducting interviews with their peers and co-writing this article.

Nick Maxwell for contributing the True/False quiz and collaborating on the ‘DRAFT’ table.

 

References

Bromley, J. 2023. I used ChatGPT to research my dissertation — here’s why it’s fine. Evening Standard, [online] 2 March 2023. Available at: https://www.standard.co.uk/insider/chatgpt-dissertation-essay-education-gen-z-b1055820.html [Accessed 2nd October 2023]

Chan, C.K.Y. and Hu, W., 2023. Students’ Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Education. arXiv preprint arXiv:2305.00290

Crompton, H. and Burke, D., 2023. Artificial intelligence in higher education: the state of the field. International Journal of Educational Technology in Higher Education, 20(1), pp.1-22.

Dergaa, I., Chamari, K., Zmijewski, P. and Saad, H.B., 2023. From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40(2), pp.615-622.

Vygotsky, L.S. and Cole, M., 1978. Mind in society: Development of higher psychological processes. Harvard university press.

Wallbank, A. J., 2023. Prompt engineering as academic skill: a model for effective ChatGPT interactions. Times Higher Education, [online]. Available at: https://www.timeshighereducation.com/campus/prompt-engineering-academic-skill-model-effective-chatgpt-interactions [Accessed 2nd October 2023]

2 thoughts on “Challenging chats: Exploring AI Literacy on a Pre-sessional Course

  1. Natt, Haoyuan, YenEn, Liming and Martha, thank you for reporting on your highly relevant and interesting experience with AI on the Pre- sessional. I’m sure it will play an important role informing the work moving forward on the programme development side. Something you mention, which hadn’t occured to me yet in terms of using AI, was the issue of privacy. I look forward to reading the Chan and Hu reference to find out more.

    Thanks again for this timely blog post.

  2. This is a great insight into useful and practical responses to use of AI in the PS6 learning environment. Great to read this co-written blog Martha and our lovely students, Natt, Haoyuan, YenEn, Liming. Well done!

Leave a Reply

Your email address will not be published. Required fields are marked *