The Impact of AI-Based Course-Recommender System on Students’ Course-Selection Decision-Making Process

Abstract

The course-recommender system (CRS), designed to aid students’ course-selection decision-making process by suggesting courses aligned with their interests and grades, plays a crucial role in fulfilling curricular requirements, enhancing career opportunities, and fostering intellectual growth. Recent advancements in artificial intelligence (AI) have empowered CRSs to deliver personalized recommendations by considering individual contexts. However, the impact of AI-based CRS on students’ course-selection decision-making process (inter alia, search and evaluation phases) is an open question. Understanding student perceptions and expectations of AI-based CRSs is key to optimizing their decision-making process in course selection. For this purpose, we employed speed dating with storyboards to gather insights from 24 students on five different types of AI-based CRS. The results revealed that students expected AI-based CRSs to play an assistive role in the search phase, helping them efficiently complete time-consuming search tasks in less time. Conversely, during the evaluation phase, students expected AI-based CRSs to play a leading role as a benchmark to address their uncertainty about course suitability, learning value, and serendipity. These findings underscore the adaptive nature of AI-based CRSs, which adjust according to the intricacies of students’ course-selection decision-making process, fostering fruitful collaboration between students and AI.

1. Introduction

In higher education, university students’ course-selection decision-making is an important process that has a profound impact on their curricular requirements, career opportunities, and intellectual growth [1,2]. However, students often grapple with the challenge of navigating course listings, overwhelmed by the abundance of information related to each course, making it difficult to quantitatively assess which course aligns best with their needs and aspirations [3]. Although academic advisors can help students select appropriate courses, it is difficult to give this support to all students as it requires a lot of time and effort from the human advisor [4]. In this context, the implementation of artificial intelligence (AI)-based course-recommender systems (CRS) emerges as a solution to these issues [5]. These systems can automatically provide personalized course recommendations to individual students in real time, leveraging their historical data [6,7].

Historically, CRSs have predominantly relied on conventional data-driven recommendation algorithms, including content-based, collaborative filtering-based, and hybrid approaches [8]. While effective in maintaining students’ existing course-selection patterns by offering data-driven recommendations, these approaches are not without their shortcomings [9]. They can inadvertently introduce problems like filter bubbles and algorithmic bias, potentially preventing students from discovering valuable yet unpopular courses that could enrich their intellectual growth [10,11]. To prevent these issues and ensure a more balanced course-selection decision-making process, there is a growing need for a new breed of AI-based CRS that departs from the traditional data-driven model.

In recent times, efforts have intensified to address these limitations by integrating Large Language Models (LLMs), like ChatGPT, into recommender systems and exploring novel recommendation methodologies [12,13]. LLM-powered recommender systems have the potential to decipher intricate patterns within students’ textual interactions, discern nuanced preferences, and align with their learning objectives [14,15]. Leveraging the capabilities of LLM in recommender systems is anticipated to deliver adaptive and context-aware recommendations, therefore tackling the challenges associated with evolving student needs and mitigating issues related to a cold start and data sparsity [16,17]. However, it is equally important to consider how these systems can foster meaningful collaboration between students and AI, with a focus on user-centered design [18,19]. Establishing a harmonious partnership between students and AI in the realm of education is critical for ensuring a seamless and effective personalized learning experience that aligns with the diverse needs of students [20]. From this perspective, it is important to understand the impact of AI-based CRSs on students’ course-selection decision-making process [21].

To address this need, we employed the speed dating research method [22] to expose students to different types of AI-based CRSs and subsequently assess their perceptions and expectations of AI-based CRSs during the course-selection decision-making process [23]. The results of the speed dating activity were analyzed through a thematic analysis approach [24]. We found that students wanted AI-based CRSs to retrieve, filter, and visualize high-quality information related to their courses during the course-search phase and quantify difficult-to-evaluate qualitative information during the course-evaluation phase. These findings showed that students expect AI-based CRSs to play different roles at each phase of the course-selection decision-making process.

This paper is organized as follows: Section 2 delves into the course-selection decision-making process and reviews literature related to AI-based CRSs. Section 3 outlines the experimental procedure employing the speed dating research method. Section 4 sheds light on what students expect from AI-based CRSs at different stages of the course-selection decision-making process. Finally, Section 5 presents insights into the future of AI-based CRSs, aiming to assist students in their course-selection decisions based on our research findings.

2. Related Work

This paper explores the impact of AI-based CRSs on students’ course-selection decision-making process. Building upon existing research, we established a theoretical framework to analyze students’ course-selection decision-making process. We then reviewed AI-based CRSs currently used in higher education.

2.1. Course-Selection Decision-Making Process

In higher education, a student’s course-selection decision-making involves the process of gathering course-related information and determining whether to enroll in it or not [25]. This process comprises several stages: recognizing educational needs, searching for relevant information, evaluating available alternatives, and ultimately deciding on a course [26]. With the advent of the digital age, the widespread use of social media has added complexity to how students acquire, compare, and opt for courses [27]. Othman and colleagues [1] have structured university students’ course-selection decision-making process into a three-phase framework, which includes the search, evaluation, and choice/decision phases, each outlined in

Table 1. Three phases of the course-selection decision-making process of university students adapted from Othman et al. (2019) in ref. [28].

Course-Selection Decision-Making Process	Description
Search phase	Students search for information regarding the course through their peers, seniors, and word of mouth
Evaluation phase	Students evaluate which course they will enroll in, considering the costs and benefits based on the information they have gathered in the search phase
Choice/decision phase	Students make choices depending on the information that they have evaluated in the evaluation phase

.

During the search phase, students collect course-related information such as course content, instructor reputation, scheduling details, and more from various sources. These sources include syllabi, course descriptions, peer reviews from fellow students [29], and word-of-mouth recommendations. Moving on to the evaluation phase, students compare courses based on criteria such as expected workload, educational value, level of difficulty, and personal interest [26]. Social media also emerges as a significant resource during this phase, aiding students in assessing and comparing courses [27]. Finally, in the choice/decision phase, students make conclusive decisions about which courses to enroll in based on their assessments and evaluations.

The successful decisions students make when selecting courses are of paramount importance as they guarantee the fulfillment of academic requirements, enhance career prospects, and foster intellectual growth [1,2]. Recently, the research community has proposed CRSs as a remedy to support students in making effective and efficient course selections, therefore empowering them [30]. In particular, there has been considerable focus on utilizing automated AI applications for course recommendations within the realm of education [31].

2.2. AI-Based Course-Recommender Systems

AI-based CRSs have long been studied for their potential to aid students in making informed course-selection decisions tailored to their academic paths. Many universities now deploy AI-based CRSs; for instance, the University of California, Berkeley, offers a system named AskOski [32]. This system assists students in exploring their interests, linking course concepts across departments, and planning future semesters in accordance with their program requirements. Recent literature [7] highlights a multitude of recommendation techniques—ranging from collaborative filtering to knowledge representation, fuzzy logic, and various machine learning algorithms—along with input parameters such as learning style, objectives, preferences, and academic information, all tailored to cater to the diverse educational needs of students. For example, collaborative filtering, widely employed in AI-based CRSs, suggests courses based on community records of peers with similar characteristics [33,34]. Similarly, content-based filtering techniques aid in recommending courses aligning with students’ academic interests [35,36]. Moreover, AI-based CRSs extend their utility to suggest courses facilitating the acquisition of competencies relevant to students’ future careers [37,38]. Conversely, some AI-based CRSs focus on optimizing academic outcomes by identifying courses with the highest probability of yielding favorable results and recommending them [39]. Additionally, AI-based CRSs explore students’ latent interests by suggesting courses that are unexpected yet intriguing—a concept often termed serendipity [40].

Although various forms of AI-based CRSs have been developed, their impact in adequately supporting students’ course-selection decision-making process remains uncertain. The crux lies in comprehending students’ perceptions of these AI-based CRSs and pinpointing areas where their expectations are not met during the course-selection decision-making journey. In this context, Zawacki-Richter and colleagues [31] emphasized the necessity of investigating how real users (i.e., students) perceive AI applications within higher education. Similarly, Lee and colleagues [41] discovered that students emphasize academic interests in choosing elective courses and require clear keyword information about courses due to often vague titles. Nevertheless, gaps persist in our comprehension of how AI-based CRSs influence students’ decision-making processes. To address these concerns, we employed speed dating with storyboards, an exploratory research method [22], to expose students to the aforementioned AI-based CRSs [42]. This approach facilitated a deeper exploration of how students envision AI-based CRSs augmenting their course-selection decision-making process at each phase.

3. Materials and Methods

In this study, we employed the speed dating research method, where students are presented with potential future scenarios through a four-cut storyboard [22]. This approach is valuable as it allows students to explore various AI-based CRS scenarios without the need for actual implementation [42]. By exposing students to diverse AI-based CRSs, we aimed to uncover their perceptions and expectations at different phases of the course-selection decision-making process, including the search and evaluation phases [23]. Through this investigation, we sought to discern the impact of AI-based CRSs on students’ course-selection decision-making process. Overall, we address the following research question: “What are the perceptions and expectations of students regarding AI-based CRSs during their course-selection decision-making process, specifically in the search and evaluation phases?”

3.1. Creating Storyboards

To develop technically viable and effective storyboards for AI-based CRSs in higher education, we organized an online brainwriting session [43]. This session involved four researchers, each a faculty member with an average of 13.3 years (SD = 5.4 years) of experience in learning science. They collaboratively developed scenarios for potential AI-based CRSs using a Google Slides file, building upon previous studies outlined in Section 2. The scenarios were iteratively exchanged and refined among the researchers seven times, ensuring a consensus on their technical feasibility and potential effectiveness in course recommendations.

Furthermore, to validate these initial scenarios, we conducted semi-structured interviews with six AI experts who had an average of 18.6 years (SD = 6.3 years) of research experience. These interviews were facilitated via Zoom by the corresponding author. Each scenario was presented to the AI experts, who were then prompted with the questions: “Could you enhance this scenario to ensure its technical feasibility?” and “Could you refine this scenario to enhance course recommendations for students based on your AI research experience?” Following the presentation of all scenarios, the AI experts were asked: “Do you have any suggestions for new scenarios?” Adjustments were made to the original scenarios based on the feedback received, aligning them with expert insights. On average, these interviews lasted 32 min (SD = 5.6 min), and each expert received a compensation of 100 USD for their participation.

As shown in

Table 2. AI-based CRS scenario titles and summaries.

ID	Scenario Title	Scenario Summary
1	Community-based CRS	AI recommends courses using implicit and explicit feedback from students’ assessments of the courses
2	Academic interest-based CRS	AI recommends courses whose content most closely matches students’ academic interests
3	Career interest-based CRS	AI recommends pertinent skills and relevant courses based on students’ career interests
4	Grade prediction-based CRS	AI recommends top-n courses in which students would perform well by predicting grades in the next semester based on their course enrollment patterns
5	Serendipity-based CRS	AI recommends serendipitous courses that are novel or unexpected but still relevant to individual students’ interests

, five scenarios outlining AI-based CRSs for guiding students through the course-selection decision-making process, including the search and evaluation phases, were derived. Four scenarios (Scenarios 1, 2, 3, and 5) have well reflected the representative AI-based CRSs identified in Section 2. The Serendipity-based CRS (Scenario 4) was created based on insights from AI experts’ research. It is important to note that these five scenarios were not intended to comprehensively cover all AI-based CRSs in higher education or to systematically address all related topics. Rather, their purpose was to delve into students’ course-selection decision-making processes and their perceptions and expectations of AI-based CRSs.

We generated four-cut storyboards based on the scenarios presented in Table 2. As demonstrated in Figure 1, an example storyboard is shown, providing a visual depiction of a scenario with accompanying captions. To address concerns related to gender and ethnic biases and to enhance participants’ identification with the characters, we employed a uniform visual style characterized by flat cartoon shading [23,42]. For a comprehensive repository of AI-based CRS storyboards, please visit https://osf.io/bwg7c/?view_only=8b5122454ced444fa53fad3940f929d6 (accessed on 7 March 2024).

3.2. Speed Dating

3.2.1. Participants

Next, we conducted a speed dating activity using storyboards. For a speed dating activity, we recruited 24 participants (see

Table 3. Summary of the students’ information.

ID	Major	Year of Study	Age	Gender
S1	Computer Science	1	20	M
S2	Biochemical Engineering	4	24	F
S3	Electronics Engineering	2	23	M
S4	Information Engineering	4	26	M
S5	Computer Science	1	20	F
S6	Electronics	2	24	M
S7	Information Science	1	22	F
S8	Information Engineering	4	25	M
S9	Electronics	2	24	M
S10	Electronics	1	21	F
S11	Physics	1	20	M
S12	Statistics	2	23	M
S13	Russian	1	20	F
S14	Education	3	22	F
S15	Media and Arts	2	22	F
S16	Computer Science	1	22	M
S17	Airlines Service	2	22	F
S18	Information Security	3	22	F
S19	Economics	3	22	F
S20	Media Engineering	3	22	F
S21	Management Engineering	2	22	F
S22	Education	2	22	F
S23	Law	3	22	F
S24	Nursing	2	21	F

) from the university community. For diversity, we recruited students from different grades (i.e., seven freshmen, nine sophomores, five juniors, and three seniors) and 16 different majors (electronics, nursing, education, economics, and computer science). We required participants to have at least two semesters of course-selection decision-making experience at the university to maintain consistency in expectations and experiences regarding course recommendations. However, we did not require any prior knowledge of AI from the participants, as our aim was to capture their unfiltered experiences and expectations for AI-based CRSs. Additionally, previous studies showed that speed dating activity works well without prior knowledge or experience with AI [42]. Prior to participation, informed consent was obtained from each individual, and participants were remunerated with a coffee voucher worth 10 USD for their time. The Institutional Review Board at Seoul National University of Science and Technology approved the entire process.

3.2.2. Procedure

Figure 2 illustrates the comprehensive process of our experiment. We conducted semi-structured interviews with participants through a video conferencing platform (i.e., Zoom). We briefly introduced the purpose of this study and started speed dating activity using storyboards. We showed participants each storyboard in random order. The storyboard describes possible interaction where an AI-based CRS supports a student’s course-selection decision. This setup allowed participants to indirectly experience how five distinct types of AI-based CRSs facilitate decision-making. After participants read each storyboard aloud, we asked the following questions to investigate their thoughts on specific AI-based CRS represented in individual storyboards: “How do you think this AI-based CRS will affect your course-selection decision-making?” and “Do you have any expectations or concerns about this AI-based CRS?” After participants reviewed all five storyboards, wrap-up questions were asked to find out general expectations for AI-based CRSs: “Which AI-based CRS do you think will help students’ course-selection decision-making?” and “How do you think AI-based CRS will change students’ course-selection decision-making?” All the questions were open-ended. Furthermore, we asked follow-up questions such as “Why did you think this scenario would help your course-selection decision-making?” after initial responses were given. These probing questions aimed to deepen participant engagement and elicit more detailed insights. The entire interview lasted around 30.1 min (SD = 6.9 min), with 4–6 min spent sharing each storyboard and probing participants on its specific implications.

3.2.3. Data Analysis

Each interview was audio recorded and then transcribed using software (i.e., NAVER CLOVA Note 1.9.0) for analysis. The transcribed interview data were analyzed by two authors (first author and corresponding author) using the thematic analysis approach [24]. After becoming familiar with the interview data, the first and corresponding authors generated an initial set of semantic codes associated with interesting statements or phrases from the data. The two authors then coded the entire interview dataset via Google Docs to identify additional patterns through an extended investigation of the dataset. Any conflicts that occurred during this coding process were resolved through discussions between the two authors. The two authors came up with 12 final codes through a total of four discussions. Then, all three authors had three iterative discussions to derive six final themes from 12 codes. These themes, which represent students’ perceptions and expectations of AI-based CRSs, were categorized according to the students’ phase of the course-selection decision-making process [1]. Specifically, in the course-search phase, students expressed expectations for (1) Subject-based course retrieval, (2) Social information filtering, and (3) Course-related text visualization from the AI-based CRS. Conversely, in the course-evaluation phase, students anticipated (4) Course suitability analysis, (5) Value proposition, and (6) Serendipity.

4. Results

Our findings revealed that students prefer to actively engage with the AI-based CRS, utilizing it for both information search and evaluation rather than merely receiving passive course recommendations [28,44]. Specifically, during the search phase, students anticipate that the AI-based CRS will effectively retrieve, filter, and visualize high-quality course-related information, therefore enhancing their personalized course searches. In the evaluation phase, students expect the AI-based CRS to quantitatively assess difficult-to-evaluate qualitative aspects such as course suitability, learning value, and serendipity.

Table 4. Students’ expectations and response to the AI-based CRSs in each phase of the course-selection decision-making process.

Course-Selection Decision-Making Process	Students’ Expectations for AI-Based CRS	Summary of Students’ Responses
Search phase	Subject-based course retrieval	Students reported the need for ‘subject-based course retrieval’ to make it easier to find courses they want to search for
	Social information filtering	Students responded that they would like the CRS to filter out untrustworthy social information related to courses
	Course-related text visualization	Students wanted the CRS to use natural language processing technology to analyze course-related text information and visualize the results
Evaluation phase	Course suitability analysis	Students hoped to be able to quantitatively compare courses suitable for them in various aspects among the courses selected through the search phase
	Value proposition	Students wanted to be recommended a course if it was valuable to them for their future careers or to improve their skills
	Serendipity	Students wanted the CRS to consider unexpected fun (i.e., serendipity) when evaluating courses

provides a summary of students’ expectations and responses to the AI-based CRSs at each phase of the course-selection decision-making process.

4.1. Search Phase

During the course-selection decision-making process, the search phase involves students searching for information about courses from various sources such as syllabi, course descriptions, peer reviews, and word of mouth. In this phase, students anticipate that the AI-based CRS will assist them in efficiently locating desired courses, filtering out unreliable social information pertaining to the courses, and analyzing and visualizing textual information, as outlined below.

4.1.1. Subject-Based Course Retrieval

Students emphasized the necessity for ‘subject-based course retrieval’ to streamline the process of finding desired courses. Presently, most course enrollment systems rely on title-based course retrieval methods, which students find limiting. As one participant noted, “the title and content of the course often do not match (S23)”, making it challenging to identify courses solely based on titles. Another student (S24) expressed frustration, stating, “It is difficult to find a course of interest only by looking at the title alone”. Similarly, students S13 and S17 recounted instances where they mistakenly enrolled in courses with content vastly different from their expectations based on the course title. To avoid such errors, students S15 and S20 stressed the importance of thoroughly reading through the course syllabus. Consequently, most students (17 out of 24) expressed their expectation that an AI-based CRS offering subject-based course retrieval would significantly streamline the search process, saving them considerable time and effort in identifying courses of interest.

4.1.2. Social Information Filtering

Students expressed a desire for the AI-based CRS to effectively filter out unreliable social information pertaining to courses. While many students (20 out of 24) rely on social cues such as word of mouth or peer assessments when searching for courses, they harbor concerns regarding the reliability of such information. For instance, one student (S6) voiced apprehension, noting that while some students provide genuine course reviews, others may offer unhelpful or misleading information. Additionally, student S21 cautioned against bias in social information, highlighting instances where negative assessments might stem from personal grievances rather than objective evaluations. Students anticipated that if the AI-based CRS could effectively screen out untrustworthy social information, they would be less likely to be misled during their course searches.

4.1.3. Course-Related Text Visualization

Students expressed a desire for the AI-based CRS to utilize natural language processing technology to analyze course-related text information and present the results in a visual format. Many students (14 out of 24) voiced concerns about the overwhelming volume of textual information they encountered during the course-search process, including details such as course objectives, teaching methods, assessment criteria, and prerequisites. One student (S14) remarked, “It is difficult to comprehend course-related information because most of it consists of lengthy text passages”. In response to this challenge, another student (S20) suggested that if AI could transform complex textual information into visual representations such as tables or figures, it would significantly facilitate the course-search process.

4.2. Evaluation Phase

In the course-selection decision-making process, the evaluation phase involves students assessing the courses in which they intend to enroll and weighing the costs and benefits based on the information gathered during the search phase [1]. During this phase, students envisioned the AI-based CRS serving as a benchmark to quantitatively compare the suitability, value, and serendipitous elements of courses that are challenging to evaluate independently.

4.2.1. Course Suitability Analysis

Students expressed a desire to quantitatively compare the suitability of various courses selected during the search phase across multiple dimensions. Several students (11 out of 24) emphasized that they did not want a course to be recommended solely based on its popularity among peers or its high ratings from historical data. Instead, they wished for the AI-based CRS to quantitatively evaluate the suitability of courses for them based on factors such as their background knowledge (S20), the instructor’s characteristics (S19), preferred lecture style (S14), and personal learning preferences (S15). In particular, student S20 believed that a quantitative analysis of these qualitative factors would greatly assist in making objective decisions about which course to select from the available options.

4.2.2. Value Proposition

Even if a course may not align perfectly with their immediate needs, students expressed a desire to receive recommendations for courses that hold value for their future careers or skill enhancement. Some students (9 out of 24) highlighted the challenge of quantitatively assessing the value of specific courses. Consequently, students emphasized the importance of AI-based CRSs being able to quantify the potential benefits of a particular course for their future career trajectories and skill development. For example, student S8 remarked, “Even if I have a clear career goal, it’s difficult to determine which courses are necessary for staying competitive. In this regard, it would be beneficial if AI could guide me on the competencies I can gain from specific courses”. Regarding skill enhancement, student S4 noted, “If AI can indicate the value of prerequisite courses essential for advanced upper-level courses, it would aid in selecting courses at lower levels”. Students anticipated that this value proposition would assist them in making informed decisions, even if the recommended courses were not their top preferences.

4.2.3. Serendipity

Students expressed a desire for the AI-based CRS to consider unexpected enjoyment, known as serendipity, when evaluating courses. Several students (13 out of 24) emphasized the importance of not only evaluating the suitability and value of a course but also considering serendipitous opportunities. For instance, students anticipated that serendipitous course recommendations could “provide an opportunity to expand one’s interests and explore new areas (S12)” or even facilitate “the discovery of previously unknown areas of interest (S16 and S21)”.

In conclusion, students anticipate that AI-based CRSs will assist them in navigating extensive course lists by offering personalized course searches through features like retrieving, filtering, and visualizing high-quality course-related information. Additionally, students envision AI-based CRSs playing a pivotal role in facilitating their final course-selection decisions by evaluating and comparing nuanced factors such as suitability, learning value, and serendipitous enjoyment of potential courses. The theoretical and practical implications of these findings will be elaborated upon in the following section.

4.3. Students’ Preferences

Table 5. Students’ preferences for different types of AI-based CRS.

ID	Scenario Title	Preferred Students	Total
1	Community-based CRS	S1, S2, S3, S5, S7, S11, S13, S16, S22, S24	10
2	Academic interest-based CRS	S1, S7, S8, S12, S14, S15, S22, S23	8
3	Career interest-based CRS	S2, S3, S4, S5, S6, S11, S13, S14, S16, S18, S23	11
4	Grade prediction-based CRS	S1, S7, S10, S17, S18	5
5	Serendipity-based CRS	S6, S9, S10, S16, S19, S20, S21, S24	8

illustrates the distribution of students’ preferences for various AI-based CRSs based on their responses to the question: “Which AI-based CRS do you think will help students’ course-selection decision-making?” Notably, approximately half of the students (11 out of 24) preferred the Career interest-based CRS, with the Community-based CRS ranking a close second, chosen by 10 students. Both the Academic interest-based CRS and Serendipity-based CRS were selected by 8 students each, whereas the Grade prediction-based CRS was the least favored, chosen by only 5 students. It is important to highlight that participants had the option to choose more than one CRS, resulting in a total of 42 preferences, which exceeds the actual number of participants (24).

5. Discussion and Conclusions

In this study, we explored how AI-based CRSs influence students’ course-selection process to identify the most effective support for their decision-making. In the search phase, students anticipated the AI-based CRS to retrieve, filter, and visualize course-related information to facilitate personalized course searches. In the evaluation phase, students expected the AI-based CRS to quantify difficult-to-evaluate qualitative information, including course suitability, learning value, and serendipity. These findings underscore the variability in students’ interactions with AI-based CRSs based on the particular phase of their course-selection decision-making process.

Interestingly, students anticipated a change in the role of the AI-based CRS corresponding to each phase of the course-selection decision-making process, namely the search and evaluation phases. For instance, research by Chang and colleagues [45] highlights a personalized hybrid CRS designed to dynamically adapt to the diverse needs of students throughout their course-selection journey. During the search phase, students envisioned the AI-based CRS playing an assistive role in gathering course-related information. This limited assistive role arose from concerns regarding the AI’s ability to fully grasp complex factors, such as academic interests and preferences, based solely on semiannual course history data. Students expressed skepticism about fully automated recommendations during this phase, fearing they might not accurately reflect their evolving academic contexts, reminiscent of the ‘cold-start problem’ in AI. Students believed that the search phase could be improved when the AI-based CRS assisted by retrieving, filtering, and visualizing relevant data from diverse, text-heavy sources, with students taking the lead in utilizing this information.

Conversely, during the evaluation phase, students envisioned the AI-based CRS taking on a more prominent role, serving as a benchmark for quantitatively analyzing difficult-to-evaluate information. This expectation stemmed from their difficulty in objectively weighing the advantages and disadvantages of various courses, which often eroded their confidence. Research by Chong and colleagues [46] suggests that individuals tend to lean more on AI recommendations when their confidence in decision-making wanes. Consequently, students who felt overwhelmed during the evaluation phase tended to rely more on AI suggestions, expecting the AI-based CRS to assume a primary role. Students believed that AI’s quantified assessments would assist them in overcoming uncertainties in assessing course suitability, learning value, and serendipity. In summary, to effectively and efficiently support students in course selection, the AI-based CRS must adapt its role throughout the students’ decision-making process, acting as an assistant during the search phase and as a leader in the evaluation phase.

As indicated in Table 5, the distribution of students’ preferences shows no discernible pattern, reflecting the diverse backgrounds and unique needs of each student. This observation underscores the significance of recognizing individual variations in preferences and needs when providing personalized course recommendations rather than adopting a one-size-fits-all approach. Consequently, this study emphasizes the necessity for future AI-based CRSs to incorporate these insights, enabling them to offer tailored course selections that align with the specific preferences and requirements of each student.

5.1. Theoretical Implications

This study offers theoretical insights into how students can cultivate a symbiotic relationship with AI-based CRSs throughout their course-selection decision-making process, encompassing both the search and evaluation phases [1]. While many studies have delved into the implementation of AI-based CRSs in university settings, most systems are tailored to the specific needs of individual institutions, leaving a gap in understanding the broader support required for student course-selection decisions. Setting itself apart from other technology-centric research, this study underscores the importance of AI-based CRSs adapting their function to align with each phase of the course-selection process, thus fostering effective collaboration between humans and AI [20]. Future research endeavors should explore how AI-based CRSs can recognize different stages in a student’s course-selection journey and adjust their role accordingly to enhance decision-making support.

5.2. Practical Implications

This study, taking into account advancements in LLMs like ChatGPT, presents several practical implications. Our research highlights the need for AI systems to go beyond merely reacting to explicit student requests and to engage proactively during both the search and evaluation phases to provide genuinely personalized support. First, in the course-search phase, AI-based CRSs could benefit from offering a chat-based interactive platform to assist students. Rather than simply presenting a list of courses, the system could engage in conversational interactions to elucidate course topics, therefore aiding students in autonomously discovering courses that align with their interests. Through these narrative-driven recommendations [17], the AI would serve an assistive role by guiding students through sifting through and rectifying misinformation, highlighting crucial aspects of courses, and summarizing key information, thus empowering students to navigate their course search autonomously.

Second, during the evaluation phase, the AI-based CRS should take on a more leading role, prioritizing explainability. Providing transparent and comprehensible explanations throughout the AI-assisted decision-making process can mitigate uncertainties and bolster confidence in the system [47]. For instance, in the scenario involving a Career interest-based CRS, the AI could explain to students: “This course is recommended because it aligns with your career interests in machine learning engineering, which could significantly benefit your future career path based on your current academic and extracurricular profile”. As students interact with the AI-based CRS’s reasoning during evaluations, they will have the chance to develop their own perspectives and improve their capacity to assess and select courses autonomously [48].

Thirdly, AI-based CRSs should provide unexpected or serendipitous course suggestions. This strategy tackles the filter bubble problem by urging students, especially those uncertain about their academic path, to venture into new domains [48]. These recommendations have the potential to expand students’ perspectives, exposing them to unfamiliar subjects and revealing interests they may not have previously considered. Future research endeavors should strive to identify the key characteristics of serendipitous recommendations and assess the capacity of LLMs to deliver such suggestions reliably and precisely, avoiding the propagation of misleading or erroneous information.

There are several limitations that should be considered when interpreting our findings. First, the outcomes may vary based on participants’ socio-demographic factors, such as their university affiliations, countries of residence, or other specific conditions. Future research should explore how these factors influence students’ preferences and expectations regarding AI-based CRSs. Additionally, characteristics of the courses themselves, such as the academic domain, might affect the results. For instance, further studies could examine how CRS designs need to be customized for different fields, such as Engineering versus Medicine, to enhance their applicability and effectiveness. Second, while this research encompassed various types of AI-based CRS, further exploration into additional types of systems could provide a more comprehensive understanding of how students navigate their course-selection decisions. Thirdly, although speed dating with storyboards proved effective in assessing user experience, it may not fully capture the intricacies of real-time interactions with AI-based CRSs in authentic educational environments. Therefore, future research endeavors should focus on developing and empirically testing genuine AI-based CRS models. Lastly, this study does not address the evolution of students’ expectations regarding AI-based CRSs over time. As students become more accustomed to interacting with AI in their course-selection process, their expectations and reliance on AI may change. Future studies should investigate how the relationship between students and AI in the course-selection process evolves over extended durations.

Despite the outlined limitations, this study has yielded intriguing insights into students’ expectations regarding AI-based CRSs during their course-selection decision-making process, encompassing both the search and evaluation phases. During the search phase, students prefer the AI-based CRS to play an assisting role, aiding in retrieving, filtering, and visualizing complex data to streamline exhaustive search tasks. Conversely, in the evaluation phase, students envision the AI-based CRS assuming a more leading role, serving as a benchmark for quantifying difficult-to-evaluate factors such as course suitability, learning value, and the potential for serendipitous discoveries. These findings shed light on the dynamic nature of student interactions with AI-based CRSs, underscoring the importance of these systems adapting their roles to align with each phase of the course-selection decision-making process. Leveraging advanced LLMs, future AI-based CRSs can be tailored to meet these student expectations, potentially fostering a cohesive and effective partnership between students and AI in the realm of course-selection decision-making.