Introduction

In recent years, massive open online courses (MOOCs) have become quite popular. Popular does not mean regular. MOOC faces the challenge of not being recognized as “regular” courses (i.e., the ones taken at traditional learning institutions). Shrader et al. (2016) found that MOOC learners described different preferences for exploratory or instructor-directed instructional strategies. The implications for the instructional design of MOOC for attitudinal learning included recognizing that MOOC learners often view MOOC more as entertainment than formal education. MOOC is only seen as an auxiliary way to upskill. One of the major reasons for this is the assessment in an unsupervised environment, for example, the problems of test takers' online searches or interaction with others to find the answers to test questions (Beg et al., 2020). As a form of free and self-paced education, MOOC often does not count toward a formal university qualification. In addition, the low course completion rate is the most criticized problem of MOOCs, a difficult problem that online courses have always faced (Tauber, 2013). As a result, assessments valued by universities as important learning outcomes are thought to be less relevant to MOOC learners or designers (Zhao et al., 2017; Chudowsky et al., 2003; Earl & Torrance, 2000).

Assessment is a central character in the design of massive open online courses (MOOC) (Sandeen, 2021). In the past few years, a few universities started designing credit-bearing MOOCs as part of university programs, with the aim to encourage quality learning and good outcomes that satisfy formal university assessment criteria. Chunwijitra et al. (2020) suggest that the MOOC service framework consists of five layers: authentication, resources, learning, assessment, and credential layers. Assessment is one of the emerging key themes of MOOC (Bayne & Ross, 2013). Self-assessment is also very important in MOOC, especially in our case, as we received a lot of letters with questions. It has been suggested that self-assessment should be used as an assessment for learning instead of an assessment of learning (Admiraal et al., 2015). Some scholars suggest developing and embedding student self-assessment courses in subject teaching (Brown & Harris, 2014; Olivares et al., 2021). As a result, more and more universities are conducting assessment research on MOOCs. Earl (2003) divided the assessment into three categories according to the purpose of the assessment: assessment for learning, assessment of learning, and assessment as learning.

Peer and self-assessment offer an opportunity to scale both assessments and learning in global classrooms (Kulkarni et al., 2013). Gradually, more and more scholars are beginning to study the effectiveness of the assessment. Among them, many scholars place more emphasis on peer assessment. Stančić (2020) showed that despite being stressful and uncomfortable for many students, peer assessment was more beneficial for the student's learning than self-assessment. Peer assessment is becoming an increasingly popular tool to assess complex assignments in MOOCs (Capuano & Caballé, 2018; Reinholz, 2016). But some learners do not have the necessary knowledge and experience to assess their peer's work. Hence, there are some problems with the quality of the peers’ respective feedback (Hew & Cheung, 2014). One of the major challenges facing Massive Open Online Courses (MOOC) is assessing learner performances beyond traditional automated assessment methods (Cho & Cho, 2011; Watson et al., 2017). However, there is no in-depth and systematic study on student self-assessment of MOOC (Liyanagunawardena et al., 2013). Today, students who self-assess effectively often learn better, and creating effective, comprehensive methods to help them do so is still a critical challenge. Although self-assessment appears to be a skill that can be improved, both students and professionals continue to have difficulty with accurate self-assessment (Motycka et al., 2010). For example, Ivaniushin et al. (2016) proposed the approach of assessment learning outcomes in collaborative project-based learning. Ashton and Davis (2015) identified that training students to assess will improve their ability to provide quality feedback. Different researchers have different priorities in defining self-assessment. Andrade and Du (2007) believed that self-assessment belongs to formative assessment. The concept emphasizes that the purpose of students' self-assessment is not only to make a self-judgment to identify what’s lacking, but also to adjust their learning on this basis. Some researchers have expanded the concept of self-assessment to self-judgment based on assessment criteria, emphasizing the use of assessment criteria. As Rolheiser and Ross (2000) put it, “Self-assessment is ‘Students judging the quality of their work based on quality criteria and assessment of good performance’”. Some researchers directly define self-assessment as student self-assessment. There is no special emphasis on the use of assessment criteria in judging learning. For example, Brown and Harris (2014) defined student self-assessment as the description and assessment of students on their performance and academic ability. There are different types of assessment used by different MOOCs (Papathoma-Köhle et al., 2015; Zeng, 2017; Wong, 2016), for example, automated assessment (Pieterse, 2013; Ashton & Davis, 2015), peer assessment (Kulkarni et al., 2013; Sadler & Good, 2006; Stančić, 2020), and self-assessment (Wilkowski et al., 2014).

Reviewing the existing research, there are three popular directions for student self-assessment in the academic world. First, Boud and Brew (1995) proposed that different learning and assessment tasks serve different cognitive interests, and then proposed three types of student self-assessment: student self-assessment in technical interest, practical interest, and emancipatory interest. Student self-assessment in technical interest means that we check whether knowledge and skills have been acquired and understood against established standards and the level achieved. Second, Panadero et al. (2013) identified three strategies for teachers to help students develop self-assessment skills, namely self-assessment, self-assessment gauge, and self-assessment script. A script is an ordered set of structured statements built from task execution steps (Alonso-Tapia & Panadero, 2010). The self-assessment script indicates that students conduct learning activities according to the questions in the self-assessment script, then reflect on and evaluate the learning results. Self-assessment scripts are ordered reflective questions based on evaluation criteria constructed according to task execution steps (Alonso-Tapia & Panadero, 2010). For example, Lepp et al. (2017) showed that the two tools of self-assessment (self-assessment questions and troubleshooters) complement each other and can be suitable for different participants.

In a survey of 15 MOOC pilot schools in China, the reason lies in the lack of clear criteria, which makes it difficult for teachers and students to control this form of assessment. For this situation, some scholars’ research provides some help. For example, Panadero et al. (2013) discussed the question of “whether, and in what form, assessment standards exist” in student self-assessment activities. According to power sharing between teachers and students, Taras (2016) rates student self-assessment as low, medium, and high, including a standard model, learning contract design, self-marking, sound standard, and self-assessment with an integrated tutor and peer feedback. One of the power prerequisites is resources (Burns, 1996). In self-assessment, reflective learning can help learners speed up knowledge updating, make students become learning subjects, promote online collaboration and communication of learners, effectively improve their information literacy, and complete the transition from superficial learning to deep learning (Wang et al., 2018). Some researchers (Wang & Sun, 2002) point out that student self-assessment mainly takes place in learning. Student self-assessment should include three stages: the beginning of learning activities, the middle of learning activities, and the end of learning activities. In terms of the results of learning, it should include a self-assessment of knowledge, skills, habits, attitudes, and personality (Wang & Sun, 2002; Eschenbrenner & Nah, 2007). Scholars also focus on the technical aspects of how to apply it, and there is no in-depth research on how to assess it. Valdivia-Vázquez et al. (2021) indicate that the EIIMC-MOOC is a valid, reliable, and stable tool to evaluate initial motivation and prior knowledge of participants regarding energy-related topics. Student self-assessment in MOOC is not only lacking in China but has also received little attention in other countries. Therefore, this study hopes to establish an integrated student self-assessment paradigm for MOOC learning, especially for university students in China.

Material and methods

Theoretical framework

To optimize the assessment of MOOC learning, this research has three major objectives. First, we will sort out influencing factors that affect Chinese student self-assessment in MOOC. Second, we will use an analytics tool to analyze student self-assessments in MOOC. Third, we will construct an integrated student self-assessment paradigm. For this reason, the following research question was proposed: What factors influence student self-assessment in MOOC? Student self-assessment is a comprehensive assessment in which students try to find the changes in their deep and implicit learning. We need supportable theories to classify and simplify the process of student self-assessment. To answer the question, a theory called “the Seven Pillars of Assessment” can help. The main contents of this theory are as follows: Falchikov (2004) described seven basic questions in the assessment area: “Why to Assess”, “How to Assess”, “What to Assess”, “When to Assess”, “Who Assesses”, “How Well” and “Whither”. It will be more concise and logical to select the factors that affect student self-assessment from these seven basic questions. According to the characteristics of MOOC learning, the author consulted fifteen experts, including educational technology experts from Chinese universities, principals and backbone teachers from pilot schools of MOOC classrooms, and finally determined nine factors that affect student self-assessment in MOOC. The nine factors are shown in Table 1.

The interpretative structure model

We need to clarify the relationship between these important factors above to provide a basis for the establishment of an integrated self-assessment paradigm. For this purpose, the interpretative structure model (ISM) was selected as the analytical tool in this study. The ISM is a structural model put forward by Professor John Warfield in 1973, which aims at analyzing complex social structure problems. The basic idea is to make use of people's practical experience, professional knowledge, and computer assistance. By structuring and layering the complex and disorderly relationships among system elements, a multi-level and hierarchical explanatory structure model is constructed. The characteristic of this model is that many fuzzy factors of the system are decomposed into a visual, organized, and hierarchical internal structure so that people can clearly understand the relationship among the factors, grasp the essence of the problem and find solutions. It is especially suitable for system analysis with many variables, complicated relations, and unclear structure. The ISM plays a very important role in revealing the system structure, especially in analyzing the content and structure of teaching resources, designing and developing learning resources, and exploring the mode of the teaching process. It is a unique research method in educational technology research. The ISM is a valuable management tool and a qualitative and interpretative method used to generate solutions for complex problems and identify the relevant importance of each variable (Shen et al., 2016; Pfohl et al., 2011). The interpretative structural model (ISM) is to create a structural model made up of nodes and directed edges by calculating the logical relationship among the elements. The model is to describe the hierarchy and causality within the complex system through mathematical methods. Given the wide applicability and the effectiveness of the model in analyzing internal factors, this paper used the ISM to analyze the influence factors of student self-assessment so that we can understand the internal structure of the process of student self-assessment in MOOC. The various steps involved in the ISM are extracted from (Abbas et al., 2022; Ravi & Shankar, 2005; Shahabadkar, 2012). The analysis process is as follows:

• Step 1. E is the set of all the elements in system S, and R is the set of all the relations in system S. There are n elements in system S.

Adjacency matrix A is used to represent the influence relationship between two elements of the system, where,

• Step 2. Solve the reachable matrix M. There is a transitive relationship between e_i and e_j. If there is a directed path from node i to node j on a directed graph, e_i to e_j is considered to be reachable, where

The reachable matrix B can be obtained by logical operation of the adjacency matrix and the identity matrix, and the formula is:

• Step 3. According to the reachable matrix, the antecedent set and reachable set are given:

Further, the underlying elements are given:

If e_i is the underlying element, then the antecedent set A (e_i) contains e_i itself and strongly connected elements. The reachable set R (e_i) contains itself, elements strongly connected to e_i and elements reachable from e_i. If there is an element e_j at the lower level of e_i, e_j can only be contained in A (e_i) but not in A (e_i) ∩ R (e_i), that is, A (e_i) ≠ A (e_i) ∩ R (e_i).

Constructing the ISM of student self-assessment in MOOC

Based on Table 1, this study takes MOOC teaching in Chinese universities as an example to establish an integrated student self-assessment paradigm. According to the definition of the interpretative structure model, we use MATLAB software to get the reachability matrix B. First, this study consulted 15 experts with rich experience in MOOC teaching in Chinese pilot universities. Based on the literature review and expert discussions, the relationships among the nine factors were determined. Then the adjacency matrix A was created.

The model results

According to the solution formula of ISM, the level of division of the reachable matrix can divide the influence factors into six levels. The first level: L1={S3, S4, S6, S7}. The second level: L2={S8}. The third level: L3={S9}. The fourth level: L4={S1}. The fifth level: L5={S2}. The sixth level: L6={S5}. As shown in Table 2.

According to the hierarchy above, matrix A is presented as a directed graph as shown in Figure 1, and ellipses of different colors represent elements at different levels. According to the horizontal and vertical relationship of the factors, this paper divides the six levels into surface factors, middle factors, and deep factors. Surface factors refer to L1 {S3, S4, S6, S7}. Middle factors include L2 {S8}, L3 {S9}. Deep factors refer to L4 {S1}, L5 {S2}, L6{S5}.

The MICMAC analysis

The MICMAC analysis is carried out based on the principle of matrix multiplication. The model is a method proposed by Duperrin and Godet to analyze the relationship and interaction between factors in the system and is commonly used to identify factors with high driving and high dependence in the system.

System elements can be divided into the following four quadrants: autonomous variables (a), dependent variables (b), linkage variables (c), and driving variables (d). As shown in Figure 2, the driving power and dependence power of autonomous variables are weak.

The dependence power of dependent variables is strong, but the driving power is weak. The driving power and dependence power of linkage variables are both strong. The driving power of driving variables is strong, but the dependence power is weak.

Calculate the sum of the rows and columns of the reachable matrix B, denoted as the driving force P, and the dependence J. P indicates the degree of influence of this factor on other factors. J indicates the extent to which this factor is affected by other factors. By calculating P and J values for each system element, the system element is divided into four different quadrants, as shown in Figure 2.

Based on the four-quadrant diagram, we further classify these nine factors. In Figure 3, the four-quadrant diagram is divided into three parts by two dashed lines. The three parts are called the surface factor, middle factor, and deep factor respectively.

As shown in Figure 3, assessment method (S3), assessment technology (S4), and assessor (S7) are in the first part, and they are the surface factors. Reliability of assessment (S8) and appreciation of self-assessment (S9) belong to the middle factors. Except for the continuity of the self-assessment process (S6), motivation for assessment (S1), assessment objective (S2), and the gaining of knowledge and skills (S5) are classified as deep factors.

We believe that the shaded region where S6 is located belongs to the region with weak dependence and power, so factors in this region cannot be considered deep factors. Combined with the classification results of the ISM before, the S6 is in the first level but it doesn’t have any arrows associated to other factors in Figure 1. So, we consider that the continuity of the self-assessment process (S6) should be an independent factor for the student self-assessment in MOOC.

Discussion

Based on the ISM and MICMAC analysis, we obtained a hierarchical structure (Figure 1) of students’ self-assessments and the classification of influencing factors (Figure 3). In the following part, we will discuss the details of the analysis above.

Constructing integrated student self-assessment paradigm in MOOC

Through the analysis of different levels of factors, this study established an integrated student self-assessment paradigm in MOOC, as shown in Figure 4. In the figure, the element node (○), with different colors and heights, represents different levels. The next section will expand on Figure 6, starting with L1 (the surface factor).

First, for MOOC learning in China, student self-assessment should be run through the entire process of MOOC learning. Students should carry out a self-assessment of their learning progress and achievements before, during, and after MOOCs. Meanwhile, it is best to adopt different assessment methods in different learning periods.

Second, students should be the primary executors of the assessment. However, being an excellent self-assessor is not something that can be accomplished by accepting a large number of assessment activities given by teachers. Instead, students need to carry out a large number of self-assessment practices and learn through doing. Therefore, teacher assessment will gradually give way to self-assessor and peer mutual assessment, and the situation dominated by teacher assessment will be transformed into students' self-assessment as the basis and leading. Universities should change the assessment trend from the traditional terrace model to the supporting lifelong learning model. This, to awaken students’ self-awareness and initiative so that they can be transformed from passive assessment objects into conscious and active assessment subjects. Teachers should make students aware of their responsibility to monitor, assess, and regulate their learning.

Third, the validity of an assessment depends on its reliability. To ensure the reliability of student self-assessment in MOOC, it is necessary to improve students’ self-assessment skills and set up assessment courses to enhance the quality of assessment. On the premise of ensuring the effectiveness of self-assessment, universities should make student self-assessment become the basis of MOOC assessment. According to the current situation of MOOC assessment in China, teachers should be diligent in “making concessions”, transferring the assessment rights to students, and turning the assessment activities into a process of students’ active participation, self-reflection, self-education, and self-development. After clarifying the importance of student self-assessment in MOOC, the motivation of student self-assessment is further determined to optimize the learning of MOOC. Then, the self-assessment object (MOOC learning process) in MOOC is determined.

Finally, we should focus on the deep factors (the gaining of knowledge and skills) of student self-assessment. First, the content of student self-assessment can be divided into “knowledge and skills”, and “cognitive process”. So, the gaining of knowledge and skills mastered in MOOC is what the assessor needs to assess first. This assessment process is cognitive and we need to use different assessment methods. Based on the characteristics of MOOC, it is more suitable for the assessment method with the characteristics of constructive assessment criteria and dynamic assessment process. Specifically, if the content of the assessment is the knowledge and skills, self-assessment methods with a middle of power-sharing, clear assessment criteria, and emphasis on grades or scores can be adopted, like the following types, self-marking, sound standard, and self-assessment rubrics. If the content of the assessment is a cognitive process, self-assessment in the practical interest, self-assessment in the emancipatory interest, and self-assessment with the integrated tutor and peer feedback are more suitable. Through communication and discussion with teachers or peers, students form their personalized understanding of the assessment tasks and construct their assessment criteria. Students not only construct individualized understanding and individual assessment criteria but also reflect on the construction process itself and carry out self-assessment activities independently.

Conclusion

Given the above discussion, we propose several suggestions for student self-assessment of MOOC learning. First, consider student self-assessment in MOOC as a formative assessment method. Student self-assessment is not only an assessment activity after a learning stage. It is a kind of formative assessment, running through the entire process of student learning. Therefore, student self-assessment is a form of “formative assessment”. Second, universities should enhance student awareness of self-assessment through publicity. Make students aware of their responsibility to monitor, evaluate and regulate their learning. Third, we could set up assessment courses to enhance the quality of student assessment. For example, video micro-lectures can be created so that students can acquire assessment knowledge and skills in class. Then, choose assessment methods to increase the reliability of the results. Finally, schools should optimize the environment of student self-assessment with the help of technology. With the deep integration of information technology and education and teaching, the networked nature of assessment activities is becoming increasingly apparent. All kinds of network assessment systems provide students with personalized learning data, diversified assessment tools, and communication platforms, which break through the limitations and difficulties of traditional assessment in data collection and can optimize the environment of student self-assessment.

To sum up, a good lifelong learner must be a good self-assessment. Teachers can still assume the responsibility of assessing students’ learning, but in the long run, students would rely on teachers’ evaluation and feedback, and would find it difficult to form self-assessment quality. Thus, students would find it hard to judge their growth after leaving school and even miss learning opportunities. Therefore, cultivating student “assessment quality” through the process of self-assessment not only helps to optimize students’ current learning but also lays a good foundation for students’ lifelong learning and development. In addition, student self-assessment should integrate qualitative methods and quantitative methods. For example, for the assessment of knowledge and skills, quantitative methods, such as self-assessment rubrics can be used. For the assessment of the cognitive process, qualitative methods, such as self-assessment script, self-assessment report, and reflective logging can be used. Among them, reflective learning not only develops students’ learning behavior but also enables students to have more comprehensive abilities in the learning process, making them more complete people (Cristianti et al., 2020).

In the current case of MOOC in China, the focus on student self-assessment is not enough (Li, 2017). It is necessary to break the one-sided tendency of “student self-assessment just means students grade themselves”. We need to form a holistic and comprehensive understanding of self-assessment. Besides China, according to the review of existing literature, there is little research focusing on student self-assessment in MOOCs in other countries. For online learning, such as MOOC, self-assessment should be the main assessment method because of the characteristics of self-study in online teaching. Therefore, the first thing is to change the attitude of teachers and students towards self-assessment and form a comprehensive self-assessment understanding. Then, sort out the logical path of student self-assessment and construct an integrated student self-assessment paradigm in MOOC. This is the contribution of this study. In the future, we will focus on how to optimize the learning effect of MOOCs through student self-assessment. (1)