Introduction and state of the art

Digital competence is one of the basic competences that all students should have acquired by the end of their compulsory education in order to develop as individuals and be able to successfully integrate in society (Diario Oficial de la Unión Europea, 2006). This competence can be defined as "that which involves the creative, critical and safe use of information and communication technologies in order to meet the goals related to work, employability, learning, use of free time, inclusion, and social participation” (Ministerio de Educación, Cultura y Deporte, 2015: 10). In its most recent policies, actions and communications, the European Commission has confirmed that acquiring an adequate level of proficiency in the use of information and communication technologies (ICT) is one of its most relevant priorities (Comisión Europea, 2010, 2018).

In order to improve citizens’ level of digital competence, the European Commission has developed the framework “DigComp: The Digital Competence Framework for Citizens” (Vuorikari, Punie, Carretero, & Van den Brande, 2016). Despite the efforts made by government institutions, recent studies indicate that younger people, in spite of being considered “digital natives”, have an insufficient level of digital competence (Johnson & al., 2014; Pérez-Escoda, Castro-Zubizarreta, & Fandos-Igado, 2016). This fact is proof that digital competence is not inherently acquired by having access to the Internet and making intensive use of technology, but rather, specific training is required, an issue that had been previously pointed out in the literature (Fernández-Cruz & Fernández-Díaz, 2016; Napal, Peñalva-Vélez, & Mendióroz, 2018; Pérez-Escoda & al., 2016). Another related issue that previous studies have also raised is the threat of a new digital divide, not due to lack of access to technology, but due to lack of digital competence (Pérez-Escoda & al., 2016; Van-Deursen & Van-Dijk, 2011).

Teachers should play a central role in ensuring that their students acquire the digital competence they lack. Nonetheless, in order to successfully achieve this goal, it is necessary that teachers themselves have an adequate level of digital competence. In this regard, it should be taken into account that the use that educators make of ICT is very different from that of other professions (Røkenes & Krumsvik, 2014). For this reason, the term “teacher’s digital competence” has been coined to refer specifically to the “set of abilities, knowledge, skills, dexterity and attitudes related to the critical, safe and creative use of the information and communication technologies in education” (INTEF, 2017a: 2). In order to facilitate the development of teacher’s digital competence, several initiatives have emerged at both national and international levels. UNESCO published a framework describing the competences that teachers need to have in order to effectively use ICT in their professional practice (UNESCO, 2011). Subsequently, the European Commission developed the framework “DigCompEdu: European Framework for the Digital Competence of Educators” (Redecker & Punie, 2017) with the aim of defining the digital competence that educators must have in order to succeed in making students digitally competent. In addition, the European Commission has elaborated a digital education action plan that includes eleven initiatives to support ICT use and development of digital competence in the educational context, which are meant to be applied before the end of 2020 (Comisión Europea, 2018). In Spain, the Instituto Nacional de Tecnologías Educativas y de Formación del Profesorado (INTEF) published the “Marco Común de Competencia Digital Docente”, conceived as a reference framework for diagnosing and improving the digital competence of teachers (INTEF, 2017b). Despite the numerous actions taken by different national and international organisations, results of recent research show that there is an alarming difference between the digital competence that teachers should have in order to develop digital competence in their students, and the one they actually have (Almerich, Suárez, Jornet, & Orellana, 2011; Falcó, 2017; Fernández-Cruz & Fernández-Díaz, 2016; Fernández-Cruz, Fernández-Díaz, & Rodríguez-Mantilla, 2018; Kaarakainen, Kivinen, & Vainio, 2018; Napal & al., 2018; Suárez-Rodríguez, Almerich, Díaz-García, & Fernández-Piqueras, 2012). Therefore, there is a compelling need for initial and in-service teacher training in digital competence.

Teacher’s digital competence encompasses multiple areas, as shown by the different frameworks developed to date (INTEF, 2017b; Redecker & Punie, 2017; UNESCO, 2011). Among the areas in which lack of training is of special concern, that related to safety and responsible use of technology stands out. There is strong evidence that teachers have a clear lack of knowledge in this area (De-los-Arcos & al., 2015; Falcó, 2017; Govender & Skea, 2015; Mannila, Nordén, & Pears, 2018; Napal & al., 2018; Pusey & Sadera, 2011; Shin, 2015). Specifically, previous studies have shown the lack of teacher training concerning the different risks to which children are exposed on the Internet (Govender & Skea, 2015; Mannila & al., 2018; Pusey & Sadera, 2011; Shin, 2015), device and personal data protection (Mannila & al., 2018; Napal & al., 2018; Pusey & Sadera, 2011), digital identity (Napal & al., 2018; Pusey & Sadera, 2011), rules of behaviour on the Internet (Falcó, 2017; Napal & al., 2018; Pusey & Sadera, 2011), and copyright and licensing of digital educational materials (De-los-Arcos & al., 2015; Falcó, 2017; Mannila & al., 2018; Napal & al., 2018; Pusey & Sadera, 2011; Shin, 2015). Without knowledge of these topics, teachers will hardly be able to educate their students in safe and responsible use of technology, as demanded by the teacher’s digital competence frameworks developed. This deficiency in teacher training is a severe problem given that there is a clear need to teach children to use technology in a safe and responsible way since they lack the necessary knowledge (Ey & Cupit, 2011; Gamito, Aristizabal, Vizcarra, & Tresserras, 2017; Sharples, Graber, Harrison, & Logan, 2009).

Teachers should play a central role in ensuring that their students acquire the digital competence they lack. Nonetheless, in order to successfully achieve this goal, it is necessary that teachers themselves have an adequate level of digital competence.

Children are not fully aware of many of the risks that Internet use entails (Ey & Cupit, 2011; Gamito & al., 2017), which is specially concerning considering that most of them are exposed to these risks from a very young age, sometimes leading them to experience adverse incidents (Garmendia, Jiménez, Casado, & Marcheroni, 2016). For this reason, educational institutions should teach children not only about privacy, digital identity, and rules of behaviour on the Internet, but also how to protect themselves against the various dangers of the Internet.

Another major deficiency is the lack of digital competence in creating digital educational materials (Fernández-Cruz & al., 2018; Napal & al., 2018; Ramírez-Montoya, Mena, & Rodríguez-Arroyo, 2017). One consequence of this deficiency is that currently, most teachers do not use authoring tools to create digital educational resources (Fernández-Cruz & al, 2018), which have proven to be capable of providing several benefits for student learning (Gordillo, Barra, & Quemada, 2017; Gürer & Yıldırım, 2014). This deficiency is not only due to the lack of skills in using authoring tools but also to the aforementioned lack of knowledge with regard to licensing of digital materials and copyright, which makes it difficult for teachers to reuse existing content on the web as well as to distribute their own creations.

In view of the unavoidable and compelling need to train teachers to effectively develop their digital competence, new training actions must be undertaken. One possible solution is the use of courses in MOOC format for teacher training. MOOCs are online courses that allow for massive participation and that can be accessed without restriction and free of charge (Siemens, 2013). The overwhelming student-teacher ratio in MOOCs makes individual guidance and monitoring unfeasible, which is why these courses adopt instructional designs that are different from those of traditional online courses in order to allow massive assessment and feedback. The instructional design of a MOOC is a key aspect since it exerts a great influence on the motivation and academic performance of participants (Castaño, Maiz, & Garay, 2015). Based on MOOCs, new types of online courses have emerged such as SPOCs: courses with the same characteristics as MOOCs, except that the number of participants is relatively small and access is only granted to a specific group of people. The term “courses in MOOC format” encompasses all online courses with instructional designs that are characteristic of MOOCs, that is to say, courses that are designed to allow massive participation even if it does not occur.

Courses in MOOC format meet all the necessary conditions to offer a low-cost solution for initial and continuous training of all teachers in digital competence. In fact, prior studies have pointed out that teachers find these courses attractive for digital competence training (Castaño-Muñoz, Kalz, Kreijns, & Punie, 2018; Gómez-Trigueros, 2017; Ortega-Sánchez & Gómez-Trigueros, 2019). The suitability of courses in MOOC format for addressing teacher training deficiencies has not gone unnoticed by the European Union either, who led an initiative in 2018 to train teachers in safe Internet use through a MOOC (Better Internet for Kids, 2018). Although there is a notable and growing amount of research about MOOCs in the scientific literature (Chiappe-Laverde, Hine, & Martínez-Silva, 2015; Deng, Benckendorff, & Gannaway, 2019; Liyanagunawardena, Adams, & Williams, 2013; Veletsianos & Shepherdson, 2016), not enough attention has been devoted to examining the instructional effectiveness of these courses since, as Deng and others (2019) point out in their recent literature review, the measures of learning outcomes in MOOCs taken to date are not very sophisticated and are often based on a single variable such as the final grade or the completion rate. The majority of existing scientific literature on MOOCs has focused on topics such as course characteristics, types of MOOCs, challenges, potential impacts on education, participant characteristics and behaviour, and certification (Chiappe-Laverde & al., 2015; Deng & al., 2019; Liyanagunawardena & al., 2013; Veletsianos & Shepherdson, 2016).

Existing evidence on the effectiveness of courses in MOOC format aimed at teacher training in digital competence is even weaker than that which exists for MOOCs in general. Different experiences have been reported in the literature in which courses in MOOC format were used to train teachers in different areas of digital competence (Castaño-Muñoz & al., 2018; De-La-Roca, Morales, Teixeira, Hernandez, & Amado-Salvatierra, 2018; Gómez-Trigueros, 2017; Ramírez-Montoya & al., 2017; Sánchez-Elvira & Santamaría-Lancho, 2013; Tsvetkova, 2016). Notwithstanding, several of these studies did not carry out any evaluation of the effectiveness of the courses, and those that did only provided evidence obtained by means of questionnaires completed by the participants themselves as the sole instrument for gathering information. Whereas current evidence on the effectiveness of courses in MOOC format for training in teacher’s digital competence is scarce and weak, evidence that these courses can be effective in educating teachers in safe and responsible use of ICT is directly non-existent. Thus, further research is needed on the ability of courses in MOOC format to produce positive impacts on teachers in terms of learning outcomes related to digital competence and especially to the safe and responsible use of technology. This study examines the instructional effectiveness of courses in MOOC format for teacher training in safe and responsible use of ICT by means of the analysis of three official courses.

Research method

The aim of this study is to provide empirical evidence on the effectiveness of online courses in MOOC format for teacher training in safe and responsible use of ICT, in order to determine whether this type of instruction is an adequate solution to remedy the existing lack of teacher training on this subject. The research questions were as follows:

a) Are courses in MOOC format an effective way of training teachers in safe and responsible use of ICT?

b) Are courses in MOOC format an effective way of developing in teachers the digital competence to create digital educational materials for teaching safe and responsible use of ICT?

Evidence of effectiveness was obtained through the analysis of three courses in MOOC format organised by official public entities, whose characteristics are summarised in Table 1. The three courses covered the following topics on safe and responsible use of ICT: digital identity, privacy management, risks for children associated with Internet use (including access to inappropriate content, identity theft, cyberbullying, grooming, sexting, dangerous online communities, and technology addiction), good practices in the use of social networks, rules of behaviour on the Internet (netiquette), and licensing of digital materials. These are subjects in which, as seen in the introduction, teachers generally have a great lack of knowledge. In addition to providing training in the aforementioned topics, the courses also aimed to help teachers develop their digital competence to create digital educational materials. The courses were delivered through a virtual learning environment and consisted of a wide range of resources and activities, including videos recorded by experts, interactive multimedia resources (which presented examples of practical cases), additional materials to be used in the classroom with students, video tutorials on how to use different applications, forums, links to external resources, self-assessment tests, guided exercises, and digital resource creation workshops with peer review evaluation.

The final task in all the courses consisted of a workshop in which participants had to employ an authoring tool to create a digital educational resource about any of the topics related to safe and responsible use of ICT covered in the course. The aim of this final task was for participants to apply the digital competence acquired throughout the course to create and publish an educational resource that could be later utilised, both by themselves to teach their students how to make safe and responsible use of technology, as well as by other members of the educational community to educate on this subject and create new digital educational resources.

Three different instruments were used for the analysis of the courses. In order to measure the participants’ perception of the different characteristics of the courses, a questionnaire was used, which had Likert questions with five possible answers (1 totally disagree – 5 totally agree) and closed-ended questions. These questionnaires were completed by the participants after finishing the courses. Two additional measures were taken with the aim of analysing the learning outcomes.

On the one hand, the knowledge about safe and responsible use of ICT acquired by the participants during each course was measured by means of a pre-test and a post-test. The pre-test was the first activity completed by the participants whereas the post-test was the last one. Both tests were identical and were comprised of multiple-choice questions. On the other hand, with the aim of obtaining a measure of the digital competence for creating digital educational materials on safe and responsible use of ICT acquired by the participants during each course, the LORI instrument (Leacock & Nesbit, 2007) was used for evaluating, in each course, the quality of 40 educational resources created by participants chosen at random. Thus, 120 resources were evaluated, 14% of the total. Each one of these resources was evaluated by three reviewers with extensive experience in the use of LORI and in the creation of digital educational materials. The score for each criterion was obtained by averaging all the evaluations.

Results

Participants’ perception

The results of the questionnaire completed by the participants are shown in Table 2. The overall course scores lie within a range of 3.8-4.1 on a scale of 1 to 5, indicating that participants were, in general, satisfied with the training. The high degree of acceptance of the courses is also reflected in the fact that between 87 and 93% of the participants stated that they would recommend them to other teachers. The courses were rated positively in terms of structure, guidance, assessment, duration, and difficulty, although it is true that, in one of the courses analysed, the participants did not agree that the workload was adequate. The results evidence that the safe and responsible use of ICT is an important topic for teachers, and that the courses were effective for teacher training, not only in this area, but also in other areas of digital competence, such as digital content creation. Further proof of this latter fact is that teachers claimed that the digital resources they had created during the courses were of high enough quality that they could be used to teach their students how to use technology in a safe and responsible way.

Acquired knowledge

Table 3 shows the results of the pre-test and post-test taken by the participants of the courses analysed. In order to determine the magnitude of the difference between the scores achieved by the participants in the post-test and the pre-test, the Cohen’s d effect size (Cohen, 1992) was calculated. When using Cohen’s d, a value of 0.2 indicates a small effect size; a value of 0.5, a medium one, and a value over 0.8, a large one. In all courses it was found that the difference between post-test and pre-test scores was statistically significant with a large effect size (with Cohen’s d values ranging from 1.6 to 1.8). These results prove that the courses had a strong positive impact on the participants in terms of knowledge acquired regarding safe and responsible use of ICT.

Digital content creation

Table 4 shows the results of the evaluation conducted with LORI to measure the quality of a sample of the digital educational resources created by the participants during the courses. The overall quality of the resources evaluated, calculated as the average of the scores obtained for each of the LORI items, reached an average score greater than 3 on a scale of 1 to 5 in all courses. Taking into account that educational resources rated above that threshold using LORI can be considered of good quality (Gordillo, Barra, & Quemada, 2014), it can be stated that most participants finished the course with an acceptable digital competence to create digital educational materials. However, about 30% of the participants of courses A and C, and 13% of those of course B were not capable of creating high-quality resources. Overall, the resources evaluated were positively rated in terms of content quality, learning goal alignment, motivation, design, usability, reusability and standards compliance. However, notable deficiencies were observed regarding the resources’ ability to provide feedback to students and adapt to their behaviour. Quality evaluations also show that teachers had difficulty creating accessible resources.

Discussion and conclusions

This study provides, for the first time, strong empirical evidence that online courses in MOOC format are an effective way of training teachers in safe and responsible use of ICT. Based on the results obtained, it can be stated that these courses offer a possible solution to the concerning lack of teacher training in the area of digital competence related to the safe and responsible use of technology. Given that measurements of learning outcomes in MOOCs reported in the scientific literature to date are overly simplistic and frequently based on a single variable such as the completion rate or the final grade (Deng & al., 2019), this study makes an important contribution to research on MOOC courses by reporting on the measurement of learning outcomes from three different courses, which is based on three aspects: the participants' perception, the knowledge acquired by the participants calculated as the difference between the scores achieved in a post-test and a pre-test, and the quality of a set of digital educational resources created by the participants during the courses. In this respect, an important finding of this study is that completion rates of courses in MOOC format should not be used as a measure of learning outcomes. Although the completion rate for the three courses analysed in this study was very varied (49%, 66% and 89%), the knowledge acquired by the participants who completed them was very similar.

This study also provides solid empirical evidence on the effectiveness of courses in MOOC format in the development of teacher’s digital competence to create digital content aimed at teaching how to make safe and responsible use of technology. Although Ramírez-Montoya and others (2017) previously reported on the use of a MOOC to train teachers in the creation of digital learning resources, that work did not provide any evidence on the real effectiveness of the course for that purpose. The results of this study show that most participants of the courses were capable of creating good-quality educational resources on safe and responsible ICT use and considered that they would be able to use these resources with their students. However, the results also show that a significant percentage of the participants (between 13% and 30% depending on the course) did not acquire the digital competence needed to create high-quality digital educational resources. Moreover, difficulties were observed on the teachers’ part in creating content with a high level of accessibility, as well as educational resources with the ability to provide feedback and adapt to the students’ behaviour. Nevertheless, these difficulties had their origin not only in a lack of digital competence, but also in the limitations of the current authoring tools. While the results obtained suggest that courses in MOOC format can be of great help for developing teacher’s digital competence to create digital educational materials, these also indicate that this help might not be sufficient for some educators. Future works should investigate the profile of these educators for whom other training activities could turn out to be more effective. The results also indicate that the training activities that address the content creation area of the digital competence should, in addition to teaching teachers how to use authoring tools, pay special attention to technical aspects such as accessibility and content reusability, and delve into the creation of adaptive resources and the provision of feedback. These training activities should include active learning, one of the most popular strategies for teacher training in ICT use (Røkenes & Krumsvik, 2014).

Future research works should examine the instructional effectiveness of online courses in MOOC format for teacher training in areas of teacher’s digital competence other than safe and responsible use of ICT and digital content creation. Another interesting line of research would be to compare the instructional effectiveness of courses in MOOC format with that of other training activities. Of special interest would be to analyse effectiveness according to the profile of the participants since, that way, it would be possible to determine when the use of courses in MOOC format is the most suitable solution for overcoming the training shortcomings of teachers, and when the most suitable solution is another type of training activity. Although the evidence provided by this study suggests that online courses in MOOC format can be an effective solution to the unavoidable need to train teachers in certain areas of digital competence, there might exist other training activities that are more effective for teachers with a specific profile.