Introduction and state of the art

Since the 1980s, audiovisual media has been widely used as a language teaching resource (Geddes & Sturtridge, 1982; Sherman, 2003; Stemplesky & Aracario, 1990; Tomalin, 1986). Its use in the language classroom has garnered learning benefits for students who can work with authentic language in class and interact in communicative situations (Buck, 2010; Ghia & Pavesi, 2016; Izquierdo et al., 2017; Navarro-Pablo et al., 2019; Pavesi, 2015). Technical progress has informed evolution in the way in which we currently use technology in the classroom, where student-centred approaches require students to play active roles in the use of Information and Communication Technology (ICT) (Motternan, 2013). Against this background and dating back a number of decades, audiovisual translation (AVT) began to be used as a resource in foreign language classes specifically by using subtitling as a support element (Duff, 1989; Holobow et al., 1984; Maley & Duff, 1983; Price, 1983; Vanderplank, 1988). The proactive role of students in the subtitling and dubbing of videos has more recently demonstrated that a beneficial effect on language learning can be appreciated and has generated increasing interest in didactic audiovisual translation (DAT) (Wang & Díaz-Cintas, 2022; Zabalbeascoa et al., 2012).

DAT specifically refers to the use of modes of translation like subtitling, dubbing, subtitling for the deaf and hard of hearing (SDHH), audio description (AD) or voiceover as pedagogical resources in language didactics (Talaván, 2020). In other words, DAT is focused on the design, development and application of didactic tasks where students are required to subtitle or dub a video clip using distinctive strategies and make use of the available technology (which includes different applications and software, such as Aegisub). The research regarding DAT possibilities in language learning is an interdisciplinary area in which applied linguistics, language and literature didactics, translation studies and educational science all converge (Lertola, 2021). In addition to offering promising results related to the development of the communicative skills of primary, secondary and university education students (Ávila-Cabrera & Rodríguez-Arancón, 2021; Bausells-Espín, 2022; Beltramello & Nicora, 2021; Bolaños-García-Escribano & Navarrete, 2022; Incalcaterra-McLoughlin, 2019; Marzà et al., 2022; Pintado-Gutiérrez & Torralba, 2022; Soler-Pardo, 2020), DAT likewise has obvious implications in terms of digital literacy, given that subtitling or active dubbing of videos by students facilitates working on their digital skills, multimodal working or mediation, amongst others (Martínez-Sierra, 2021). Furthermore, it is of importance to recognise that DAT can be used in face-to-face or digital educational environments, which is of special interest given the prominence that non face-to-face and hybrid modes of language learning have been afforded as a result of the COVID-19 pandemic.

For two decades, research in this field has provided empirical evidence that shows that the use of DAT facilitates the acquisition of vocabulary (Lertola, 2019), the promotion of intercultural skills (Borghetti & Lertola, 2014), the improvement of oral comprehension (Ávila-Cabrera, 2022; Navarrete, 2020; Sánchez-Requena, 2018) and also the advancement of written skills (Ibáñez-Moreno & Escobar, 2021). Linguistic advantages aside, it has been observed that the use of the DAT leads to an improvement in other fundamental elements in education, such as creativity (Ávila-Cabrera, 2022), the fostering of interaction in the classroom and motivation (Alonso-Pérez, 2019), the activation of previously-acquired knowledge, working with higher and lower order cognitive abilities, or translanguaging -the use of both the mother tongue and the foreign language in the classroom- (Baños-Piñero et al. ., 2021). Recently, research results have underlined the benefits of working with integrated skills (Sánchez-Requena et al., 2022) and confirm the pedagogical possibilities of DAT in new environments, such as in speech therapy (Fernández-Costales et al., 2023). Until now, most research has focused on analysing the results of case studies with small samples carried out in a single institution and research with inferential statistics or analysis using large samples collated in different contexts is infrequent. In addition, previous studies predominantly look into a single aspect: the improvement in students' written skills (Ibáñez-Moreno & Escobar, 2021), the impact on vocabulary acquisition (Elsherbiny, 2021) and the effect of DAT on motivation (Beseghi, 2021). However, the effect on different dimensions in language learning has not been comprehensively investigated. Against this background, the TRADILEX project (https://tradit.uned.es/proyecto-tradilex/) worked on the principle that it was necessary to research the application of DAT from a broader perspective and by using a robust research design, with a considerable sample size made up of participants from diverse national locations. The sample consists of 566 participants from eight Spanish universities enrolled in the 2021-2022 academic year. The general objective of the project was twofold: on the one hand, to evaluate the impact of DAT on the four linguistic skills of the participants and, on the other, to assess the learning progress from the first to final lesson plans (LP) carried out. In order to estimate the effect of DAT on the L2 learning process, 30 LPs were designed for the B1 and B2 levels from the Common European Framework of Reference for Languages ​​-or CEFR- (Council of Europe, 2018) in which the participants worked with the five AVT modes: subtitling, SDHH, dubbing, AD and voiceover. With that in mind, the specific objectives of this article are formulated as follows:

1) Analyse the impact of DAT on oral production.

2) Analyse the impact of DAT on written production.

3) Analyse the impact of DAT on oral reception.

4) Analyse the impact of DAT on written reception.

5) Evaluate the progress of L2 skills and knowledge of AVT throughout the didactic intervention.

6) Evaluate the participants’ perception during the didactic intervention.

There are several reasons why we believe that this piece of research can contribute to educational improvement in language learning. Firstly, empirical evidence confirms the effectiveness of DAT as a pedagogical resource in the teaching of languages ​​in different linguistic settings. Secondly, the academic progress of students who integrate this didactic resource into their learning itinerary has been verified. Thirdly, timely educational implications of a didactic proposal that goes beyond improving linguistic competence are established, given that it promotes digital literacy, multimodal learning and the use of new technologies in the classroom. Finally, the results presented here are the result of an interdisciplinary research project in which researchers from philology, language didactics, and pedagogy have participated, all of whom can contribute to new interdisciplinary perspectives to the language teaching process.

Methods and materials

The study has a pre-experimental mixed research design and is placed within a national R+D+I project developed between 2020 and 2023. It is essential to have a general understanding of the project in order to become “au fait” with the methodology used in this study. Figure 1 shows the timing of the project which revolves around 9 phases:

• Launch (O1). This first phase took place in the first term and consisted of the creation of the research group, the building of the website and the carrying out of a bibliographical review.

• Proposal design (O2). During the second term, the methodological proposal and the general research framework were designed. The modern language centres where the investigation would be implemented were also selected.

• Task design and centre network (O3). In the third term of the first year, 30 LPs were designed for B1 and B2 levels and a collaborative network was established with the language centres involved in the project.

• Pre-experimental study (O4) and its continuation (O5). In the first term of the second year, specific training was given to the participating teachers so they could rigorously carry out the project implementation. Additionally, studies began within the different language centres and the pilot data were collected. This phase continued during the second term of the second year through the investigation presented here, with constant monitoring by the research team.

• Study closure (O6). In the third term of the second year, the study in language centres was closed and data analysis began.

• Platform design and other languages ​​(O7). In this seventh phase, the virtual platform was developed to host the 30 LPs for use by both students and teachers.

• Implementation (O8). The digitalisation of the LPs in the new platform format was the eighth phase. In addition, a manual of good practices in DAT was finalised.

• Piloting, transfer and dissemination (O9). In this final phase, platform pilot tests were carried out with some institutions, transfer possibilities to other contexts were analysed and an international conference on DAT was held.

The main objective of this research is to analyse the data obtained in the phases associated with the pre-experimental study (O4 and O5). Temporarily, these phases took place in the final term of 2021 and the first term of 2022. This is of particular interest as the terms coincided with the SARS-CoV-2 pandemic. This situation resulted in the intervention being carried out completely online in a Moodle, where virtual courses were created for each level and centre.

The process which was followed was identical for each institution where each institution was provided with a file containing the instructions the students would need to complete their course registration in the virtual learning space. Each corresponding centre boasted not only teaching staff but also the supervision of an individual belonging to the research group, whose main function was to supervise and ensure the correct development of the intervention. There were weekly follow-up sessions and the course forums and email were used as the main means of communication. The courses had the following structure:

• Course presentation and initial tasks. This section provided a general guide in PDF and interactive support with the main steps to follow, alongside access to the pre-test and the initial questionnaire.

• Lesson plans. Subsequently, five blocks with the different DAT modes were offered, each with an introductory guide and interactive support with the initial steps to follow and guidelines for the corresponding DAT mode, software tutorials, etc. The first block contained 3 subtitled LPs, the second had 3 voiceover LPs, the third included 3 dubbing LPs, the fourth consisted of 3 audio description (AD) LPs, and in the fifth block there were 3 SDHH LPs. Within each modality, the level of difficulty was incremental, as is the case in the full set of DAT modes. With this strategy, sequential learning is anticipated which favours the learning process and reduces the dropout rate.

• Final tasks. At the end of the intervention, the students had to complete a final questionnaire and do an integrated skills test to verify their progress.

The instruments used for data collection were the following:

• Initial questionnaire. An 11-item questionnaire (largely closed-ended questions) that collated data of interest on the sample. The questionnaire contained a block for sociodemographic data, self-perception of communicative competence levels in L2 and previous DAT experience. The questionnaire also explored the sample’s expectations regarding the course.

• Initial Test of Integrated Skills (ITIS). This test evaluated the four basic macro-skills and measured the students’ performance before starting the intervention (Couto-Cantero et al., 2021).

• Intervention. The intervention was evaluated through rubrics designed for each modality. On a weekly basis, the students received information about their delivered tasks, therefore allowing for reflection on their strengths and areas for improvement.

Regarding the intervention, Figure 2 (https://doi.org/10.6084/m9.figshare.23123426) shows a combination of the LPs that were designed and those that were used for each didactic sequence: the sequences of 6 LPs for a single modality and the sequences of 15 LPs for 5 combined modes. Although not all the LPs were used in the intervention analysed here, specific courses were indeed implemented in which short sequences of a single modality were worked on. This occurred specifically during the piloting carried out in the O4 phase (González-Vera, 2022). After piloting, necessary corrections and improvements were implemented in the design of the intervention, in the virtual platform and in some LPs.

The LPs were built around an excerpt from a short film which had previously been selected in order to achieve specific educational objectives. The estimated time that each participant had to dedicate to undertaking each LP was 60 minutes, in which the tasks were divided into four blocks: warm-up tasks, video watching, DAT tasks and consolidation tasks (post-DAT tasks).

• Final Test of Integrated Skills (FITIS). This post-test was identical to that of the pre-test and its main objective was to quantitatively measure the degree of development of each macro-skill after the intervention, in order to be able to compare this information with that of the pre-test. This allows one to have a quantitative vision of student progress and, at the same time, is a core element for the research project.

• Final questionnaire. A 21-item questionnaire (largely closed-ended questions) aimed at assessing students' perception of the experience. The questionnaire was built around four blocks: process perceptions, perceptions of the results obtained, perceptions of the impact of the intervention on motivation and the development of interculturality and levels of general satisfaction with the project.

The sample is made up of N=566 participants from different Spanish universities: The National University of Distance Education, the European University of Madrid, the University of Castilla-La Mancha, the University of Almería, the University of Córdoba, the University of A Coruña, Jaume I University and the University of Lleida. Figure 3 (https://doi.org/10.6084/m9.figshare.22259278) illustrates the gender distribution according to each centre. The majority are women (60.78%), with men representing 38.64% of the sample and non-binary 0.88%.

Regarding the age of the participants, the data have been grouped into four ranges. From 18 to 30 years old (A), from 31 to 40 years old (B), from 41 to 50 years old (C) and 51 or more years old (D). Figure 4 (https://doi.org/10.6084/m9.figshare.23139083) illustrates the distribution of the sample by age and gender, and from it, one can appreciate that most of the participants belonged to group A (N=300, 53.01%), while range B accommodated 18.37% (N=104), and range C 16.6% (N=94). The least represented range is D with 12.02% (N=68).

The sample includes participants from both state and private universities, and from face-to-face and distance education institutions. There is also a balance in both the age and gender of the participants. To address the code of ethics under which the project operated, on July 5^th 2021, a certificate of suitability was obtained. This certificate allows for the involvement of human beings as part of a research project. In order to guarantee anonymity when working with collated data, each participant was issued a code concealing their identification.

Results and analysis

In this section, a quantitative and inferential analysis is explored. Firstly, a study of the normality of the oral and written production variables and of the oral and written reception is carried out, the results of which have been obtained through the pre-test (ITIS) and post-test (FITIS). Additionally, each variable is studied separately through different statistics. Participant performance is also analysed via the grades obtained in each DAT task.

Inferential analysis

To execute the inferential analysis, it is essential to consider both the sample size and the normality analysis. When dealing with variables that do not meet the normality conditions, non-parametric tests must be used. In this case, the analysis of the impact of the intervention was conducted through average comparisons, using the Wilcoxon test for paired samples. The analysis is divided into four sections, each of which corresponds to each of the basic language skills.

Oral reception

To study the difference in averages in this variable, an alternative hypothesis is proposed. This hypothesis puts forward that the oral reception value in the pre-test is lower than the value of oral reception in the post-test. To arrive at this hypothesis, the Wilcoxon paired sign test was implemented, obtaining the values ​​detailed in Table 2 (https://doi.org/10.6084/m9.figshare.23139518).

The p-value of the Wilcoxon test is statistically significant, entailing the rejection of the null hypothesis and the acceptance of the alternative hypothesis. Therefore, one can assert that the difference in the averages is statistically significant, and the value of the FITIS oral reception is significantly higher than that of the ITIS oral reception, where there is a percentage increase of 16.43%.

Perhaps one of the most interesting aspects to highlight is the homogenizing and beneficial effect of the intervention with regards to this variable, given that the mean score increases at the same time as the standard deviation decreases, as shown in Table 3 (https://doi.org/10.6084/m9.figshare.23139707). Delving into the standard deviation, it is important to clarify that this homogenizing effect is reflected in a percentage reduction, since the standard deviation of the FITIS (oral reception) falls by 21.08%.

Written reception

The analysis of this variable proceeds in a similar vein to that of the oral reception variable. Firstly, the non-parametric test is applied, enabling the study of the alternative hypothesis that asserts that the written reception score in the post-test (FITIS) is higher than that of the pre-test (ITIS), obtaining the results shown in Table 4 (https://doi.org/10.6084/m9.figshare.23139893). Based on the results obtained, the null hypothesis must be rejected and the alternative hypothesis accepted since the p-value is less than .05. As previously mentioned, the representation of the written reception data has been chosen using raincloud diagrams (Figure 6) in order to offer a more complete vision of the data set.

The raincloud diagram depicting the written reception results shows an effect not dissimilar to that of the oral reception variable, albeit in this case the difference is less pronounced. Inspection of Table 5 (https://doi.org/10.6084/m9.figshare.23145800) reveals that this is because the difference between the average and the standard deviation is smoother and, furthermore, the distribution is different. However, the homogenizing effect of the intervention continues to be appreciated, meaning that students improve in a harmonious manner. In percentage terms, the reduction in the standard deviation is 33.39%.

Oral production

In the case of oral production, a non-parametric test is also applied. On this occasion, the alternative variable is that the ratings of the variable are higher after the intervention. The Wilcoxon non-parametric test is applied and the results shown in Table 6 (https://doi.org/10.6084/m9.figshare.23145905) are obtained. Regarding the effect of the intervention on the distributions of the sample, the raincloud diagram (Figure 7) is very illustrative since the distribution of the values ​​of measure 2 (FITIS oral production) around higher values (close to 9) ​​is clearly appreciated.

The descriptive statistics of oral production (Table 7, https://doi.org/10.6084/m9.figshare.23145986) follow the trend of the previous variables whereby the difference in the standard deviation is very pronounced. This is directly related to the shape of the raincloud diagram, since the values ​​after the intervention tend to be more homogeneous and higher, as is the post-test distribution (orange colour). The aspect of the percentage reduction is noteworthy, since there is a drop of 62.91% in the standard deviation from the pre-test to the post-test. This is a very positive statistical indicator given that if the percentage increase in the average qualification of this variable (48.05%) is taken into account, the trend is clearer than in the previous variables towards the improvement of this skill and the homogeneity of performance.

Written production

Written production is studied in a similar way. As previously seen, the alternative hypothesis is considered to be that the scores obtained by the participants in the post-test are higher than those obtained in the pre-test. The Wilcoxon test is applied and the results obtained are shown in Table 8 (https://doi.org/10.6084/m9.figshare.23146079).

The raincloud graph (Figure 8) confirms the trend of the other studied variables (oral production, oral reception and written reception). In the post-test case, the sample distribution generates greater values and the dispersion is lower. The descriptive statistics analysis (Table 9, https://doi.org/10.6084/m9.figshare.23146160) confirms what can be appreciated in Figure 8. Although in this case the decrease in the standard deviation is not excessively pronounced, this does transpire to be the case in the average difference. With regards to this variable, the percentage reduction of the standard deviation is statistically significant since the pre-test is 42.68% greater than the post-test. This results in the differences becoming less pronounced and the hypothesis that this type of intervention has a considerable homogenizing potential can be readdressed.

Discussion and conclusions

This piece of research is pioneering on an international level by presenting results boasting a large sample collated from numerous universities on the use of DAT as a didactic teaching resource and tool for learning a foreign language.

The main result to highlight, and which encompasses the first four proposed objectives, is that there are statistically significant differences that allow us to attest that DAT contributes to improvement in the English learning process as participating students improve in the four linguistics skills: written and oral reception and production. Integrated skills enhancement had been previously advocated for but never explored until now in this field of study. Additionally, there is notable progress in the students' skills in DAT and in a foreign language since the beginning of the didactic intervention, concurring with similar results from previous research (Talaván, 2020). Furthermore, students express a favourable perception of the didactic intervention and the use of DAT as a didactic resource in the classroom, declaring clear preferences for the modes which are more familiar to them: dubbing and standard subtitling.

This research opens new doors in the field of language didactics by providing empirical data on the didactic possibilities of DAT. The objective of the research is not to propose a specific language teaching method, but to contribute to the teaching and learning process by providing teachers with new tools and pedagogical resources that they can integrate into their teaching. To address weaknesses, it is worth mentioning the dropout rate (which, although expected, could always be lower) and the time to complete the LP, which, although it was adjusted to what was expected in many cases, in others it exceeded more than was expected. Both of these points will be studied more closely in future related research. It is also necessary to mention that the value of the results must be underplayed, taking into account the frenetic rate of technological transformation, which clearly impacts time and research processes, as well as paradigm changes in university education and in education in general.

DAT is a versatile tool that can be used at different educational levels (primary, secondary and university) and can be included both in traditional learning and in hybrid or virtual modes. The field of DAT is a fertile line of research and based on the present piece of research, more studies can be carried out on it. Future studies could focus on understanding teacher perceptions of the use of DAT as a teaching tool, analysing the cumulative effect of DAT as a resource pedagogy in longitudinal projects, or its inclusion in bilingual education programs that work with two languages. To the same degree, analysis of the use of DAT in multilingual environments (with co-official and/or minority languages), as well as the possibilities it offers to improve accessibility for students in the language classroom would also be of interest. (1)