A Search for Accountability in Low-stakes Tests: An Evidence-based Approach to Validation of a Local Writing Test

Ansari, Ali

doi:10.22034/ijpie.2024.469159.1032

A Search for Accountability in Low-stakes Tests: An Evidence-based Approach to Validation of a Local Writing Test

Document Type : Original Article

Author

Ali Ansari

Education Ministry

10.22034/ijpie.2024.469159.1032

Abstract

Validation has always been a significant part of assessment in educational settings. However, due to its complexities, most officials prefer to use ready-made tests to avoid validating their own tests. This study validates a screening writing test in an EFL context using an evidence-based validation approach. This study reflects the researcher’s attempts to collect evidence to justify the newly developed test use and interpretation. The participants of this study were 53 Iranian B1 English language learners. To collect the required evidence, the researcher employed a questionnaire to investigate the learners’ cognitive processes that they went through to write the texts. Furthermore, the scores were correlated with the participants’ scores on a First Certificate in English (FCE) writing task to attain the criterion-related validity evidence. Evidence related to scoring validity and consequential validity was also collected by employing simple statistical measures, such as correlation, t-test, and percentage, that are easy for teachers to run. The findings of the study showed the (cognitive, context, scoring, consequential, and criterion-related) validity of the developed test. The pieces of evidence collected in this study supported the validity of the test for the intended use. The present report can enlighten novice teachers as to how to collect a body of evidence to ensure different stakeholders about the validity of their tests. While validation might be perceived as a complicated process, which is not within teachers’ ability, this study can provide a blueprint to guide teachers on how to show their accountability through easy-to-implement steps.

Keywords

Accountability

Evidence-based Validity

Second Language Testing

Validation

Writing

Subjects

Testing

Ali, A., & Kaivanpanah, S. (2024). Writing Assessment Literacy in an EFL Context: Insights from Iraqi Kurdish Teachers. Applied Research on English Language, 13(1), 55-78.

Allan, D. (2004). Oxford Placement Test. Oxford University Press.

Allison, D. (1999). Language testing and evaluation: An introductory course. Singapore: Singapore University Press/ World Scientific.

Bachman, L. (1990). Fundamental Considerations in Language Testing. Oxford: Oxford University Press.

Bachman, L. (2005). Building and supporting a case for test use. Language Assessment Quarterly 19 (4), 453-476.

Bachman, L., & Palmer, A. (2010). Language assessment in practice. Oxford University Press.

Brunfaut, T. (2023). Future challenges and opportunities in language testing and assessment: Basic questions and principles at the forefront. Language Testing, 40(1), 15-23.

Chapelle, C. A., Enright, M. K., & Jamieson, J. (Eds.) (2008). Building a validity argument for the Test of English as a Foreign Language. Routledge.

Cronbach, L. J. & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin 52, 281–302.

Fattah, Z. (2024). The Effect of Portfolio Assessment on Iranian EFL Learners’ Writing Ability. International Journal of Practical and Pedagogical Issues in English Education, 2(2), 13-34.

Field, J. (2005). Second language writing: a language problem or a writing problem? Paper presented at IATEFL 'Writing Revisited' Conference, Cambridge, 25-27 February 2005.

Ginther, A., & Grant, L. (1997). The influence of proficiency, language background, and topic on the production of grammatical form and error on the Test of Written English. In A. Huhta, V. Kohonen, L. Kurki-Suonio & S. Luoma (Eds.), Current developments and alternatives in language assessment (pp. 385–97). University of Jyväskyla.

Granena, G. (2023). Cognitive individual differences in the process and product of L2 writing. Studies in Second Language Acquisition, 45(3), 765-785.

Hoomanfard, M. H., & Meshkat, M. (2015). Writing on a computer and using paper and pencil: Is there any difference in the internal cognitive processes? GEMA Online Journal of Language Studies, 15(2), 17-31.

Hoomanfard, M. H., Jafarigohar, M., Jalilifar, A., & Masum, S. M. H. (2018). Comparative study of graduate students’ self-perceived needs for written feedback and supervisors’ perceptions. Journal of Research in Applied Linguistics, 9(2), 24-46.

Huggins-Manley, A. C., Huang, J., Danso, J. A., Li, W., & Leite, W. L. (2024). Classroom assessment and instructional modes: An exploration of school-level contextualized psychometric challenges. The Journal of Experimental Education, 92(3), 513-530.

Izadpanah, M. A., Rakhshandehroo, F., Hoomanfard, H. M., & Mahmoudikia, M. (2014). On the consensus between holistic rating system and analytic rating system: A comparison between TOEFL IBT and Jacobs et. al. composition profile. International Journal of Language Learning and Applied Linguistic World, 6(1), 170-187.

Jacobs, H. L., Zinkgraf, D. R., Wormuth, V. F., Hartfiel, V. F. & Hughey, J. B. (1981). Testing ESL composition: A practical approach. Newbury House.

Jafarigohar, M., Hoomanfard, M. H., & Jalilifar, A. (2018). A Typology of Supervisor Written Feedback on L2 Students' Theses/Dissertations. Iranian Journal of Applied Linguistics, 21(2), 43-87.

Jensen, C. & Hansen, C. (1995). The effect of prior knowledge on EAP listening-test performance. Language Testing, 12(1), 99-119.

Jia, W., & Zhang, P. (2023). Rater cognitive processes in integrated writing tasks: From the perspective of problem-solving. Language Testing in Asia, 13(1), 50.

Kane, M. T. (2006). Validation. In Brennan, R. L. (Ed.), Educational measurement (pp.18–64). American Council on Education/Praeger.

Kim, M., Tian, Y., & Crossley, S. A. (2021). Exploring the relationships among cognitive and linguistic resources, writing processes, and written products in second language writing. Journal of Second Language Writing, 53, 100824.

McNamara, T. (2001). Language assessment as social practice: Challenges for research. Language Testing, 18(4), 333-350.

Mehrpour, S., Hoomanfard, M. H., & Vazin, E. (2023). Peer Feedback Accuracy in Synchronous and Asynchronous Computer-Mediated Conditions in an EFL Context. Iranian Journal of Language Teaching Research, 11(1), 97-116.

Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H.I. Braun (Eds.). Test Validity. Lawrence Erlbaum Associates. 33-45.

Messick, S. (1989). ‘Validity.’ In Linn, R. L. (ed.) Educational measurement (pp. 13-103). Macmillan/American Council on Education.

Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of assessment arguments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3–62.

Norton, B. (1997). ‘Accountability in language assessment.’ In Clapham, C. and Corson, D.(eds) Encyclopedia of Language and Education, vol. 7: Language Testing and Assessment (313–322). Kluwer Academic Publishers.

Shaw, S., & Weir, C. J. (2007). Examining writing in a second language, studies in language testing 26. Cambridge University Press/Cambridge ESOL.

Sunderland, J. (2000). New understandings of gender and language classroom research: texts, teacher talk and student talk. Language Teaching Research 4 (2), 149-173.

Van Dijk, T. A. (2011). Discourse studies: A multidisciplinary introduction (2nd ed.). SAGE Publications.

Wang, T., & Lajoie, S. P. (2023). How does cognitive load interact with self-regulated learning? A dynamic and integrative model. Educational Psychology Review, 35(3), 69.

Weigle, S.C. (2007). Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194-209.

Weir, C. (2005). Language testing and validation: An evidence-based approach. Palgrave.

Yakut, I., & Bada, E. (2022). Interlanguage development of Turkish speakers of English: Exploring oral and written communication strategies. Journal of Teaching English for Specific and Academic Purposes, 611-626.

Zainal, A. (2012). Validation of an ESL writing test in a Malaysian Secondary School Context. Assessing Writing 17(1), 1-17.