VALIDATION STUDY OF COLOMBIA’S ECAES ENGLISH EXAM.

The ECAES exam is a test designed for students in their last year at the university. As a pre-service English teacher, I feel curious about the ECAES English exam due to the fact, that I have to take it next year and I don’t get so much information of what it is about.

This is a mandatory test and it has four main purposes:

Ø  To assess the level of competency of students after completing a higher education program.

Ø  To provide of information for building indicators of the quality of higher education programs and institutions which inform institutional qualification processes, policy design, and decision making, at every level of the system.

Ø  To compare and analyze the performance of higher education institutions and identify trends from the characteristics of these institutions which account for differences in quality between levels, types of institution, teaching methods, etc.

Ø  To produce indicators of the value added by higher education in terms of acquired competencies. (icfes.gov.co)

The document:” VALIDATION STUDY OF COLOMBIA’S ECAES ENGLISH EXAM.” is a research study conducted at Universidad de los Andes. The participants were 15 university English language teachers.Data was collected using two tools:   Content Evaluation Sessions and Think-aloud protocol.

In the document, the researchers present the findings after examining the validity of the mentioned test.

First, the authors explained the following: Students take a battery of three exams, one of which is related to their field of study to prove proficiency in the subject area. The other two exams are the same for all students: a Spanish reading exam and the English exam. (López and Janssen, 2010).

They explained the concepts of validity that we studied in our assessment class and based on those concepts, they analyzed the parts of the test,the validity of it and the usefulness of it. 

The ECAES English Exam can be considered a low-stakes test since it is only used to inform the public about the effectiveness of the new language education policy, and no important decisions are taken based on test scores. (López and Janssen, 2010).

The exam only assesses the following skills: reading, vocabulary and grammar.  Listening, speaking and writing are not assessed because of practicality reasons (difficulty in setting up adequate technology to assess listening and difficulty in training raters for speaking and writing).

In the findings the authors mentioned relevant aspects such as:

POSITIVE

NEGATIVE

-Assessing grammar and vocabulary skills are more authentic than other parts of the test. - These parts are discourse-based tasks and do not assess syntactical and lexical knowledge and skills in isolation. - The test has no major negative effects on students, teachers, or programs.

- Teachers feel that the ECAES English Exam is not adequately aligned to the CEFR level descriptors. - The content of the test is not fully aligned to the content of their foreign language programs. - Some parts of the ECAES English Exam are too difficult or too easy for some test-takers. - Lack of authentic language use in the tasks. - A lot of guessing, due to the selected response test format. - the construct of the test fails to include relevant aspects of general English language ability (e.g. listening, speaking, and writing) and many test items do not require students to use a wide range of areas of English language knowledge

I found the document useful for our assessment class and for the project that I'm doing. It shows facts based on evidence and the concepts are really well defined, so I found room for clarification of some aspects of validation.



INTERPRETING TEST SCORES

Before reading the text:”Interpreting test scores”, I haven’t thought about the different ways of interpreting scores based on factors such as: -The frame of reference for the scores, -the type of scores, - how the test has been constructed.

I realized that interpreting results is such a complex process that I ignored completely. First, we need to consider the type of test, if it is a standardized test, mastery test, open-ended test or any other, because each type of test has a purpose on evaluating.

Second, we have to select a base for the interpretation.For instance, when we want to interpret scores based on a frame of reference it means that we know other learners’ results on the same test and taking into account the grade level and general language ability we can make a comparison between the performances of a specific group and the reference group. In this kind of interpretation we don’t analyze what students have learned.

Another frame mentioned is the domain-referenced test, an exam that reflects Ss’ domain of language skills in which the score can be interpreted in terms of the achievements each one has gotten.

It is also stated by the authors that the level of difficulty of test influences the sorts of interpretations that are possible (Genesee and Upshur 1996).

In the process of assessing we need to have clear objectives and select the task for evaluating the achievement of those objectives but keeping in mind that students’ performance will be influenced by the difficulty of the task.

Another factor to consider when interpreting is the type of score: Counts or Ratings. Due to the fact, that there are established bases for interpreting counts or ratings following two procedures.

As I said before, I found that interpret a score or result from any exam is a complex process which requires time, attention and the appropriation of such important concepts regarding interpretation of students’ results in order to have accurate analysis. After reading this chapter, I learned that grading a test is more than put a mark, letter or score. It is more than tell a student you need to work harder on this language aspect. It is definitely a process that demands interpretation.