VALIDATION STUDY OF COLOMBIA’S ECAES ENGLISH EXAM.

The ECAES exam is a test designed for students in their last year at the university. As a pre-service English teacher, I feel curious about the ECAES English exam due to the fact, that I have to take it next year and I don’t get so much information of what it is about.

This is a mandatory test and it has four main purposes:

Ø  To assess the level of competency of students after completing a higher education program.

Ø  To provide of information for building indicators of the quality of higher education programs and institutions which inform institutional qualification processes, policy design, and decision making, at every level of the system.

Ø  To compare and analyze the performance of higher education institutions and identify trends from the characteristics of these institutions which account for differences in quality between levels, types of institution, teaching methods, etc.

Ø  To produce indicators of the value added by higher education in terms of acquired competencies. (icfes.gov.co)

The document:” VALIDATION STUDY OF COLOMBIA’S ECAES ENGLISH EXAM.” is a research study conducted at Universidad de los Andes. The participants were 15 university English language teachers.Data was collected using two tools:   Content Evaluation Sessions and Think-aloud protocol.

In the document, the researchers present the findings after examining the validity of the mentioned test.

First, the authors explained the following: Students take a battery of three exams, one of which is related to their field of study to prove proficiency in the subject area. The other two exams are the same for all students: a Spanish reading exam and the English exam. (López and Janssen, 2010).

They explained the concepts of validity that we studied in our assessment class and based on those concepts, they analyzed the parts of the test,the validity of it and the usefulness of it. 

The ECAES English Exam can be considered a low-stakes test since it is only used to inform the public about the effectiveness of the new language education policy, and no important decisions are taken based on test scores. (López and Janssen, 2010).

The exam only assesses the following skills: reading, vocabulary and grammar.  Listening, speaking and writing are not assessed because of practicality reasons (difficulty in setting up adequate technology to assess listening and difficulty in training raters for speaking and writing).

In the findings the authors mentioned relevant aspects such as:

POSITIVE

NEGATIVE

-Assessing grammar and vocabulary skills are more authentic than other parts of the test. - These parts are discourse-based tasks and do not assess syntactical and lexical knowledge and skills in isolation. - The test has no major negative effects on students, teachers, or programs.

- Teachers feel that the ECAES English Exam is not adequately aligned to the CEFR level descriptors. - The content of the test is not fully aligned to the content of their foreign language programs. - Some parts of the ECAES English Exam are too difficult or too easy for some test-takers. - Lack of authentic language use in the tasks. - A lot of guessing, due to the selected response test format. - the construct of the test fails to include relevant aspects of general English language ability (e.g. listening, speaking, and writing) and many test items do not require students to use a wide range of areas of English language knowledge

I found the document useful for our assessment class and for the project that I'm doing. It shows facts based on evidence and the concepts are really well defined, so I found room for clarification of some aspects of validation.



INTERPRETING TEST SCORES

Before reading the text:”Interpreting test scores”, I haven’t thought about the different ways of interpreting scores based on factors such as: -The frame of reference for the scores, -the type of scores, - how the test has been constructed.

I realized that interpreting results is such a complex process that I ignored completely. First, we need to consider the type of test, if it is a standardized test, mastery test, open-ended test or any other, because each type of test has a purpose on evaluating.

Second, we have to select a base for the interpretation.For instance, when we want to interpret scores based on a frame of reference it means that we know other learners’ results on the same test and taking into account the grade level and general language ability we can make a comparison between the performances of a specific group and the reference group. In this kind of interpretation we don’t analyze what students have learned.

Another frame mentioned is the domain-referenced test, an exam that reflects Ss’ domain of language skills in which the score can be interpreted in terms of the achievements each one has gotten.

It is also stated by the authors that the level of difficulty of test influences the sorts of interpretations that are possible (Genesee and Upshur 1996).

In the process of assessing we need to have clear objectives and select the task for evaluating the achievement of those objectives but keeping in mind that students’ performance will be influenced by the difficulty of the task.

Another factor to consider when interpreting is the type of score: Counts or Ratings. Due to the fact, that there are established bases for interpreting counts or ratings following two procedures.

As I said before, I found that interpret a score or result from any exam is a complex process which requires time, attention and the appropriation of such important concepts regarding interpretation of students’ results in order to have accurate analysis. After reading this chapter, I learned that grading a test is more than put a mark, letter or score. It is more than tell a student you need to work harder on this language aspect. It is definitely a process that demands interpretation.

CHOOSING AND DEVISING TEST TASKS

Along this course, we have been talking about different ways, tools and strategies for assessing students in second language learning processes. We have also discussed important concepts such as Reliability, validity, practicality, washback among others that we are starting to implement in our practices.

Now, new concepts need to be considered according to the reading Choosing and Devising test tasks. It seems to be hard for me as a teacher to deal with the word devising, frankly speaking I would like to avoid devise test tasks, but when you have been teaching a course and at the end of it you have to apply an assigned test, sometimes it could be harder than devise one.

I had a bad experience regarding test tasks at the beginning of this year. I was teaching a level 1 group of learners and when the time for the final test arrived I administered an assigned test. The exam seemed to be appropriate for them but when they started to answer it, the problems showed up. The listening part was so complex for their level, besides the complexity of the content, we haven’t worked in the type of exercise that they had to develop. My students complained and although I knew they were right I couldn’t skip that part I just encouraged them to do it. I felt so bad because the majority of them failed the listening section and it wasn’t the purpose. While reading the text, I realized that the fault was in the instructional objectives and instructional activities,(concepts cited by the authors).

 In the reading “Choosing and devising test tasks” the authors mentioned four important factors to take into account for choosing the tasks of a test.

ü  Instructional objectives: The task will be focused on the same kind of language skills described in the objectives

ü  Students’ level of proficiency: which task is more suitable for a beginner and which is more suitable for an advanced learner. In this part, we talk about open-ended and close-ended response tasks.

ü  Instructional activities:  the activities selected must be known for the students, if they are familiar with the kind of exercise they will have a good performance.

ü  Testing resources: Resources for taking into account such as time for administering and scoring the test, contents and physical space.

Maybe we had a wide idea about these new concepts but, what about applying them in our tests? I found some useful questions that we can ask ourselves at the moment of choosing the objectives: Do the objectives include all important outcomes of the course? Are the objectives realistic in terms of the abilities of the students and the time available? (Linn & Miller 2004).

It is also important to have a clear idea about what type of learning outcome do we expect from our students. Is it about Knowledge, Thinking skills, Performance skills,  Application?

Having set the objectives, activities, resources, level of proficiency and a clear outcome we will be ready for designing a test with the appropriate tasks.

AMPLIFYING SOME CONCEPTS

In our last class of Assessment, We were talking about the concept of collaboration and collaborative learning, but the time was not enough to discuss about what cooperative learning is and what the differences between those terms are.

When the class finished I was confused with the definitions of the words, so I came back home and decided to look for the concepts and the differences, and I want to share this information in my blog because it would help us to clarify and amplify our ideas about cooperative and collaborative learning.

After reading the definitions of the concepts, I picked up some and wrote them on a chart, and here are some of the results:

COLLABORATIVE LEARNING

COOPERATIVE LEARNING

Collaboration is a philosophy of interaction and personal lifestyle. -Collaborative learning is a personal philosophy, not just a classroom technique. In all situations where people come together in groups, it suggests a way of dealing with people which respects and highlights individual group members' abilities and contributions. (Panitz 1996) -Rockwood (1995) states: collaborative learning is connected to the social constructionist's view that knowledge is a social construct. -Collaborative learning is based on the idea that learning is a naturally social act in which the participants talk among themselves (Gerlach, 1994). It is through the talk that learning occurs.

Cooperation is a structure of interaction designed to facilitate the accomplishment of an end product or goal. - Cooperative learning is defined by a set of processes which help people interact together in order to accomplish a specific goal or develop an end product which is usually content specific. It is more directive than a collaborative system of governance and closely controlled by the teacher. (Panitz 1996) - Rockwood (1995) defines: Cooperative learning is the methodology of choice for foundational knowledge. - Cooperative learning involves students in a major task and gives responsibility to all the individual learner in the learning process.(Paulsen 2003)

Now, I have a clear idea about these methodologies and learning processes.

Here are some links, where we can find and read more about collaborative and cooperative learning; the first one is a slide share presentation that gives definitions, characteristics, comparisons and advantages about the concepts mentioned.

http://www.slideshare.net/carolzinhanep/webinar-1-9071080

http://celt.ust.hk/teaching-resources/cooperative-and-collaborative-learning

http://www.londonmet.ac.uk/deliberations/collaborative-learning/home.cfm

OBSERVATION IN THE CLASSROOM

Our classrooms environments are plenty of new experiences each time when a lesson starts. Students’ expectation towards English class grows or reduces in certain way, and that is one of the reasons why we have to deal with different behaviors, attitudes, feelings and thoughts in the day-to-day practices.

 Classroom observation becomes one of the most useful tools for teachers when making inferences about learning and teaching processes, collecting information, assessing students’ performance and our own practices. However, this method needs to have a purpose in order to work properly.

First of all, observation should be organized with reference to specified language objectives. (Genesee & Hupshur). Thus, classroom observation will be manageable and systematic because it defines and delimits what is to be observed. It is also important to take into consideration the observation of instructional plans, instructional content, materials and activities. Because,it provides an explanation about students’ difficulties in their learning process, and better ideas for us when implementing a different strategy to teach.

It is mandatory to record classroom observations in order to not forget what was observed, for that task we can use different options such as anecdotal records, checklist and rating scales. How to determine what do we want to use? It depends mainly on the purpose of the observation we have, the way in which the information will be collected and how we want to report the analyzed results.

Due to the fact, that we are observing our students all the time, as a personal matter I consider that is really important for me to organize my observations keeping in mind the concepts in the reading and take the time to choose the appropriate way to record the information collected.  

COLLECTING INFORMATION

We have been learning that evaluate or assess is a really complex process which demands different tools and strategies to give an accurate result. This process also requires the collection and interpretation of data (qualitative and quantitative) for providing outcomes. For doing this, it is necessary to mention three principles: PRACTICALITY, How practical in terms of money and time the information was collected, RELIABILITY, how reliable the information is in order to give outcomes, and VALIDITY, how valid the information is regarding what is being assessed.

Although practicality is not the most relevant aspect in here, we need to mention again the importance of time and cost in the methods we are using to evaluate. Reliability is split up in three: Assessor related reliability (it has to do with instability among the people collecting the information), Object-related reliability (it is regarding the person about whom the information is being collected) and Instrument- related reliability (Procedures used to collect information).

The validity factor is the extent to which the information you get is the information you want. (Genesee & Upshur).

As teachers we need to be aware of those aspects when the time of assessing comes, taking into account that we are tools either for improving or frustrating learning processes. A personal question arises here. How reliable and valid have been our processes of gathering information so far?

PRINCIPLES OF LANGUAGE ASSESSMENT

 Since I started my teaching experience I have been involved in the process of designing exams to assess or sometimes just to grade my students. When the time of evaluations arrives, it is a nightmare for all the teachers, just for thinking on how long the test would be or how much time we are going to take for grading it. However, after reading the text Principles of language assessment I can realize why we have this conception. 

There are five principles that we need to know and take into account in our tests mention by the author, those are:  practicality, reliability, validity, authenticity and washback. What are they about? It is mention that a test shouldn’t be difficult to administer and needs to have a specific and time efficient scoring procedure to make it practical. The reliability is related to accuracy, consistency, dependability or fairness of scores. Validity is the most relevant and complex factor. It deals with content, objectives, and students criteria. Besides, it has to be appropriate, meaningful and useful in terms of purpose of assessment. An authentic test has interesting topics, describes a real world context, items are contextualized rather than isolated and the language used is as natural as possible. Finally, it is important to give students the chance to be aware of their weaknesses and how they could improve, this is the washback principle.

I was not aware of those concepts and perhaps I was not the only. After reading the chapter and learn about the principles of assessment, It would be a challenge for me design test which includes these features but I’m sure that this challenge will bring a great satisfaction to me as teacher and at the same time better and meaningful results for my students.