Language Testing Article Alert

La pr�sente �tude vise � v�rifier si un enseignement du vocabulaire dans deux contextes d'immersion diff�rents permet une am�lioration des connaissances lexicales. Cet article examinera premi�rement le contexte de notre �tude effectu�e aupr�s de 11 �tudiants en droit et 13 �tudiants en histoire inscrits � l'universit� d'Ottawa dans un cours d'encadrement linguistique associ� � un cours d'immersion. Puis, nous d�crirons le protocole utilis� : chaque semaine, pendant une session universitaire, les �tudiants ont compl�t� une s�rie d'activit�s reli�es � un vocabulaire disciplinaire provenant de leurs lectures et de leurs cours. Nous pr�senterons �galement les r�sultats des analyses aux diff�rents tests (pr�-test, post-test et test diff�r�) pour v�rifier l'am�lioration des connaissances chez les �tudiants et la r�tention de ce vocabulaire entre le d�but et la fin de la session, entre les mots enseign�s et les mots non enseign�s. Nos r�sultats permettront ainsi de proposer les meilleures approches p�dagogiques dans les deux cours vis�s, approches qui tiendront compte des caract�ristiques sp�cifiques au vocabulaire de chacune des disciplines.

Content Type Journal ArticleCategory ArticlesPages 1-27DOI 10.3138/cmlr.68.1.001Authors Alysse WeinbergDalila BoukacemSandra Burger Canadian Modern Language Review/ La Revue canadienne des langues vivantesOnline ISSN 1710-1131Print ISSN 0008-4506 Volume 68 Volume 68, Number 1 / February 2012
(show less)

In this cross-sectional study, we explore the relationship between prosodic sensitivity (suprasegmental phonology) and phonological awareness (segmental phonology) and investigate whether a group of poor readers display significant suprasegmental phonological deficits in comparison to chronological age-matched controls and younger, reading age-matched controls. Phonological awareness assessments were administered along with a battery of prosodic sensitivity assessments drawn from recent literature. The results showed that poor readers were outperformed by their chronological age-matched counterparts on all measures of prosodic sensitivity. A significant main effect of group was found on the revised stress mispronunciations task and the stress assignment task from the prosodic assessment battery, the former of which remained even after controlling for individual differences in receptive vocabulary and measures of phonological awareness. Significant relationships were also found between measures of prosodic sensitivity and phonological awareness (especially phoneme awareness). These findings emphasise the importance of both segmental and suprasegmental phonological skills in children's reading development.
(show less)

Publication year: 2012
Source: Journal of Pragmatics, Available online 30 January 2012
Tuomo�Hiippala
This paper argues that the concept of a reading path in multimodal research can be improved by previous research on visual perception in psychology and brain sciences, and particularly by the work done within eye-tracking studies. The paper argues that in its current state, the concept of a reading path is not sufficiently reliable due to the lack of empirical testing and therefore presents a methodological proposal to improve the current situation.Thus, the paper identifies common areas of interest related to visual perception, where the research interests of the disciplines meet and enable reciprocal input. It is suggested that multimodal research is capable of describing the high-level factors that affect visual perception, whereas eye-tracking equipment can track the actual reader behaviour. Applicable state-of-the-art theories of multimodal analysis are then described, along with the technological requirements for the eye tracker and its software.XML annotation, output and transformations are proposed for combining the results of multimodal analysis and the observer behaviour captured using an eye tracker. Finally, the paper presents a hypothesis on the relationship of visual perception and multimodal semiosis, which may be evaluated using the proposed method combining multimodal analysis and eye-tracking.
Highlights? The concept of a ?reading path? in multimodal research is contested by research in psychology and brain sciences. ? Multimodal research may describe certain high-level factors of visual perception. ? Methods of multimodal analysis and eye-tracking studies can be interfaced via XML.
(show less)

Publication year: 2012
Source: System, Available online 30 January 2012
Cindy�Brantmeier, Robert�Vanderplank, Michael�Strube
In an investigation with advanced language learners, Brantmeier [Brantmeier, C., 2006. Advanced L2 learners and reading placement: self-assessment, computer based testing, and subsequent performance. System 34 (1), 15?35.] reports that self-assessment (SA) of second language (L2) reading ability, when measured with self-rated scales, is not an accurate predictor of subsequent reading performance as measured via multiple choice items. In another experiment with advanced learners that utilizes criterion-referenced SA items, Brantmeier and Vanderplank [Brantmeier, C., Vanderplank, R., 2008. Descriptive and criterion-referenced self assessment with L2 readers. System 36 (3), 456?477] reveal that learners accurately estimate their reading comprehension when it is measured via multiple choice items. For the present study, an SA instrument of language learning achievement was designed according to specific course content to take into consideration the direct experience learners have had in practicing reading, listening, speaking, and writing [Ross, S., 1998. Self-assessment in second language testing: a meta-analysis of experimental factors. Language Testing 15, 1?20.]. With 276 participants, the study examines skill-based SA across beginning, intermediate and advanced levels of language instruction, and it offers evidence to validate the relationship between the SA instrument and achievement on an online abilities test with advanced learners. Findings hold important implications for language learner assessment, especially in terms of advanced students? ability to rate themselves when given specific criteria. A discussion about the value of SA as a complement to other traditional approaches for language program assessment is offered.
(show less)

Using the nationally representative, cohort-based data of the Education Longitudinal Study of 2002 (ELS:02), this study employs multiple regression to examine the effects of exit exams on student achievement and school completion. This study finds that exit exams as a whole do not have substantial effects on student achievement in mathematics, twelfth grade GPA, or school completion. Standards-based exams are a positive predictor of dropping out of school but lose their predictive power once GED recipients are coded as completing school. Exit exams do not affect GED seeking and acquisition. When exit exams are disaggregated by type and students are sorted by ninth grade GPA quartiles, end-of-course exams have some negative effects on mathematics test score gains. Students in the bottom two quartiles see reduced test score gains of 28% and 29% of a grade level equivalency (GLE). These effects disappear when students in North Carolina are coded as taking a different type of exam. Standards-based exams had a small positive effect, about 37% of a GLE, on the top quartile of students. Overall, the findings showed no results for school completion and mixed results for test score gains. The article concludes that policymakers looking to boost high school achievement would be better served by working to boost student accomplishments before high school. Normal.dotm 0 0 1 214 1224 cmc 10 2 1503 12.0 0 false 18 pt 18 pt 0 0 false false false
(show less)

Abstract Randomization tests are often recommended when parametric assumptions may be violated because they require no distributional or random sampling assumptions in order to be valid. In addition to being exact, a randomization test may also be more powerful than its parametric counterpart. This was demonstrated in a simulation study which examined the conditional power of three nondirectional tests: the randomization t test, the Wilcoxon?Mann?Whitney (WMW) test, and the parametric t test. When the treatment effect was skewed, with degree of skewness correlated with the size of the effect, the randomization t test was systematically more powerful than the parametric t test. The relative power of the WMW test under the skewed treatment effect condition depended on the sample size ratio.

Content Type Journal ArticlePages 1-15DOI 10.1007/s11336-012-9249-5Authors Bryan Keller, University of Wisconsin?Madison, Madison, WI, USA PsychometrikaOnline ISSN 1860-0980Print ISSN 0033-3123
(show less)

Kane’s argument-based framework is summarized and examined. He implicitly appeals to the backgrounded concepts of fairness and justice. From there it is a short distance to grounding the whole system in the mundane notion of truth. In fact, valid argument systems must depend on representations that are ‘true’ by virtue of agreement with purported facts. As a friendly amendment, therefore, I argue that (provided the ceteris paribus, all else being equal, requirement is met) agreement with known facts in testing, experimental research, and scientific measurement counts for a great deal more than disagreement. It follows by Peircean ‘exact logic’ that higher test scores (if the tests have any validity at all) are invariably more informative (interpretable in general) and thus more useful than lower scores. Why? Because higher scores show more agreement between the test-makers and the higher scoring test-takers about whatever facts (or performances) may be at issue. Exceptions are cases where the ceteris paribus requirement is not met. Necessary (but testable) inferences follow for interpretations and uses of ‘cutscores.’

(show less)

This study investigated the relationships among the quality of L2 test takers’ notes evaluated in terms of different levels of information and test takers’ performance on open-ended listening tasks tapping into different comprehension subskills. In addition, this study examined the invariance of the structural relationships among the variables across two different note-taking formats, that is, a blank format and an outline format, by employing a multi-group structural equation modeling (SEM) approach. The results indicated that note quality measures, in particular the number of topical ideas found in the notes and the organization of these notes, may be good indicators of test takers’ second language academic listening proficiency. It was also found that despite the invariance of structural relationships among variables across the two note-taking formats, the associations between the open-ended listening measures and note quality measures were slightly stronger in the outline format than in the blank format. The implications of these results for L2 academic listening assessment are considered.

(show less)

The important yet under-researched role of item writers in the selection and adaptation of texts for high-stakes reading tests is investigated through a case study involving a group of trained item writers working on the International English Language Testing System (IELTS). In the first phase of the study, participants were invited to reflect in writing, and then audio-recorded in a semantic-differential-based joint discussion, on the processes they employed to generate test material. The group were next observed at a simulated item writers’ editing meeting to refine their texts and items for an IELTS reading test module. The participants’ written descriptions and recorded discussions provided rich data on how source texts were perceived, selected and adapted for the Test. The study reports findings from textual analyses using indices of readability and lexical density from the original material sourced by the item writers and their adapted versions for the test. Results from qualitative and quantitive analyses are discussed in terms of the implications for the IELTS reading module of editing actions such as: reducing redundancy and technical language, changing styles, deciding on potentially sensitive issues and relationships between texts and test items. The important issue of text authenticity in tests such as IELTS is also broached.

(show less)

Abstract In a number of psychological studies, answers to reasoning vignettes have been shown to result from both intuitive and deliberate response processes. This paper utilizes a psychometric model to separate these two response tendencies. An experimental application shows that the proposed model facilitates the analysis of dual-process item responses and the assessment of individual-difference factors, as well as conditions that favor one response tendency over another one.

Content Type Journal ArticlePages 1-12DOI 10.1007/s11336-012-9251-yAuthors Ulf B�ckenholt, Kellogg School of Management, Northwestern University, 2001 Sheridan Road, Evanston, IL 60208, USA PsychometrikaOnline ISSN 1860-0980Print ISSN 0033-3123
(show less)

This study examined the effectiveness of explicit instruction on second language (L2) learners’ implicit and explicit knowledge of English. Explicit instruction on the generic and non-generic use of English articles was delivered by CALL activities. Four tasks assessed acquisition: elicited imitation, oral production, grammaticality judgement, and metalinguistic knowledge tasks. A pretest and two posttests were conducted immediately and six weeks after the treatment. Durable effects for explicit instruction were found on measures of implicit knowledge and on ungrammatically supplied items on measures of explicit knowledge. This study’s findings contribute towards our understanding of the efficacy of explicit instruction on implicit and explicit knowledge at relatively advanced stages of L2 acquisition.

(show less)

Abstract The aim of the present research is to examine the viability of the construct validity of the speaking modules of two internationally recognized language proficiency examinations, namely IELTS and TOEFL iBT. High-stake standardized tests play a crucial and decisive role in determining the future academic life of many people. Overall obtained scores of candidates are believed to reflect their general proficiency level. Appropriate interpretation and use of test scores depend on the extent to which items measuring a particular skill (here speaking) can meet the criteria to examine the intended construct. Speaking, amongst the other four skills, has a central place in assessing general proficiency of the candidates. This research seeks to scrutinize how IELTS and TOEFL iBT tap on the speaking proficiency of their candidates. Moreover, this study investigates whether obtained speaking scores of candidates in these two international high-stake tests show an acceptable degree of consistency in measuring the skill being examined. The chosen sample of the study consisted of 60 students who successfully completed TOEFL iBT and IELTS preparation courses in Tehran. The results of the statistical analysis show that there is a meaningful discrepancy between the two exams in assessing the speaking abilities of the exam-takers and therefore challenge the construct validity of the exams in question. Findings are then used to discuss the repercussions for language proficiency measurement and assessment.

Content Type Journal ArticlePages 1-15DOI 10.1007/s11092-011-9137-zAuthors Keivan Zahedi, Linguistics, Shahid Beheshti University, General Campus, Tehran, IranSaeedeh Shamsaee, TEFL, Ferdowsi University of Mashhad, No. 1, Block 5, Pamchal Complex, Goldis Boulevard, Shahid Borunsi Boulevard, Mashhad, Iran Educational Assessment, Evaluation and AccountabilityOnline ISSN 1874-8600Print ISSN 1874-8597
(show less)

Abstract The purpose of the study was to develop an improved teaching evaluation questionnaire based on students? learning experiences and selected teacher characteristics identified as indicators of teaching quality. Teaching evaluation questionnaires are commonly designed either based on agreed indicators of teaching excellence, students? suggestions of characteristics of excellence, or dimensions identified from interviews conducted with excellent teachers. In this study, however, students? evaluation of their own learning experiences in learner centered classes and a hypothesized framework derived from the literature were used to develop the items for the questionnaire. Students attended to courses designed in a learner centered paradigm and evaluated their learning experiences. Based on the hypothesized framework and students? reflections of their experiences, items were designed and validated at various levels. Exploratory factor analysis resulted in a 24-item Student Evaluation of Learning and Teaching Questionnaire (SELTQ) that comprises four factors: assessment and feedback; course organization and presentation; student self evaluation; and students? level of engagement. As opposed to the questionnaire widely used in higher education institutions in Ethiopia that focuses on the evaluation of selected teacher characteristics, the new measure enables students to evaluate teaching in terms of their own learning progress.

Content Type Journal ArticlePages 1-13DOI 10.1007/s11092-011-9140-4Authors Zenawi Zerihun, Mekelle University, Mekelle, EthiopiaJos Beishuizen, Vrije Universiteit Amsterdam, Amsterdam, the NetherlandsWillem Van Os, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands Educational Assessment, Evaluation and AccountabilityOnline ISSN 1874-8600Print ISSN 1874-8597
(show less)

The present study is an attempt to empirically test and compare the results of three methods of word association (WA) analysis. Two of the methods – namely, associative commonality and nativelikeness, and lexico-syntactic patterns of associative organization – have been traditionally used in both first language (L1) and second language (L2) associative research and the third one – collocational aspect of associative responses – is a more recent perspective on associative connections. The central assumption behind the study is that each method captures a different aspect of language users’ lexical organization and, in that, their findings may paint a different picture of how the L1 and L2 lexicons compare as proficiency increases. At the same time, the sensitivity of each method to reflect differences related to proficiency may also point to their potential as a research and assessment tool.

Three equal size groups of participants (N = 180) at different proficiency levels – native speakers (NSs), second language (L2) advanced and intermediate learners of English – completed a familiarity and a WA test in writing. The tests contained 36 items equated for lexical class (nouns, verbs, and adjectives) and frequency of occurrence (high, mid, and low). The participants’ WAs generated to familiar vocabulary were analysed in three different ways, following the methods of analysis practiced by each of the three traditions under investigation. In the main, the results showed that examining the lexico-syntactic patterns of associative organization as well as the collocational aspect of associative links are two more ‘promising’ ways of looking at WAs than examining them with respect to their nativelike associative commonality.

(show less)

Abstract Using Singapore as an example, we argue that schools need to equip and encourage teachers to adopt authentic assessment in teaching and learning so as to develop the students? higher-order thinking. The importance of teaching and assessing higher-order thinking in Singapore classrooms is encapsulated in the vision of ?Thinking Schools? launched by the Ministry of Education in 1997. Underpinning this vision is a shift from conventional assessment to authentic assessment. Unlike conventional paper-and-pencil tests that focus on knowledge reproduction and low-level cognitive processing skills in artificial, contrived contexts, authentic assessment tasks underscore knowledge construction, complex thinking, elaborated communication, collaboration and problem solving in authentic contexts. However, the creation of thinking schools in Singapore remains a constant challenge as many teachers tend to rely on conventional assessment and are often ill-prepared to implement authentic assessment. By presenting the findings from a recent empirical study, we propose that schools build teacher capacity by providing ongoing and sustained professional development on authentic assessment for teachers.

Content Type Journal ArticlePages 1-15DOI 10.1007/s11092-011-9138-yAuthors Kim H. Koh, Policy and Leadership Studies, National Institute of Education, Nanyang Technological University, 1 Nanyang Walk, Singapore, 637616 SingaporeCharlene Tan, Policy and Leadership Studies, National Institute of Education, Nanyang Technological University, 1 Nanyang Walk, Singapore, 637616 SingaporePak Tee Ng, Policy and Leadership Studies, National Institute of Education, Nanyang Technological University, 1 Nanyang Walk, Singapore, 637616 Singapore Educational Assessment, Evaluation and AccountabilityOnline ISSN 1874-8600Print ISSN 1874-8597
(show less)

Publication year: 2011
Source: Journal of English for Academic Purposes, Available online 31 December 2011
James�Donohue
On film studies courses, students are asked to treat as objects of study the same films which they may more commonly experience as entertainment. To explore the role of academic writing in this, an action research project was carried out on a university film studies course using a systemic functional linguistics approach. This paper presents a key assessment essay genre, referred to as ataxonomic film analysis. This genre was analysed drawing on the work ofHalliday and Mathiessen (2004), Martin (1992)and Lemke (1993), focussing on three aspects: the genre acts performed in the process of analysing film; the conceptual frameworks of film studies knowledge, or ?thematic formations? (Lemke, 1985, 1990) drawn on and re-constituted in the assignment; the particular ways that language is used to perform these acts and build these thematic formations. For EAP to be relevant to film students, it is proposed that EAP specialists need to engage with these three aspects of film study. This application of SFL in film studies EAP is intended as an illustration of how SFL tools can be used for relevant EAP provision across the HE curriculum.
Highlights? The key genre identified was taxonomic film analysis. ? Deploying film studies language and genre converts film into an object of study. ? EAP lecturers need to engage with both the language and the meaning making of film students.
(show less)

A differential item functioning (DIF) detection method for testlet-based data was proposed and evaluated in this study. The proposed DIF model is an extension of a bifactor multidimensional item response theory (MIRT) model for testlets. Unlike traditional item response theory (IRT) DIF models, the proposed model takes testlet effects into account, thus estimating DIF magnitude appropriately when a test is composed of testlets. A fully Bayesian estimation method was adopted for parameter estimation. The recovery of parameters was evaluated for the proposed DIF model. Simulation results revealed that the proposed bifactor MIRT DIF model produced better estimates of DIF magnitude and higher DIF detection rates than the traditional IRT DIF model for all simulation conditions. A real data analysis was also conducted by applying the proposed DIF model to a statewide reading assessment data set.

(show less)