Web-Based Russian Test Reflection

I created an online Russian diagnostic test intended for college students who are preparing to study abroad in Russia, hosted here. I chose to write a test for this particular audience because I hope to one day accompany students on semester-long study abroad programs, and therefore anticipate working closely with this population in the future. The purpose of my test is to assess intermediate students’ knowledge of speech acts, pragmatic routines, gestures and cultural schemata, as well as listening comprehension ability for diagnostic purposes. This test originated online, thus my goal for the CALL and Assessment course has been to expand and improve upon the original online test.

I chose to use the ANVILL platform for my test because of its potential to host various multimedia. I knew that I wanted to embed video. ANVILL is an online course management system designed for language educators that allows users to create tests that incorporate images, audio, video, and a variety of item types (multiple choice, scales, short and long answer, spoken response).

I wrote my items within ANVILL and was able to tweak items as needed to fit the platform. Uploading images and audio, embedding video, and formatting text (using the WYSIWYG editor) proved to be simple and straightforward. I appreciated the diversity of items I was able to create: multiple choice discourse completion tasks; image-enhanced multiple choice items; listening comprehension questions with audio and YouTube video prompts; short answer written discourse completion tasks; reading comprehension multiple choice questions based on a text; items that call for a spoken response, with both aural and written prompts; and self-assessment items using a Likert scale. I wrote many of the items myself, using video clips from the MIMEA project and an image from a book on Russian gestures (“A Dictionary of Russian Gesture” by Barbara Monahan) as prompts. The remaining questions that were added to further enhance the test were gathered from various sources:  practice reading, listening, and speaking items from a sample Test of Russian as a Foreign Language (TORFL-2) and a self-assessment battery created by Peter Shaw, David Tsugawa, and Kathleen Bailey contained in “Learning about Language Assessment” by Kathleen Bailey (pp. 231-232). The TORFL contains the written version of the listening and speaking prompts, which I then recorded myself in Audacity and uploaded into ANVILL.

As far as taking the test, one of my goals in this project was to create an assessment that would be fun for test-takers. By keeping prompts authentic, items short and interesting, and focusing on areas not normally tested (e.g., pragmatics), I feel that I was able to accomplish this. Several of my test-takers commented that the test was fun, and as I took the test, I thought about how much I wished that more of the tests I had taken in my career as a learner of Russian were like this one.

I think that making my test computer-based definitely enhanced its attractiveness and practicality. I was able to get a 60% response rate, and half of those respondents completed the entire test. I doubt that I would have been able to get those figures if the test had been paper-based. For whatever reason, completing a test or survey online is easier and more enjoyable, at least for me; there’s less commitment involved. Also, the audio and video prompts are much better suited to online delivery, because test-takers are in control of how often they listen to them, and can adjust their sound settings as needed. As far as measuring the constructs is concerned, I think that my test measured most of the constructs as well as or better than a paper version would have, excluding perhaps the speaking sections. I say this because recording yourself speaking to a computer is different from talking to a live interlocutor; in my assessment, test-takers have unlimited time to prepare a response, and can even rerecord a response as many times as they wish.

I am very proud of the work I have done on this test, and I think it is an excellent starting point for future test development. I would like to create diagnostic tests for needs assessments purposes for classes I teach in the future, incorporating items similar to the ones I used in this online test, particularly the self-assessment questions. I would also like to try different platforms for assessing language that more authentically replicate real-life situations. I think that this kind of authentic assessment is the wave of the future and I hope to implement it in my own teaching.

12/9/09