ALTA Language Services, Inc. is a recognized leader in language services and language testing in the United States. Currently, we are seeking native to nearly native Iraqi-Arabic linguists, preferably with a strong degree of familiarity with the Interagency Language Roundtable (ILR) scales, to be a part of a two-day teleconference panel determining the passing score for government listening comprehension tests.

Passing scores will be set according to the ILR scale for the 0+ – 2 levels. The teleconference is currently slated to take place in late July and will last two full working days. If you are interested in obtaining more details regarding participation in the teleconference panel, please contact Jocelyn McFarlane at jmcfarlane[at]altalang.com with an attachment of your current curriculum vitae.

You may also reach her at 404.920.3826.

From a young age, our lives are filled with assessments: standardized tests, driving exams, placement tests, and, if a business or government agency tests the English or foreign language skills of potential hires and current staff, there is a good chance that they use ALTA language testing. Creating a valid assessment of an individual’s language skill is an important and difficult task: one that requires us to take many steps to guarantee that every test we administer meets the Standards for Educational and Psychological Testing.

So, what does it mean for a test to be valid?

In psychometric terms, a test’s validity is the degree to which the theory behind the test and the interpretation of the test’s score accurately measure the test’s intended purpose. In other words, a valid language test works to assess language ability, and the scores can be defended.

ALTA spends a lot of time and resources to ensure that our language tests are valid. While the process is complex, here is a basic synopsis of 9 important steps we take, whereby each step contributes to the overall validity of a language assessment. Figure 1 illustrates this validation cycle, and each step is described below:

Figure 1: Validation Cycle

1. Job Analysis/KSAs:

The first step in test development is to identify the knowledge, skills, and abilities (KSAs) that the test will be designed to measure. For tests that are designed to qualify an individual to perform a specific job, these KSAs are identified through the performance of a job study in which individuals knowledgeable of what the job entails – or, subject-matter experts (SMEs) – are interviewed to collect this information. The identification of KSAs is a crucial step in providing focus to the development efforts that follow.

2. Create Test Blueprint:

The test blueprint is created based on the KSAs identified, and their relative importance to the job. The blueprint specifies to the test developer the content that will be included in the test, the amount of content in each skill area, and any other instructions needed to properly develop the content. Using the blueprint as a guide, test developers are engaged to create the actual test items.

3. Create Test Items:

Item development is carried out according to the specifications outlined in the test blueprint. More than the ample amount of test items are created to allow for the possibility that some of the items will need to be eliminated based on pilot-testing and item analysis results.

4. Review Test Items:

All test items are submitted to a separate panel for review and comment. This panel reviews each test item and verifies that each aligns with the specifications as outlined in the test blueprint. Any need for modification is recorded, and comments are provided to the developers so that the appropriate changes can be made. This review process is repeated for any changes that are made until the pilot version of the test is complete.

5. Pilot Test Items:

Once the final draft version has been reviewed and approved by test developers and the review panel, the items are pilot-tested to gather data around item performance. Pilot testing is done using a sample of candidates representative of the target population. Following the pilot-testing, psychometric analysis is performed on the results to determine the test’s performance.

6. Create Final Test Form:

Results from the statistical analysis yield the items that will constitute the final test form, and these items assembled into the operational version of the test.

7. Angoff Method:

Using the final test versions, an Angoff panel is assembled to determine the cut-score of the test, or the percentage of correctly-answered items that the candidate needs to successfully pass the test. Although various standard-setting methods exist, ALTA typically uses the Angoff method, which relies on the judgments of the panel as to the percentage of minimally-qualified candidates who would perform successfully on each item.

8. Administer Test:

Upon determining the cut-scores for the final test versions, the tests are available for operational use and are administered according to the operational policies set up by the test administrator using a prescribed scoring rubric.

9. Quality Assurance:

Quality assurance is performed continuously to ensure that the items are performing properly over time. Quality assurance also provides a method for monitoring overexposure and identifying items which may have been compromised.

It is important to note that validation is a cycle, and testing organization should continue reviewing the test and collecting evidence of the test’s validity. At various points in the lifetime of a test, each step may be revisited for review and/or revision.

____________________________________________________________________________________________

ALTA is a leader in language testing and large-scale language solutions for government agencies and corporations nationwide. In addition to being the official language testing provider for the cities of Los Angeles and New York, ALTA works with many of the country’s largest corporate organizations, from DELTA Airlines to Wells Fargo. Learn more about us at altalang.com

ALTA has completed a comprehensive study of court interpreter exams for the Judicial Council of California. The published report, California’s Assessment of the Consortium for Language Access in the Courts’ Exams, appears on the California Court’s website.

In May of 2009, the Judicial Council of California contracted ALTA to assess the Consortium for Language Access to the Courts’ (CLAC or Consortium) test development process. The aim of the study was to perform a comparative analysis between the examinations developed by CLAC to certify its member state court interpreters and the examinations developed by California to certify California state court interpreters.

The study includes literature and document reviews and interviews with members of the Consortium’s staff both past and present, including language consultants and exam writers. Additionally, ALTA and a team of psychometricians performed a psychometric audit of the Consortium’s test development process to identify its key strengths and any areas that were in need of improvement to meet test industry standards according to the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999.)

Once the psychometric audit was complete, a group of subject-matter experts examined both the written and oral Consortium exams to identify the knowledge, skills, and abilities required for successful court interpreting , as determined in the Study of California’s Court Interpreter Certification and Registration Testing. Results from the comparative analysis showed a high degree of overlap between the California and CLAC written and oral exam KSAs, as well as a strong degree of comparability between the structure, content, and level of difficulty found in the oral exams for each program.

The final step of the study involved performing a standard-setting session applying the Angoff Method to establish the level of functional equivalency between the passing requirements of each program’s oral examinations.
____________________________________________________________________________________________

ALTA is a leader in language testing and large-scale language solutions for government agencies and corporations nationwide. In addition to being the official language testing provider for the cities of Los Angeles and New York, ALTA has worked with the Judicial Council of California on language testing studies for several years. Learn more about us at altalang.com

ALTA Language Services is conducting a pilot study for non-native English speakers. If you have taken the TOEFL or another spoken language test such as the ACTFL OPI, or the DLPT OPI within the last 6 months, please contact us at lhopkins [at] altalang.com.

ALTA will pay you US$50 to complete a test of your English speaking skills and fill out a short survey about your experience taking the test. All tests are conducted by telephone.

When you write to us at lhopkins [at] altalang.com, please provide the date you took the TOEFL or other validated spoken language test, and your score on the speaking portion.

____________________________________________________________________________________________
ALTA Language Services has provided language testing to U.S. corporations and government agencies for 30 years. For more information about us, please visit our website at www.altalang.com.

For Beyond Words readers who know ALTA primarily as a translation company, it may come as a surprise to learn that we are national leaders in the Language Testing sector. Since ALTA does not develop academic language tests (like the TOEFL & SAT, which get so much attention) it’s not surprising that our services might fly under many people’s radar. Our leadership in medical, government, and corporate language testing, however, keeps us busy with interesting projects for clients and partners who you’ve definitely heard of: Kaiser Permanente, the Defense Intelligence Agency, the City of Los Angeles, major US airlines, financial institutions, and non-profit organizations throughout the world.

We test nearly 100 languages, but just as our top translation services provide an interesting glimpse at language trends for business and government, the top languages we test reflect much about current political and economic conditions, and the importance of valid and reliable language skill verification for business, health care, and national security. Here is the data on the top languages we tested in 2009:

Top 10 Languages Tested (2009)

Rounding off the list, the next 10 most-tested languages after Russian are: German, Hindi, Japanese, Korean, Italian, Portuguese, Somali, Turkish, Cantonese, and Sorani.

Other Language Testing Articles

Language Testing and Health Care Reform:
ALTA and Kaiser Permanente Breaking Cultural Barriers

Psychometricians: What They Are, and What They Do

What is the Angoff Method?

Norm Referenced vs Criterion Referenced Testing

Multiple Choice Test Development 101

The Value of Independent Language Testing

What is a Psychometrician?

A psychometrician is someone who practices the science of educational and psychological measurement, or in other words, testing. Psychometricians measure the validity, reliability, and fairness of an exam program, and are an integral part in the process of creating valid and reliable language tests. They may be involved in all aspects of the development of the program, such as defining the knowledge, skills and abilities (KSAs) that the exam is designed to measure; they may also be involved in the development of the test specifications, the item writing process and the standard setting studies. While practicing or overseeing each of these areas, the psychometrician is highly concerned with ensuring that each part of the process conforms to testing industry standards, such as those in the Standards for Educational and Psychological Testing. These standards are considered the primary source for test publishers, test takers, and test users.

How Do I Become a Psychometrician?

There are currently no certifications for becoming a psychometrician; rather, people practicing in the field will often have an advanced degree from an Educational Measurement or Psychology program. Generally, a psychometrician needs skills and experience in quantitative analyses, critical thinking and statistics, as data analyses is an integral part of psychometrics. Many psychometricians go on to work for testing organizations. The Bureau of Labor Statistics estimates that the job outlook for psychometricians will grow faster than the average rate for all occupations, with excellent salary prospects.

A list of over fifty advanced degree programs in the U.S. and Canada that focus on Educational Measurement was compiled by the National Council on Measurement in Education (NCME). It can be found here.

Since earning recognition for Innovation in Multicultural Health Care, Kaiser Permanente’s Clinician Cultural Linguistic Assessment language testing tool, which is administered by ALTA, has been receiving a lot of positive attention from people in the medical community. ALTA receives a mention in the following article from the California Healthfax publication:

Read the rest of this entry »

As our nation’s policy makers and health care providers debate the future of health care reform, ALTA and Kaiser Permanente are working together to improve the quality of care for ethnic minorities and multilingual patients.

Having recently been recognized with an award for Innovation in Multicultural Health Care from The National Committee for Quality Assurance (NCQA), ALTA and Kaiser are pleased that their language testing program is making a positive impact. The award recognizes the Clinician Cultural and Linguistic Assessment for, “effectively and compassionately addressing race, ethnicity, and language proficiency in the delivery of quality, cost-effective services,” (Robert Ross, CEO of The California Endowment).

The Clinician Cultural and Linguistic Assessment (CCLA) is a tool developed by Kaiser Permanente and administered by ALTA to determine the level of target language proficiency of bilingual physicians. The CCLA is designed to assess target language communication (speaking, reading, and listening comprehension) between doctors and patients in a linguistically accurate and culturally sensitive manner without the use of an interpreter. The CCLA is the first and only validated and reliable language assessment tool of its kind, and sets a standard level for effective communication in a medical setting. The results are used to match patients with physicians who are fluent in their language. ALTA, a national leader in language testing, has administered the assessments for Kaiser Permanente since September of 2007.

Recognition of the ALTA-administered language assessment comes as U.S. health care providers increasingly face the challenges of linguistic diversity. A study conducted by The Institute of Medicine recently found that ethnic minorities tend to receive lower quality health care than non-minorities, even when factors such as insurance status and income level are controlled. Kaiser developed its language assessment program to remedy this situation as the number of ethnic minorities in need of health care is likely to increase in the coming decades.

According to NCQA president Margaret O’Kane, “one in five Americans speaks a language other than English at home…it’s imperative that health care reform efforts address the impact of disparities on the quality and cost of U.S. health care.” In addition, a 2005 study conducted by the National Alliance for Hispanic Health found that the number of elderly Spanish speakers in the U.S. will climb to 13 million by 2050, even as the U.S. Census Bureau reports that latinos 65 years and older are projected to outnumber all other elderly minorities by 1 million as soon as 2030. The CCLA is offered in Spanish and 13 other languages.

In the years that I’ve worked at ALTA, I’ve read a lot of literature about test design, validation techniques, and testing standards. Most of this literature comes in the form of dry, statistic-laden academic materials (apologies to all who enjoy statistic-laden academic materials), but one resource in particular stood out as informative as well as entertaining, and really put many aspects of what we deal with as a language company into focus. I refer to the book, The Spirit Catches You and You Fall Down by Anne Fadiman. As the subtitle explains, Fadiman’s wonderful book tells the story of “a Hmong child, her American doctors, and the collision of two cultures.”

I won’t divulge too much because I feel it is a must-read for anyone interested in language and culture, but here are the basics: A child of refugee parents from Laos, Lia Lee, has severe epilepsy and is taken to a small county hospital in California for treatment. Both parties become frustrated when they find that the Western medical practices and Eastern cultural practices don’t mesh. The Hmong parents are deeply connected to their cultural beliefs, which involve spiritual practices to help their daughter, whereas the hospital’s physicians are deeply concerned about Lia receiving the proper dosages of the medications they have provided to calm her seizures. The title of the book underscores the struggle that they both face:

The spirit catches you and you fall down is the closest translation of the word seizure into the Hmong language. How can a physician impress the need to treat a condition upon individuals who don’t believe the body contains organs? How can these individuals impress it upon the physician that they believe their daughter would be better off using healing ceremonies versus watching her suffer the side effects caused by the medication being prescribed?

The reason this book impacted me the way it did is Fadiman’s way of illustrating, not just the occasional difficulty of translating certain words, but that certain concepts are difficult to translate across cultural divides. This is the reason why we use highly skilled translators in our translation division, and why we concern ourselves with cultural equivalence in our language testing. We need to make sure that what we produce not only meets our client’s expectations in terms of the words in black and white, but also that it reflects the layers of meaning across cultures.