Language Testing Validity

With over 30 years in the language services business, ALTA has built a reputation as a trusted provider of valid and reliable language tests. Our clients depend on us to help them create defensible assessment programs whether through the use of our standard language tests, or through the customization of tests specifically for their companies, organizations, or agencies.

What are Validity and Reliability?

Validity and reliability are the two most important aspects of any testing program, because without them, the tests are meaningless. Validity refers to the ability of the test to measure what it purports to measure. Reliability is the ability of the test to be repeated and yield consistent results. For instance, imagine that a driving test asked you about banking and finance terms. Would that test be valid? Similarly, think about stepping on a scale and getting a different reading every time. Would that scale be reliable? In both cases, the results would be useless since they cannot be interpreted in a meaningful way. ALTA’s job is to create valid and reliable tests that deliver meaningful results so that you can defensibly make decisions about your candidates.

Job Analysis

ALTA’s Job Analysis is the first step for many companies in determining which type of test is right for them, and the cut score on ALTA’s scale to use to determine who is selected for the job and who is not. The job analysis consists of an in-depth review of documentation and interviews with employees who perform and supervise the position, i.e., your subject matter experts. Using this two-pronged approach, ALTA can identify the tasks in which the language is used and the level of proficiency needed to perform those tasks. Following the interviews and document review process, we provide you with a report detailing our findings and recommendations. This gives you a solid foundation for deciding how you use the test results, and provides the defensibility you need should these decisions ever be challenged.

Standard and Customized Assessments

In most cases, our clients use ALTA’s standard language tests. These exams measure general proficiency in the target language and only assume that candidates will be able to integrate any job-specific terminology into their communication. However, some clients want to verify that their employees can converse or write about topics that arise in their business situations, and can understand their unique terminology. In these cases, ALTA will provide customized tests. Customization may include modifying a standard test to include prompts that mirror the types of scenarios an employee may encounter on the job, or may simply verify knowledge of terminology in English and the target language.

How Valid and Reliable Tests are Created

When designing and creating a test, many steps go into making sure that it is valid (i.e., measures what it purports to measure) and reliable (i.e., it can be repeated and will yield consistent results). In total, there are 9 major steps:

1. Identify the knowledge, skills and abilities (KSAs) required to do the job. To identify what is required to successfully perform the job, ALTA engages your subject matter experts who have in-depth knowledge of the position in a job analysis.
2. Create the Test Blueprint: This document serves as the foundation for creating the test. It describes the content that must be included in the test based on the KSAs identified in Step 1.
3. Create Test Items: Developers follow the specifications to create the test questions.
4. Review Test Items: Preliminary test items are reviewed and any changes that need to be made are noted. This process continues until a final set of questions is complete.
5. Pilot the Test Items: Using the final test version, the questions are tried out on a group of test-takers to ensure they perform properly.
6. Create the Final Test Form: Based on the results of the pilot, an operational test form is created.
7. Set the Cut Score: This step involves assembling a panel to make judgments regarding the number of questions the minimally qualified candidate would be able to answer correctly.
8. Administer the Test: Using the operational version, the test is administered to the test-takers and the cut-score is employed to determine qualified versus unqualified candidates.
9. Quality Assurance: Quality assurance is the responsibility of the test administrator, and is an ongoing step. The administrator ensures that items continue to perform properly, and monitor the test for overexposure.

It is important to note that many clients will use the standard tests, which have already undergone this process of validation, and simply choose to employ ALTA to set their cut-score through the Job Analysis and administer the test. This is because the definition of a minimally qualified candidate can change from job to job, or from one organization to another.

Quality Assurance

No matter what kind of testing is used, ALTA pays critical attention to the quality assurance of its tests. One of the ways ALTA ensures reliability of its language test results is through scrutinizing our inter-rater reliability. This is the degree to which two independent raters would provide the same score on the same exam. To collect and monitor this data, ALTA routinely selects recorded assessments at random and distributes them to a second evaluator for scoring. This score is compared with the original evaluator’s score, and any discrepancies are resolved. This process also helps identify any opportunities for retraining or any revisions to the scoring criteria that need to be made. We are proud to say that we consistently maintain inter-rater reliability at exceptional levels.