What is RALSA?

To understand what RALSA is, we need to know what are the large-scale assessments and surveys. If you are already familiar with the methodology of large-scale assessment, please feel free to scroll down to the bottom of this page or just click here. The data from small-scale research is usually analyzed using plain statistical procedures, as we know them from basic statistics. Say, we want to compute average, correlation, or linear regression, we can simply load the data in a statistical software like SPSS, R, SAS, Minitab, etc. and use the built-in procedures. Large-scale assessments and surveys do not aim to compute sample statistics, but to estimate population parameters. That is, we would not care to compute statistic for the used sample, but to use the sampled elements to make an inference for the entire population. While we can hardly collect data for the entire target population in a country, we can draw a sample and use its data to estimate the population parameter. The usual statistical procedures would require drawing a simple or systematic random sample from the target population. The empirical studies from the past show that with a sample of about 400 elements we can reliably represent the target population. We can even assign weight to each sample element, so that it will not represent just itself, but also all similar elements in the population. This procedure, however, has drawbacks.

First, the random sampling will assume that all elements have the same probability of being selected. This may not always be true, the actual selection probability may depend on many different factors, including but not limited to, groups within the population with similar characteristics, or remote groups within the population, as well as the sizes of these groups.

Second, the sampling weights are computed as the inverse probability of being selected. However, when the selection probability is assumed to be equal, the weights for all elements will be equal too.

Third, this procedure would not allow us to oversample certain groups within the target population we may have a special interest in.

Fourth, it will require an exhaustive and precise list of all elements in the population. Such rarely exist.

Fifth, this may not be a practical and cost-efficient option. The elements in a target population which a survey or assessment is interested in usually represent a large number and are scattered across the entire country. If we sample 400 elements from a population of, say 4,000,000 people, we may easily end up in a large expenditure of resources to reach and organize a one-to-one session with each one of them to ask our questions. Such procedure would be quite time consuming as well. It is cheaper to travel to 10 schools and test 40 students in each than travel to 400 schools.

Sixth, surveys and assessments are not interested in just the characteristics of the target population, but also in their context. Large-scale assessments and surveys are often interested in educational outcomes (i.e. student achievement) which are connected with the educational context. These would include school, teacher, family background, as well as processes and educational practices. Thus, it would be highly desirable to link the student achievement to their educational context. With simple or systematic random sample we can easily end up with one sampled student per classroom and school. This would not make possible linking the outcome with the context (school and teacher) to make valid estimation of the relationship between achievement and important contextual factors.

Another complication is introduced when assessment component is present, i.e. in case of large-scale assessments like TIMSS, PIRLS and PISA. The usual testing situation involves applying a test to the respondents and, using the means of Classical Test Theory (CTT) or the Item Response Theory (IRT), a single test score to each tested person is assigned. This approach is commonly used in exams with a specific purpose and limited set of items. Large-scale assessments assess broad content and cognitive domains (e.g. mathematics and science) and a large number of items is needed to assess the outcomes reliably. For example, TIMSS 2015 mathematics domain in grade 8 employs 297 items in mathematics and 305 items in science. It is quite clear that no single student can provide a valid answer to all of them. Thus, large-scale assessments employ something called Block Incomplete Booklets (BIB) approach where small blocks of items are distributed across number of booklets and through common blocks link between the booklets is maintained. Every tested student takes one booklet only with a limited number of items. This is called a “complex assessment design”. IRT is very useful for scaling data from BIB design through the colloquially called “vertical scaling”. Various estimation procedures can produce reliable scores. There is just one issue with these procedures – they are appropriate to produce scores which will be used on individual level, e.g. in exams. However, when it comes to estimation on group level, the scores produced using this method are proven to be biased. The solution is the so called “plausible values methodology” where, simply put, the answers of items which a student did not take are treated as missing values. The plausible values (PVs) methodology is rather complex to be presented in its entirety here, so its presentation here will be quite short. It is extension of the IRT where first the item parameters are estimated. Then the achievement items’ and the principal components derived from the background/contextual data which explain 90% of its variance are pooled together and the scores are estimated applying the obtained IRT parameters. Due to the use of background data, there are many groups where students cluster together because of similarity in their characteristics. Each of these groups has its own distribution of values. There is a lot of uncertainty with this distribution, thou. The process ends with producing individual scores making several (five or 10) random draws in the distribution of the group to which the student belongs. That is, more than one score per tested student is produced to later estimate the measurement variance. It is also called “imputation variance” because actually the estimation of the scores using the PV methodology follows the multiple imputation techniques. As a consequence, each analysis involving test scores has to be repeated a number of times (as many PVs) and then averaged. The computation of the measurement error involves a complex formula following the so called “Rubin’s rule” for aggregating the results from multiple imputed data sets. The actual formula will differ across the different studies, depending on the actual implementation.

When both complex sampling and assessment designs are used, an estimate will be computed with each replicate (JRR or BRR) weight and each PV. To give you an idea what this means, in TIMSS the number of computations for an estimate (percentage, mean, regression coefficient, etc.) is done 755 times (once with the full weight, 150 replicates and five PVs – 151×5=755), then the estimates are averaged and the total error (sampling and measurement) is computed. If the estimate is computed within different groups in the population, then this procedure is repeated for each group separately.

Those who are interested in the details on the methodology of large-scale assessments and surveys this can have a look at the following publications:

• Foy, P., & Yin, L. (2017). Scaling the PIRLS 2016 Achievement Data. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and Procedures in PIRLS 2016 (p. 12.1-12.38). TIMSS & PIRLS International Study Center.
• Gebhardt, E., & Schulz, W. (2018). Scaling Procedures for ICCS Test Items. In W. Schulz, B. Losito, R. Carstens, & J. Fraillon (Eds.), ICCS 2016 Technical Report (pp. 117–138). International Association for the Evaluation of Educational Achievement.
• Meinck, S. (2015). Sampling Design and Implementation. In J. Fraillon, W. Schulz, T. Friedman, J. Ainley, & E. Gebhardt (Eds.), ICILS 2013 Technical Report (pp. 65–86). International Association for the Evaluation of Educational Achievement.
• Rust, K. (2014). Sampling, Weighting, and Variance Estimation in International Large-Scale Assessments. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of International Large-Scale Assessments: Background, Technical Issues, and Methods of Data Analysis (pp. 117–154). CRC Press.

For an overview of the issues in using large-scale assessment data see the following publication:

• Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International Large-Scale Assessment Data: Issues in Secondary Analysis and Reporting. Educational Researcher, 39(2), 142–151.

For additional information on large-scale assessments, visit the ILSA-Gateway.

So, what is RALSA?

From the presentation above it became quite clear that the complex sampling and assessment designs of a study require special care and software to analyze the data. This software needs to handle these issues depending on the actual implementation of the methodology properly. And this is exactly where RALSA comes in.

The R Analyzer for Large-Scale Assessments (RALSA) is an R package for preparation and analysis of data from large-scale assessments and surveys which use complex sampling and assessment design. The software can handle the design issues and apply the appropriate analytical methods automatically for every type of study, out of the box without user intervention. RALSA is a free of charge and open source software licensed under GPL v2.0 and is cross-platform — it works on any system which can run a full installation of R. Currently, RALSA supports a number of studies with different design and a number of analysis types (see below). Both of these will increase in future.

In addition to the traditional command-line R interface, RALSA has a Graphical User Interface for the users who lack the technical skills.

Currently, RALSA supports the following functionality:

• Prepare data for analysis
• Convert data (SPSS, or text in case of PISA prior 2015)
• Merge study data files from different countries and/or respondents
• View variable properties (name, class, variable label, response categories/unique values, user-defined missing values)
• Data diagnostic tables (quick weighted or unweighted frequencies and descriptives for inspecting the data and elaborate hypotheses)
• Recode variables
• Perform analyses (more analysis types will be added in future)
• Percentages of respondents in groups and their averages on variables of interest
• Percentiles of continuous variables within groups of respondents
• Percentages of respondents reaching or surpassing benchmarks of achievement
• Crosstabulations with Rao-Scott first- and second-order chi-square adjustments
• Correlations (Pearson or Spearman)
• Linear regression
• Binary logistic regression

All data preparation and analysis functions automatically recognize the study design and apply the appropriate techniques to handle the complex sampling assessment design issues, while giving freedom to tweak the analysis (e.g. change the default weight, apply the “shortcut” method in TIMSS and PIRLS and so on).

In addition to the tabular outputs, RALSA also has the possibility to produce graphical representation of the results for the percentages and means, benchmarks and percentiles functions.

Currently, RALSA can work with data for all cycles of the following studies (more will be added in future):

• CivED
• ICCS
• ICILS
• RLII
• PIRLS (including PIRLS Literacy and ePIRLS)
• TIMSS (including TIMSS Numeracy and eTIMSS)
• TiPi (TIMSS and PIRLS joint study)