Table of contents

• Introduction
• The benchmarks function and its arguments
• Computing benchmarks using the command line
• Computing benchmarks using the GUI

 

 

 

Introduction

The lsa.bench function computes percentages of respondents in the population at or above certain performance levels, cut-points in the distribution of test scores (benchmarks/performance levels). The benchmarks can be defined only for plausible values. The percentages of respondents in the population can be computed within groups defined by the categories of splitting variables. The splitting variables are optional. If no splitting variables are provided, the results will be computed on country level only. If splitting variables are provided, the data within each country will be split into groups by all splitting variables and the percentages of respondents in the population performing at or above certain benchmark will be computed for the last splitting variable. Note that the benchmarks can be computed only for a single set of PVs at a time, taking into account the complex sampling and assessment design of the study of interest. The percentages of respondents will be computed for each PV in a set, and then the estimates for all PVs in the set will be averaged and their standard error computed using complex formulas which will depend on the study of interest. The standard error will be computed taking into account the complex sample and assessment designs of the studies. Refer here for a short overview on the complex sampling and assessment designs of large-scale assessments and surveys. If interested in more in-depth details on the complex sampling and assessment designs of a particular study and how estimates and their standard errors are computed, refer to its technical documentation and user guide.

Like any other function in the RALSA package, the lsa.bench function can recognize the study data and apply the correct estimation techniques given the study sampling and assessment design implementation without extra care.

The benchmarks function and its arguments

The lsa.bench function has the following arguments:

• data.file – The file containing lsa.data object. Either this or data.object shall be specified, but not both.
• data.object – The object in the memory containing lsa.data object. Either this or data.file shall be specified, but not both.
• split.vars – Categorical variable(s) to split the results by. If no split variables are provided, the results will be for the overall countries’ populations. If one or more variables are provided, the results will be split by all but the last variable and the percentages of respondents will be computed by the unique values of the last splitting variable.
• PV.root.bench – The root name(s) for the set(s) of plausible values which will be used to compute the percentages of respondents reaching or surpassing certain cut-off score.
• bench.vals – A vector of integers representing the cut-off scores.
• bench.type – A character string representing how the percentages of respondents shall be computed.
• pcts.within – Logical value specifying if the percentages shall be computed within the groups defined by the split.vars (TRUE) or not (FALSE, default).
• bckg.var – Name of continuous background or contextual variable to compute the mean for. The results will be computed by all groups specified by the splitting variables and per performance group.
• weight.var – The name of the variable containing the weights. If no name of a weight variable is provide, the function will automatically select the default weight variable for the provided data, depending on the respondent type.
• include.missing – Logical, shall the missing values of the splitting variables be included as categories to split by and all statistics produced for them? The default (FALSE) takes all cases on the splitting variables without missing values before computing any statistics.
• shortcut – Logical, shall the “shortcut” method for IEA TIMSS, TIMSS Advanced, TIMSS Numeracy, eTIMSS, PIRLS, ePIRLS, PIRLS Literacy and RLII be applied? The default (FALSE) applies the “full” design when computing the variance components and the standard errors of the estimates.
• graphs – Logical, shall graphs be produced? Default is FALSE.
• perc.x.label – String, custom label for the horizontal axis in percentage graphs. Ignored if graphs = FALSE.
• perc.y.label – String, custom label for the vertical axis in percentage graphs. Ignored if graphs = FALSE.
• mean.x.label – List of strings, custom labels for the horizontal axis in means’ graphs. Ignored if bckg.var is omitted and/or graphs = FALSE.
• mean.y.label – List of strings, custom labels for the vertical axis in means’ graphs. Ignored if bckg.var is omitted and/or graphs = FALSE.
• save.output – Logical, shall the output be saved in MS Excel file (default) or not (printed to the console or assigned to an object).
• output.file – Full path to the output file including the file name. If omitted, a file with a default file name “Analysis.xlsx” will be written to the working directory (getwd()).
• open.output – Logical, shall the output be open after it has been written? The default (TRUE) opens the output in the default spreadsheet program installed on the computer.

Notes:

1. Either data.file or data.object shall be provided as source of data. If both of them are provided, the function will stop with an error message.
2. The function computes percentages of respondents which reach or surpass certain cut-off scores (benchmarks/performance levels). These percentages are computed using a set of PVs, specified in the PV.root.bench. Only one set of PVs can be added to PV.root.bench at a time. All studies (except CivED, TEDS-M, SITES, TALIS and TALIS Starting Strong Survey) have a set of PVs per content domain (e.g. in TIMSS five for overall mathematics, five for algebra, five for geometry, etc.) and cognitive domain (i.e. knowing, applying and reasoning). In some studies (say TIMSS and PIRLS) the names of the PVs in a set always start with character string and end with sequential number of the PV. For example, the names of the set of PVs for overall mathematics in TIMSS are BSMMAT01, BSMMAT02, BSMMAT03, BSMMAT04 and BSMMAT05. The root of the PVs for this set to be added to PV.root.avg will be “BSMMAT”. The function will automatically find all the variables in this set of PVs and include them in the analysis. In other studies like OECD PISA and IEA ICCS and ICILS the sequential number of each PV is included in the middle of the name. For example, in ICCS the names of the set of PVs are PV1CIV, PV2CIV, PV3CIV, PV4CIV and PV5CIV. The root PV name has to be specified in PV.root.bench as “PV#CIV”.
3. Multiple splitting variables can be added to the split.vars, the function will compute the percentages of respondents reaching or surpassing the cut-off scores for all formed groups and their means on the continuous variables. If no splitting variables are added, the results will be only by country.
4. If a continuous contextual/background variable is provided to the bckg.var, the average for that variable will be computed for each group formed by the splitting variables and the performance groups. Only one contextual/background variable can be added in the analysis. This argument is ignored when bench.type = "cumulative".
5. The cut-off scores are provided as vector of integers (e.g. c(475, 500)) to the bench.vals. If no cut-off scores are provided, the function will automatically choose all benchmark values for the corresponding study and, in some cases for the data from the specific cycle. The latter applies to ICCS and PISA where the proficiency levels differ from one cycle to another.
6. The bench.type argument has two different options: "discrete" (default) and "cumulative". Using the former will compute the percentages of respondents within the boundaries specified by the cut-off scores in bench.vals. Using the latter, the function will compute the percentages of respondents at or above the cut-off points in the bench.vals.
7. If the pcts.within if FALSE (default), the function will compute the percentages of respondents reaching or surpassing each of the cut-off scores defined by bench.vals. In this case the percentages of all respondents across the performance levels will add to 100 in each group defined by the splitting variables. On the contrary, if pcts.within = TRUE, the function will compute the percentages of respondents with given characteristic at each of the performance levels. For example, if we want to know what percentages of respondents who have or do not have a computer at home at each of the performance levels. The argument is ignored if bench.type = "cumulative".
8. If no variables are specified for bckg.vars, the output will contain only percentages of cases in groups specified by the splitting variables and the cut-off scores.
9. If include.missing = FALSE (default), all cases with missing values on the splitting variables will be removed and only cases with valid values will be retained in the statistics. Note that the data from the studies can be exported in two different ways: (1) setting all user-defined missing values to NA; and (2) importing all user-defined missing values as valid ones and adding their codes in an additional attribute to each variable. If the include.missing is set to FALSE (default) and the data used is exported using option (2), the output will remove all values from the variable matching the values in its missings attribute. Otherwise, it will include them as valid values and compute statistics for them.
10. The shortcut argument is valid only for TIMSS, TIMSS Advanced, TIMSS Numeracy, PIRLS, ePIRLS, PIRLS Literacy and RLII. Previously, in computing the standard errors, these studies were using 75 replicates because one of the schools in the 75 JK zones had its weights doubled and the other one has been taken out. Since TIMSS 2015 and PIRLS 2016 the studies use 150 replicates and in each JK zone once a school has its weights doubled and once taken out, i.e. the computations are done twice for each zone. For more details see Foy & LaRoche (2016) and Foy & LaRoche (2017). For more details see the technical documentation and user guides for TIMSS 2015, and PIRLS 2016. If replication of the tables and figures is needed, the shortcut argument has to be changed to TRUE.
11. If graphs = TRUE, the function will produce graphs, bar plots for the percentages and error bars for the means (if bckg.var is provided) per group defined by the split.vars. All plots are produced per country. If no split.vars at the end there will be a percentages and error bar plots for all countries together. If needed, custom horizontal and vertical axis labels for the bar plots and error bar plots can be defined using the perc.x.label, perc.y.label, mean.x.label and mean.y.label arguments.
12. Unless explicitly adding save.output = FALSE, the output will be written to MS Excel on the disk. Otherwise, the output will be printed to the console.
13. If no output file is specified, then the output will be saved with “Analysis.xlsx” file name under the working directory (can be obtained with getwd()).

If save.output = FALSE, a list containing the estimates and analysis information. If graphs = TRUE, the plots will be added to the list of estimates.

If save.output = TRUE (default), an MS Excel (.xlsx) file (which can be opened in any spreadsheet program), as specified with the full path in the output.file. If the argument is missing, an Excel file with the generic file name “Analysis.xlsx” will be saved in the working directory (getwd()). The workbook contains three spreadsheets. The first one (“Estimates”) contains a table with the results by country and the final part of the table contains averaged results from all countries’ statistics. The following columns can be found in the table, depending on the specification of the analysis:

• <Country ID> – a column containing the names of the countries in the file for which statistics are computed. The exact column header will depend on the country identifier used in the particular study.
• <Split variable 1>, <Split variable 2>… – columns containing the categories by which the statistics were split by. The exact names will depend on the variables in split.vars.
• n_Cases – the number of cases reaching or surpassing each of the benchmarks using a set of PVs. Please note that these may not be whole numbers because they are computed using each PV and then averaged.
• Sum_<Weight variable> – the estimated population number of elements per group after applying the weights. The actual name of the weight variable will depend on the weight variable used in the analysis.
• Sum_<Weight variable>_SE – the standard error of the the estimated population number of elements per group. The actual name of the weight variable will depend on the weight variable used in the analysis.
• Performance_Group – the labels for the performance groups defined by the bench.vals.
• Percentages_<PVs’ root name> – the percentages of respondents (population estimates) reaching or surpassing each cut-off score (in case of bench.type = "discrete") or the the percentages of respondents (population estimates) at or above each cut-off value (in case of bench.type = "cumulative") per groups defined by the splitting variables in split.vars.
• Percentages_<PVs’ root name>_SE – the standard errors of the percentages from above.
• Mean_<Background variable> – the average of the continuous <Background variable> specified in bckg.var.
• Mean_<Background variable>_SE – the standard error of the average of the continuous <Background variable> specified in bckg.var.
• Variance_<Background variable> – the variance for the continuous <Background variable> specified in bckg.var.
• Variance_<Background variable>_SE – the error of the variance for the continuous <Background variable> specified in bckg.var.
• SD_<Background variable> – the standard deviation for the continuous <Background variable> specified in bckg.vars.
• SD_<Background variable>_SE – the error of the standard deviation for the continuous <Background variable> specified in bckg.avg.var.
• Percent_Missings_<Background variable> – the percentage of missing values for the <Background variable> specified in bckg.var.

The second sheet (“Analysis information”) contains some additional information related to the analysis per country in the following columns:

• DATA – used data.file or data.object.
• STUDY – which study the data comes from.
• CYCLE – which cycle of the study the data comes from.
• WEIGHT – which weight variable was used.
• DESIGN – which resampling technique was used (JRR or BRR).
• SHORTCUT – logical, whether the shortcut method was used.
• NREPS – how many replication weights were used.
• ANALYSIS_DATE – on which date the analysis was performed.
• START_TIME – at what time the analysis started.
• END_TIME – at what time the analysis finished.
• DURATION – how long the analysis took in hours, minutes, seconds and milliseconds.

The third sheet (“Calling syntax”) contains the call to the function with values for all parameters as it was executed. This is useful if the analysis needs to be replicated later.

If graphs = TRUE there will be an additional “Graphs” sheet containing all plots.

If any warnings resulting from the computations are issued, these will be included in an additional “Warnings” sheet in the workbook as well.

Computing benchmarks using the command line

In the examples that follow we will merge a new data file (see how to merge files here) with student and school principal data from PIRLS 2016 (Australia and Slovenia), taking all variables from both file types:

lsa.merge.data(inp.folder = "C:/temp",
               file.types = list(acg = NULL, asg = NULL),
               ISO = c("aus", "svn"),
               out.file = "C:/temp/merged/PIRLS_2016_ACG_ASG_merged.RData")

As a start, let’s compute the percentages of female and male students in Australia and Slovenia reaching or surpassing all benchmarks of overall reading:

lsa.bench(data.file = "C:/temp/merged/PIRLS_2016_ACG_ASG_merged.RData",
          split.vars = "ASBG01", PV.root.bench = "ASRREA")

1. There are two variables containing the information on student sex:
   1. ITSEX, a variable containing the student tracking information; and
   2. ASBG01 (used in this analysis), a variable containing the student sex as the students responded in the questionnaire.
2. In international large-scale assessments all analyses must be done separately by country. There is no need, however, to add the country ID variable (IDCNTRY, or CNT in PISA) as a splitting variable. The function will identify it automatically and add it to the the vector of split.vars.
3. The five PVs for the overall reading achievement are ASRREA01, ASRREA02, ASRREA03, ASRREA04, and ASRREA05. In the PV.root.bench argument we need to specify only the root of the PVs, “ASRREA”. The function will use this root/common name to select all five PVs and include them in the computations. For more details on the PV roots (also for the PV roots for studies other than TIMSS and PIRLS and their additions), the computational routines involving PVS, see here.
4. There is no need to specify the benchmarks using the bench.vals argument. If you omit it, the function will automatically add all the benchmarks for the study (and in some studies even for the corresponding cycle), for PIRLS see below, for all other studies refer to the user guides and the technical documentation.
5. There is no need to specify the weight variable explicitly. If no weight variable is specified explicitly, then the default weight (total student weight in this case) will be used for the data set depending on the merged respondents’ data, it is identified automatically. If you have a good reason to change the weight variable, you can do so by adding the weight.var = "SENWGT", for example.
6. If no output file is specified, then the output will be saved with “Analysis.xlsx” file name under the working directory (can be obtained with getwd()).
7. Unless explicitly adding open.output = FALSE, to the calling syntax, the output file will be opened after all computations are finished. This is useful when multiple calling syntaxes for different analyses are executed and no immediate inspection of the output is needed.

The score points for the PVs for each benchmark in PIRLS are presented in the table below.

 

Executing the code from above will print the following output in the RStudio console:

When all operations are finished the output will be written on the disk as MS Excel workbook. If open.output = TRUE (default), the file will be open in the default spreadsheet program (usually MS Excel). Refer to the explanations on the structure of the workbook, its sheets and the columns here.

Let’s compute the percentages of female and male students reaching or surpassing the intermediate and high benchmarks (475 and 550 score points) and for each group compute the average of a variable. We will use the complex scale on how much students like reading (ASBGSLR, check the PIRLS 2016 technical documentation on how this scale is constructed and its properties):

lsa.bench(data.file = "C:/temp/merged/PIRLS_2016_ACG_ASG_merged.RData",
          split.vars = "ASBG01", PV.root.bench = "ASRREA",
          bench.vals = c(475, 550), bckg.var = "ASBGSLR")

The calling syntax from above adds the bench.vals and its value, a vector of benchmarking values to compute the percentages of students reaching or surpassing specific cut-points in the distribution – c(475, 550). It also adds bckg.var and its value, the variable name ASBGSLR. For each groupThe output has similar structure to the previous one, except that this time the columns pertinent to the mean for the scale are added.

Executing the syntax from above will overwrite the previous output because it has the same file name defined (a warning will be displayed in the console). The columns in the “Estimates” sheet will now be different. For the meaning of the column names, refer to the list here.

Computing benchmarks using the GUI

To start the RALSA user interface, execute the following command in RStudio:

ralsaGUI()

For the examples that follow, merge a new file with PIRLS 2016 data for Australia and Slovenia (Slovenia, not Slovakia) taking all student and school principal variables. See how to merge data files here. You can name the merged file PIRLS_2016_ACG_ASG_merged.RData.

When done merging the data, select Analysis types > Benchmarks from the menu on the left. When navigated to the Benchmarks in the GUI, click on the Choose data file button. Navigate to the folder containing the merged PIRLS_2016_ACG_ASG_merged.RData file, select it and click the Select button.

Once the file is loaded, you will see a panel on the left (available variables) and set of panels on the right where variables from the list of available ones can be added. Above the panels you will also see information about the loaded file and radio buttons to select the type of benchmarks from – Discrete or Cumulative. Let’s leave the chosen option to the default, Discrete (refer here for the difference between the two).

Use the mouse to select variables from the list of Available variables and the arrow buttons in the middle of the screen to add them to different fields (or remove them)  to make the settings for the analysis. You can use the filter boxes on the top of the panels to find the needed variables quickly.

As a start, let’s compute the percentages of female and male students in Australia and Slovenia reaching or surpassing all benchmarks of overall reading. Add the variable ASBG01 to the list of Split variables. From the list of Available variables locate the root of the overall reading plausible values (ASRREA). You can use the filter boxes at the top of the panel to search for it, either by name or label. Select the root and add it to the Plausible values panel using the arrow buttons. Note how the PVs are specified. The five PVs for the overall reading achievement are ASRREA01, ASRREA02, ASRREA03, ASRREA04, and ASRREA05. The lists of Available variables and Plausible values will not show the five separate PVs, but just their root/common name – ASRREA, without the numbers at the end. In the background, the function will take all the five PVs and include them in the computations. For more details on the PV roots (also for the PV roots for studies other than TIMSS and PIRLS and their additions), the computational routines involving PVS, see here. This is all that needs to be done. Scroll down and click on the Define output file name. Navigate to the folder C:/temp/Results (or to the folder where you want to save the output) and define the output file name. After you do so, a checkbox will appear next to the Define the output file name. If ticked, the output will open after all computations are finished. Underneath the calling syntax will be displayed. Under all of these the Execute syntax button will be displayed. The final settings in the lower part of the screen should look like this:

Click on the Execute syntax button. The GUI console will appear at the bottom and will log all completed operations:

1. There are two variables containing the information on student sex:
   1. ITSEX, a variable containing the student tracking information; and
   2. ASBG01 (used in this analysis), a variable containing the student sex as the students responded in the questionnaire.
2. In international large-scale assessments all analyses must be done separately by country. The country ID variable (IDCNTRY, or CNT in PISA) is always selected as the first splitting variable and cannot be removed from the Split variables panel.
3. The five PVs for the overall reading achievement are ASRREA01, ASRREA02, ASRREA03, ASRREA04, and ASRREA05 will be taken automatically, all we needed to do is to add the root of the PVs (ASRREA) in the Plausible values list. For more details on the PV roots (also for the PV roots for studies other than TIMSS and PIRLS and their additions), the computational routines involving PVS, see here.
4. The values for the benchmarks are added to the Achievement benchmarks text box. All the benchmarks for the study (and in some studies even for the corresponding cycle) will be added autumatically. For PIRLS see below, for all other studies refer to the user guides and the technical documentation. You can edit them, just make sure they are separated by spaces. If you want to restore the default values, just click on the Reset button.
5. The default weight variable is selected and added automatically in the Weight variable panel. It can be changed with another weight variable available in the data set. If the default weight variable is selected, it will not be shown in the syntax window. If no weight variable is selected in the Weight variable panel, the default one will be used automatically.
6. If the Compute percentages within benchmarks check box is ticked, then percentages of respondents reaching or surpassing each of the cut-off scores defined by the benchmark values will be computed. See here for more details.
7. The Use shortcut method for computing SE checkbox is not ticked by default. This will make the function to compute the standard error using the “full” method for the sampling variance component. For more details see here and here.

The score points for the PVs for each benchmark in PIRLS are presented in the table below.

 

If the Open the output when done checkbox is ticked, the output will open automatically in the default spreadsheet program (usually MS Excel) when all computations are completed. Refer to the explanations on the structure of the workbook, its sheets and the columns here.

Let’s compute the percentages of female and male students reaching or surpassing the intermediate and high benchmarks (475 and 550 score points) and for each group compute the average of a variable. We will use the complex scale on how much students like reading (ASBGSLR, check the PIRLS 2016 technical documentation on how this scale is constructed and its properties). Having made the settings for the previous analysis, select the variable ASBGSLR from the list of Available variables and use the arrow buttons to add it in the panel of Background continuous variables. Now the settings shall involve IDCNTRY and ASBG01 as Split variables and ASBGSLR as Background continuous variables:

Because we use the application with Benchmarks directly after performing the previous analysis, we still have the rest of the settings from the previous analysis done. The only exception is to change the default Achievement benchmarks values box. Edit the values and change them to 475 and 550:

There is no need to change any of the remaining settings, unless you want to. You could, though, change the output file name, otherwise it will be overwritten. Note that the displayed syntax will change, reflecting the inclusion of the ASBGSLR as a background continuous variable to compute the mean for:

Press the Execute syntax button. The GUI console will update, logging all completed operations:

If the Open the output when done checkbox is ticked, the output will open automatically in the default spreadsheet program (usually MS Excel) when all computations are completed. As with the previous analyses, refer to the explanations on the structure of the workbook, its sheets and the columns here.