TEDS Data Dictionary

Web Tests: A and B Batteries

The 12 Year test battery included 15 separate tests, some of which were quite lengthy. There were therefore concerns that some families, particularly those with slower dial-up connections, would not complete all of the tests, in which case the sample size would be significantly diminished for tests placed towards the end of the battery. It was therefore decided to split the 15 tests into two batteries, labelled A and B, and to present the two batteries in different orders for different families. The two batteries are described in the table below. Links will take you to detailed descriptions of each of the tests.

During piloting, attempts were made to time each test using both dial-up and broadband internet connections. It was found that some of the cognitive and language tests were significantly slower using dial-up than using broadband; whereas the other tests showed smaller differences in timing between dial-up and broadband. Therefore, it was decided that the cognitive and language tests should be put together in battery B, and that battery B should be presented first for broadband families, and last for dial-up families.

Before web testing started, in the mailing that included the 12 Year paper booklets, families were asked, via the paper consent form (pdf), whether they had broadband, dial-up, or no internet connection at home. Families were then placed into one of two groups:

• The BA group.
  Battery B was presented first, followed by battery A.
  This group included all families known to have broadband connections.
• The AB group.
  Battery A was presented first, followed by battery B.
  This group included all families known to have a dial-up connection, plus those families with no internet connection or unknown internet connection.

In wave 1 of the study, twins were rewarded with a voucher when they finished the first battery. The second battery was then presented as being optional, although completion of the second battery would earn a further voucher. This arrangement was put in place to avoid burdening dial-up families (in the AB group) with the potentially very slow tests in battery B.

Results from wave 1 suggested that a high proportion of participating families did in fact have broadband connections. Therefore, for wave 2, it was decided to reward twins with vouchers only at the end of all 15 tests (both batteries), and not at the end of the first battery. However, the allocation of families to AB and BA groups remained unchanged, as described above. This allocation was maintained in an attempt to ensure that the sample sizes for the A and B batteries would be roughly the same at the end of the study, and also to give families with dial-up connections an easier start to the tests.

The ordering of the tests for each group of families (AB and BA) is shown in this table:

Presentation of Tests on the Web Site

Before the twins could take the tests, each family had to log in and given on line consent (see the login/consent page for details). Having done this, and having clicked on the link for the activities for one of the twins, a menu of activities was displayed on screen. An example of such a menu is reproduced below; this example is from wave 2 (with all 15 tests displayed at once), for a family in the AB group (see above for the differing test orders for the AB and BA groups). Note that the text "Twin 1" and "Twin 2" in the image below would be replaced by actual twin names for each family.

The initial menu screen was changed after wave 1. In wave 1, twins were rewarded on completion of the first battery of tests, so the menu screen initially showed only the tests in the first battery (A or B, as appropriate to the family); then, on completion of the first battery, the tests of the second battery would appear in the menu, with an invitation to take the additional tests for an additional reward voucher. In wave 2, as shown in the image above, all 15 tests were shown in the menu right from the start.

The order in which activities were displayed on the menu was the same order in which twins were required to complete the activities. For example, in the list of activities in the image above, the link to "Eyes" would not work until the twin had first completed the "Introduction". Moving the mouse over the green dot next to each activity name would reveal a message displaying the expected range of times that the activity was expected to take (these time ranges were based on measurements during piloting).

The menu, as shown above, includes 17 items: the "Introduction", followed by the 15 web tests in predetermined order, followed by the "Bonus Game". The "Introduction" was a general tutorial describing how all the tests should be conducted, incorporating an animation with audio instructions. The "Bonus Game" was an entirely optional, short computerised game, which was provided for light relief at the end of the tests.

In addition to the "Introduction", each individual web test starts with its own short tutorial. The tutorials generally start with an introduction to the test (with audio and animations), followed by one or more practice examples for the twins to try out. These practice items gave feedback, for instance pointing out the correct answer if the twin answered wrongly; such feedback was not given in any of the main test items.

Parents were asked to ensure that each twin completed the tests independently. Each twin had their own independent menu of activities (as shown above). Although twins generally had to take turns to complete their activities on the home computer, they were able to do this in a flexible way, so that one twin's progress through the tests was not constrained by the other twin's progress.

On completion of the necessary tests, each twin was presented with a screen congratulating them and informing them that a reward voucher would soon be on its way. In wave 1, this would appear twice, at the end of each battery (A and B); in wave 2, this would only appear once, on completion of the last of the 15 tests.

General Test Rules

Each web test had its own set of specific rules, for example for timing, scoring, branching and discontinuing: these test rules are described in detail in the pages for each web test (see links above). However, there were several general test rules that affected all of the tests. These rules were programmed into the web tests, and were enforced automatically. The rules were as follows:

• One attempt at each test.
  Each of the fifteen web tests (not including the Introduction) could be attempted once and only once by each twin. There was no mechanism for re-setting tests, even if parents were aware of problems and asked for tests to be repeated.
• One attempt at each item.
  Every item in each of the fifteen web tests could be attempted once and only once by each twin. If a twin attempted to repeat an item, for example by clicking the "Back" button in the browser, then the test would be stopped and the twin would be presented with a message explaining the problem. The twin could then resume the test as usual from the menu of activities.
• Come Back Later.
  In each item of every web test, there was a "Come Back Later" button. Clicking on this button would halt the web test. The twin could then resume the test at any later time, by selecting the test from the menu of activities in the usual way. The twin would then continue at the same item in which they clicked "Come Back Later". This facility was provided to allow for interruptions to the twins during tests, and to make the tests as flexible as possible in terms of time (for example, a twin could complete a test in one go, or in several sessions spread over different days).
• Timing out during test items.
  Some of the web tests had their own specific timing rules; however, those tests without specific timing rules had a built-in timeout of 5 minutes for each item. If a twin did not respond (with mouse clicks) during a test item, audible prompts were played after 1.5 minutes, 3 minutes and 4.5 minutes to encourage the twin to complete the item. After 5 minutes on screen with no response from the twin, the item would end and the item score would be forfeited (zero score). The item would then be replaced on screen with a "next item" button. The twin could then resume the test at the next item by clicking on this. If the "next item" prompt is ignored (for example, if the twin has left the computer for an extended period), then the more general timeout would eventually be reached (see below), and the twin would be logged out.
• Timing out generally.
  The web site as a whole has a built-in session timeout of 15 minutes. This means that after 15 minutes with no activity (no clicking on the screen), the session ends and twins or parents are logged out. They are then taken back to the login page. During the tests themselves, shorter item timeouts apply (as described above). The longer 15 minute timeout would apply at any other time after logging in, for example on the parental consent form, the twin menu of activities, or the "next item" prompt. The timeout is a routine security measure, to protect logged in sessions.
• No feedback.
  During the tests, no feedback was provided to tell twins whether they had answered each item correctly or incorrectly, or to give twins any idea of their overall score in a test. A twin's route through a test, determined by the discontinue rules, was always hidden from the twin. The no-feedback rule was partly a means of preventing "cheating" or collaboration between twins; it was also intended to avoid discouraging twins after incorrect answers.

Web Callers

All families in wave 1, and many of the families in wave 2, were assigned to web callers. (The allocation of web callers to families is described in more detail in the main 12 Year page.) Web callers were staff employed by TEDS to telephone the families during the web study. The callers fulfilled various functions:

• Collection of verbal consents (or opt-outs) for the web study.
• Encouraging families to complete the web tests, by talking to parents or to the twins themselves.
• Offering basic technical support to the families, to help with completion of the web tests (for example, help with use of a browser, or identifying whether or not a family's computer met the necessary requirements).
• Collection of feedback from the families. This was not done systematically, but any feedback offered by families was fed back to TEDS by callers.
• Collection of parental consent (or refusal) to contact the twins' teachers, with teacher contact details, where this had not been returned previously.
• Collection of any other relevant family details communicated by the families, for example changes of address, or twin medical conditions.
• In wave 1, carrying out the TOWRE telephone test with each twin (in wave 2, this was done separately by a TOWRE caller).

Callers were required to keep track of their allocated families using specially-designed pages in the TEDS web site. Each web caller was issued with their own login (username and password) for the web site. Once logged in, web callers could view detailed information describing the web activity of families allocated to them, for example how far each twin had got in each web test, and the dates and times when test items had been attempted. Other web pages were provided for web callers to log details of telephone calls made to families.

Other information collected from families by web callers, such as opt-outs, teacher details, and changes of address, were communicated to the TEDS office by telephone, email or post. Web callers were encouraged to refer back to the TEDS office if unable to help families themselves with particular problems.