Monday September 7 2009
There is no proof yet that brain training works
Newspapers report that research shows a brain training game can “vastly improve children's school grades within a matter of weeks” (Daily Mail). The report is mixed with news that Facebook may improve working memory, but spell checks and Twitter may "work against improving working memory" (The Independent).
Aspects of these reports are based on research presented at the British Science Festival by psychologist Tracy Alloway. The research tested an online game, JungleMemory, available through subscription. The game's website claims it is “scientifically proven to improve IQ, working memory, and grades”.
The available research refers to one small study of 15 children with learning difficulties, which found the game improved measures of crystallised intelligence (skills acquired through knowledge and experience) and school performance. However, this evidence is weak and whether or not it constitutes scientific proof is debatable.
Other news of a study of more than 600 children around the world linking JungleMemory to significant improvements in academic performance do not appear to have been published and there is no further information available about this study at this time. The claims regarding Facebook and Twitter helping or hindering working memory also do not appear to be based on available research.
Where did the story come from?
The news reports are based on research carried out by Dr Tracy Alloway and Ross Alloway from Stirling University and Edinburgh University. Dr Tracy Alloway made a presentation on her work at this year’s British Science Festival. The newspapers appear to refer to two studies by this author: a study of 15 schoolchildren with learning difficulties, and an online study of 600 children worldwide. Both studies reportedly involved a training programme using proprietary software called JungleMemory.
The first study, which this appraisal focuses on, is published on a website hosted by the Nature series of medical journals called Nature Precedings. This is a repository for pre-publication research and preliminary findings where authors can post their findings before formal publication. The website states that: “documents on Nature Precedings are not peer-reviewed and, as such, should not be considered 'published' works”. It is unclear whether the research had any outside funding.
What kind of scientific study was this?
The researchers introduce their study by saying that general intelligence is thought to include aspects of crystallised intelligence (skills acquired through knowledge and experience) and fluid intelligence (problem-solving, pattern-matching and reasoning). They say there is evidence that memory training can improve fluid intelligence in adults, but whether it can improve acquired skills, such as crystallised intelligence and academic attainment, has yet to be established.
This study involved 15 students of about 13 years of age with learning disabilities. The research aimed to test a working memory training program consisting of three games. The first game involved scanning a 4x4 grid in which users had to remember the location of particular targets, initially letters then progressing to word endings. The second game involved interpreting letter rotations. In pictures of a letter facing up or down or a mirror image, participants needed to remember the location of red dots near the letters. The third game involved solving maths problems. There are up to 30 levels in each game and participants had to successfully answer eight out of 10 trials in each level to progress. If the participants struggled with the difficulty of a level, the program automatically changed to an easier one.
Children were randomly allocated to either the working memory-training program (eight participants) or the control group (seven participants). Those in the training group were asked to use the program three times a week. Each child completed an average of 75 trials for all three games over an eight-week period, lasting 30 minutes per session.
Children in the control group had targeted educational support three times a week at school. This lasted for an eight-week period and consisted of about 25 sessions of 30 minutes each.
Both groups were tested on measures of crystallised intelligence, academic attainment and working memory before and after the experiment.
What were the results of the study?
The training group displayed improvements in all cognitive measures, academic attainment and working memory compared with the control group.
What interpretations did the researchers draw from these results?
The researchers conclude that considering how important crystallised intelligence is in acquiring and using knowledge, their findings "may be highly relevant to improving education outcomes in those who are struggling".
What does the NHS Knowledge Service make of this study?
There are several points to make about this study, the claims made by the researchers and how it has been reported in the press.
- Firstly, the study was very small. Only 15 students with learning difficulties participated in the study, eight of whom received the training group.
- It is unclear whether these results apply to other children and those without learning difficulties.
- There was no long-term follow up of these children to see whether the benefits lasted.
- The claims that Facebook may improve crystallised intelligence while Twitter may have the opposite effect were not investigated by this study and do not appear to be based on currently available published research.
- The newspapers mention a study in more than 600 children around the world. The Daily Mail says that children "languishing in the bottom 10-15% of their class" improved to the top 3% through the use of JungleMemory. However, this research does not appear to have been published.
Despite these shortcomings, the findings are interesting for psychologists on a theoretical level because they demonstrate that training may improve working memory. The results from the larger study in 600 children, referred to by the Daily Mail, should confirm this or not.
On balance though, more rigorous research that is subject to peer review through the publication process is needed before the evidence is sufficient to state these types of games have been scientifically proven to “improve IQ, working memory, and grades” as stated by the JungleMemory website.