The #PISA2018 results are out. Generally, countries scored within an expected range given their past records. Except one. The scores are astonishing for B-S-J-G, an acronym for the four Chinese provinces that participated (Beijing, Shanghai, Jiangsu, and Guangdong).
Out of 77 international systems, B-S-J-G scored number one in all three subjects: reading, math, and science. Scores increased dramatically from 2015-2018. In fact, the gains are so large that they are bound to raise eyebrows.
B-S-J-G’s scores increased by 61 scale score points (from 494 to 555) in reading, by 60 points (531 to 591) in math, and by a whopping 72 points (518 to 590) in science. How uncommon are gains like these? I examined PISA data from 2006-2015.
For each three-year test interval, I computed the changes for each country on the three PISA tests and converted them to absolute values. That produced 497 observations, with a mean of 9.5 points and standard deviation of 8.6.
So the typical change in a nation’s scores is about 10 points. B-S-J-G’s 2015-18 gains are at least six times that. The B-S-J-G gains are also at least seven times the SD of all interval changes. Highly unusual.
To appreciate the improbability, consider that only six of the 498 interval changes from 2006-2015 were 40 or more points. B-S-J-G has now added three more, and its changes were 60 points plus.
The other big interval changes look random—different countries, years, or subjects. The B-S-J-G changes were all produced by one country during a single test interval. And they occurred across all three PISA subjects.
I hope @OECDEduSkills has closely scrutinized the 2018 B-S-J-G test scores. They’ve had the raw data for quite a while
The past PISA scores of Chinese provinces have been called into question (by me and others) because of the culling effect of hukou on the population of 15 year olds—and for OECD allowing China to approve which provinces can be tested.
OECD responded to the criticism by attacking critics and conducting data reviews behind closed doors. A cloud hangs over PISA scores from Chinese provinces. I urge OECD to release, as soon as possible, the results of any quality checks of 2018 data that have been conducted.
Three-year PISA gains of sixty scale score points or more are amazing, historical accomplishments that education researchers around the world will want to study for a long time. Let’s make sure that they’re real. @YongZhaoEd @bbceducation @greg_ashman @SarahDSparks
I did not realize when I composed the thread that one of the the four provinces is different in 2018--with Guangdong being dropped for Zhejiang. The acronym for the 2018 participants is B-S-J-Z. So obviously the 2018 data will not be comparable to previous years' PISA scores.
I apologize for the error. None of the statistics reported in the thread regarding the distribution and probability of gains over 60 points are affected. The 2015-18 gains that I reported, however, are comparing 2015 B-S-J-G scores with those of 2018 B-S-J-Z.
Is the change responsible for the huge leap in scores?. Guangdong is China's most populous province. Zhejiang is about half as populous. Scores of the four participating provinces of both 2015 and 2018 need to be disaggregated to assess the impact of the G-Z change.
The G-Z change underscores the questionable role of China in PISA tests. In 2009, the test was given in 12 provinces--many of them rural--but scores were only reported for Shanghai. Scores for the other provinces were never released. B-J & G were added in 2015.
Now Zhejiang is in and Guangdong is out. And the scores from Chinese provinces soared by unprecedented amounts. Allowing China to choose the provinces to assess is unorthodox for a large scale assessment, rendering China's PISA scores incomparable to others.
