• The complaint: The Community of Madrid asks the OECD to withdraw the entire PISA report for "errors of a considerable caliber": "The whole test is contaminated"
  • The errors: The PISA report freezes the data of Spain on Reading because at least 5% of the students did not respond rigorously
  • The creator.Andreas Schleicher: "The educational laws in Spain are almost nineteenth century"

The results on the academic performance of 600,000 students - 36,000 Spaniards - are presented on Tuesday in 79 countries in full controversy over the suspicions that the Community of Madrid has stoked about the reliability of the results of the almighty PISA Report . The OECD will not present the data obtained by students in Spain in Reading because at least 5% of the students responded "unlikely" in this section. But it will publish those of Mathematics and Science , despite the fact that the Madrid Government denounces that they are also "contaminated."

What happened? How does an autonomous administration to challenge a world giant like the OECD? You no longer have to give credibility to, PISA report? Here are the answers.

Easy questions

The OECD admits "anomalies" in the way Spanish students responded to a new questionnaire that had some fluent reading questions that, if they look good, are quite rare. The 15-year-olds had to answer yes or no to whether they saw meaning in sentences that were the following in the pilot exam. "The planes are made of dogs," "The happy student read the book last night" or "The man who is taller thanks the woman and the boy is shorter than both." Everything indicates that the students could believe that these questions, as easy as they were, were not part of the exam, but were a previous training.

Why were they included? According to the sources consulted, a few years ago the OECD launched the PISA for Development program , adapted to developing countries, but these governments could not pay the required fees and it did not work. This edition has incorporated aspects of this program and a good number of new countries, including very easy questions for students with low levels of reading comprehension. Spanish students have been unable to understand the meaning of this questionnaire and have responded to Tuntun, causing a drop in Reading.

Subcontractors

Madrid has indicated to the company Typsa , subcontracted by the Ministry of Education at the time of the PP, as responsible for applying the tests and coordinating nine examining teams that have moved through several communities. He maintains that three of these teams were able to give confusing instructions to the students, since these examiners have three times as many errors as the others. OECD sources denounce that this agency "is losing control of the evidence" because it delegates the bulk of tasks to subcontracted companies and the group that directly works in this area is "very small".

"There have been problems in other countries with the poor quality of the companies that sample, analyze and filter the data. Sometimes the questions have been mistranslated and sometimes there has been a bad treatment of the data," these sources point out. In Madrid they recognize that the performance of subcontracted teams has been "chaotic".

"Lack of transparency"

On November 15 the OECD reported that it paralyzed the Spanish reading data and, since then, has not given explanations. Nor has the Ministry of Education, headed by Isabel Celaá, who has shrugged and recalled that the tests were done in the spring of 2018, with the PP. " Andreas Schleicher [the OECD Director of Education] has a tight grip on the information and nobody says anything. But there is a lack of transparency. It is essential to know if PISA has failed, because this costs Spain a lot of money. It cannot be that cover errors, "say the sources consulted.

Absent with a zero

70 Madrid students who did not take the exam were given a zero, when they had to be included in the category of not presented. In the OECD they say that "70 students from a sample of 5,000 do not lower the results", but other sources say that this is an "irregularity". The OECD initially said that "no technical error or data manipulation has been appreciated," but it seems that there is. He also said that the results of Mathematics and Science seemed to be "affected" by "this anomalous behavior" detected in Reading, although to a much lesser extent. Now he says that no anomalies have been found in this data and that is why they can be published.

It happens, however, that in PISA the grade of Mathematics and Science of 60% of the students has been calculated from that of Reading, in an extrapolation that is considered by some experts as "a fudge". According to sources in the Community of Madrid, "such a crude method of determining these notes shows the decline of these tests."

Another operation questioned in this edition is the adaptability of a part of the Reading test -not that of reading fluency-, whereby students arrive at different itineraries based on their answers: those who respond correctly at the beginning are going asking increasingly difficult questions, while those who answer poorly have an easier exam. The sources consulted ask how you can compare between countries or regions if there are so many groups of students and so many exam models.

The autonomous communities

In addition to Madrid , there are more affected regions, but the OECD has not said what they are. The Madrid advisor Enrique Ossorio , who demands transparency and explanations because he has paid 300,000 euros for the tests, ensures that there are three other communities affected. Recently, critics have been shown Castilla y León and Castilla-La Mancha , whose president, Emiliano García-Page, has said that PISA's promoters "only come to ask for money." In 2016, Andalusia also questioned the validity of the data.

The 'pisaescepticism'

Madrid has put its finger on the sore and has opened the debate, for the first time in Spain, on the reliability of the results, thus giving legitimacy to the growing movement of pisaesptics who question that, as the test is given too much importance, this It has led to students being trained in some countries or History or Social Sciences have been relegated, because they are not evaluated.

There is, for example, a team of Danish experts that is increasingly critical of this evaluation. What is clear is that PISA does not measure everything and that there is room for improvement. The British consultant Valerie Hannon believes that "the current tests are very narrow and insufficient to evaluate the system" and, above all, meet the new skills demanded in the 21st century. That is why, this year, a conference with new visions of what to learn will be held in Paris , in parallel to the presentation of the results, in which billionaire Richard Branson and Alibaba founder Jack Ma participate. . By 2021, the OECD also wants to evaluate creativity and other skills.

Is it reliable in report?

Experts who know the ins and outs of the test say that, despite everything, it is a solid evaluation. In addition, it is the only one of this type that we have that allows us to compare results between autonomous communities. "We can and must continue to rely on PISA because it is one of the most rigorous scientific procedures that exist," says Julio Carabaña , Professor of Sociology at the Complutense University of Madrid and author of the book The Uselessness of PISA for Schools .

"There is no scientific company in the world of education, sociology or economics comparable to PISA, not even Eurostat," he says. He points out that the failures reported by Madrid will not have a statistically significant impact on the whole of Spain.

According to the criteria of The Trust Project

Know more

  • OECD
  • Madrid
  • Education
  • Graphics
  • Isabel Celaá

Education The school of Soria where students work as if they were Harry Potter

Education The Community of Madrid asks the OECD to withdraw the entire PISA report for "errors of a considerable caliber": "The whole test is contaminated"

Education Celaá declares war on the concerted by denying the constitutional right of families to choose a school