Assessment · ICT · NAPLAN

Chupa Chups

chupaI recently found myself quoted on the front page of the Weekend Australian.

I’ve written a couple of blogs about NAPLAN, student motivation, student stress and why the online version of the standardized test is better than the pen and paper version. That wasn’t really the thrust of the front page quote though.

Chupa Chups.

That’s what made the headline.

Chupa Chups.

I was at a Prins meeting this week. A guy I hadn’t met saw my name and school and said: “Oh you’re the Chupa Chup guy.”

A nuanced discussion on NAPLAN is difficult in the Twittersphere. It’s even more difficult via the old fashioned print media, especially when my anecdote about Chupa Chups was the quote that the sub-editor picked on for the clickbait headlines.

Let me add some nuance to the Chupa Chup quote:

NAPLAN is a standardized test. I think it’s a pretty good one. Kids do it every second year. That’s not a massive amount of time out of the teaching and learning program for testing. We are not weighing the pig too often.

NAPLAN tests basic Literacy and Numeracy. We want kids to be capable in Literacy and numeracy. No-one disagrees with that.

Standardized tests are handy for schools. Parents like the information they give. They help our teachers moderate their teacher judgments against an external measure. They help us evaluate our school programs.

I like NAPLAN. We can improve it, but even in its current format, it is useful.

We’ve worked hard to improve writing at our school. Every teacher is involved, not just the English teachers. Maths, PE, Woodwork, Art, Drama, Science teachers; everyone. We’ve all built vocab list into our courses. We explicitly teach academic vocab. We’ve used quick writes to start lessons. Each teacher has put together a folio of how they are contributing to develop writing in their subjects as part of their Performance and Development process.

We’d hope our teacher’s efforts in improving writing would be reflected by our students’ NAPLAN performance.

Standardized tests can be useful to show learning growth. We place importance on the growth data produced from NAPLAN. John Hattie is held up as a demigod in Australian Education. No-one has escaped him saying “Twelve months learning growth for twelve months teaching for all students”. One of my favorite writers on education Becky Allen has the heretical view that maybe we cannot really accurately measure Learning Growth.

She outlines some significant hurdles to accurately measuring learning growth.

A major impediment in some subjects like History and Science are the nature of the curriculum continuum. Is the curriculum really a continuum? Is the curriculum actually defined well enough so you can accurately measure growth?

A standardized test of the same curriculum delivered two years apart should allow some measure of growth. The Australian Curriculum in English and Maths is well defined.

But Allen points out another problem. Is the standardized test delivered under similar conditions?

NAPLAN was done under similar conditions when we all did it on pen and paper but not so much now.

Our Year 7 to 9 learning growth this year compares a linear pen and paper test that the students sat in Year 7 to an online adaptive test they did this year as Year 9 students. Are we comparing apples to oranges here?

Another variable that Allen points to is student engagement. An accurate measure of growth can only be obtained from a standardized test if the student takes the test seriously: both times they take it.

In Australia, we have put significant additional public resources into education and the bean counters are concerned that Year 9 test results are not improving.

Victorian Education Minister James Merlino has listened to the troops on the ground and said, correctly, that we may have a problem with motivation here. Are students actually doing their best on the test or are they not really trying? Is the bad Year 9 data a result of a lack of motivation rather than poor Literacy and Numeracy?

Queue the Chupa Chups.

I was asked to comment on student motivation and NAPLAN. While our experience with the online NAPLAN was largely positive we did “stuff up” majorly the first year we did it. Where we stuffed up was on the motivational piece of the puzzle. We’d underestimated the lure of the big button that said: “Click here when finished.”

Here is what I wrote in 2018 when we first did the Online version of NAPLAN.

Challenge 4: NAPLAN Online is nowhere near as engaging as other things students can access online.

We did the writing task first.

Anyone who thinks NAPLAN is stressful for students should come and watch our students take it on.

For most of our students, they see NAPLAN as a “no stakes” test. Many of our students are well and truly on the left of the arousal curve where NAPLAN is concerned. We certainly didn’t have any on the right.

naplanonline

The anti NAPLAN media hype seemed to have flowed through to our Year 9 group.

One lad, after spending a massive 14 minutes of the 42 minutes allotted time on his essay, decided “I don’t want to do anymore because this is only for the government anyway.” I suggested he may like to try a bit harder for “personal challenge”, “individual satisfaction” or to “see how the first 10 Years of his formal education had him tracking compared to every other student in Australia of the same age”. He was having none of that.

Now back in the pen and paper days, this young man’s next 30 minutes would have consisted of staring out the window, or at the back of the head of the student in front. This boredom may have tempted him to read over the three short paragraphs he had written in HB pencil and add some more. Maybe he would fix up a spelling mistake or two.

Not in the online version. He’d had enough. He’s clicked the “I’m done” button. This takes him out of the locked-down browser and opens up the world of games, youtube and interesting stuff on the internet to him. I look back on the 15-Year-Old me and pose the same question, “Spend more time on a “no stakes”, persuasive writing task or have some free time on the internet?” Yes, we didn’t have the internet when I was 15 but if we did have the internet, I would have probably made that choice of footy videos on YouTube in preference to writing an essay.

At the 30 minute mark of the 42-minute writing task my face went pale and wave of depression flowed over me as I walked from room to room seeing many of our Year 9’s playing games or watching YouTube videos rather than writing the persuasive essay that would be some indicator of how well we had taught them to write. I’m not confident our writing results will be great this year.

We didn’t see that one coming. We’ll do better next year.

My prediction about our 2018 writing results was accurate. They were not great. Our worst ever.

We did do better this year.

We gave strict instructions for teachers that they were not to let students log out early, particularly in the writing task. Students were offered the incentive of a Chupa Chup if they stayed in the test window the whole time. We used to use the Chupa Chup bribe on the pen and paper test prior to 2018. Students had to write at least a page to earn the Chupa Chup. Judging a page in the online assessment is more difficult. We were happy for them just to stay the course and write for the full 42 minutes in the test window.

Our Year 9 Writing results this year are our best ever. Above the state and national average. Our 7 to 9 Writing growth was nearly double the national average.

The Chupa Chups helped incentivize some of our reluctant NAPLAN participants to do their best but our hard-working staff’s dedication to improving Writing over a five year period is what led to the improvement.

We should feel rightly proud of the learning growth our students have achieved but nagging doubts about the accuracy of the progress measure, and reliability of NAPLAN concern me. The results can bounce around so much with just minor changes in incentives. “Stay in the test window” vs “Play games when you’re done.” is the difference between triumph and disaster.

We are now in that twilight zone between comparing pen and paper test to online tests. Some schools are online and some are on pen and paper. Until all schools are online, both growth and school comparative data are probably not reliable.

I still like NAPLAN but we are placing way too much emphasis on the accuracy and reliability of the data it produces.

Newspapers and politicians love the simplicity of a number to measure how well our school system is doing. The number is not that reliable.

And it’s just one measure of what we do.

Merlino’s solution to our Year 9 motivation problem is a Year 9 Certificate. I don’t think it will be anywhere near as motivating to a Year 9 student as a Chupa Chup on the day.

And I don’t think the emphasis placed on the NAPLAN test results by the Media and the Government are commensurate with the validity of the results.

Today’s news is tomorrow’s fish n’ chip paper.

I wonder if Chupa Chups will sponsor our school?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s