After issues on April 14th and 15th, glitches cropped up again today (4/21) in Minnesota's high-stakes online MCA tests. This time, nearly every district in the state experienced some issues. At least one local news organization has picked up on the problems. I have feeling more will follow. (And they have - Pioneer Press - WCCO - In fact, since I started writing this post, MDE has suspended all testing until further notice)
The MN Department of Education (MDE) and the testing vendor, Pearson held a joint conference call on Monday (4/20). During the Q & A, dozens of MN districts reported various issues that caused, at minimum, frustrating delays in starting test sessions for students and at worst, lost data. What was clear was that these issues were not isolated to specific tests or geographic location of the schools and they were not usually caused by local school district network or setup problems.
So, bring on today (Tuesday 4/21) and by 8:45 this morning, Pearson and MDE acknowledged that some of the systems were in a "degraded" state resulting in delays. For about 2.5 hours in the middle of the day, the systems were reportedly working, before going back into a degraded state and eventually becoming unresponsive. (my guess is it worked during the lunch hours when few testing sessions were being held.)
If you've read my previous posts, you know that I have questions about the validity of a testing system that introduces variables of anxiety and frustration. I've conveyed that message to many people including officials at MDE and as you saw above, MDE has now suspended testing citing some of the same concerns I and many others have pointed out. Also of note is that these issues have affected testing in other states as well.
I did find it interesting that one of the responses I received from an MDE official pointed to me to a study that had been done after the 2013 Spring testing season when similar issues occurred with the previous testing vendor, American Institutes for Research (AIR). After taking a brief look, I recalled that this was used to calm the concerns over validity as the study generally indicated that there was no significant effect to students who experienced the issues on those days in April 2013. However, there are significant problems with the study and it so narrowly examined ONLY the specific issue that occurred on those days that it certainly has NO correlation to what we are experiencing now. I'll probably do a separate post about that study - and I have a general post about testing coming up too. Check back soon if you think you might be interested in either of those ramblings!
Tuesday, April 21, 2015
Wednesday, April 15, 2015
Minnesota Online State Testing - Not Fair to Students
So, Minnesota's high-stakes online student testing (known as MCA's) kicked off on March 9th, but because of Spring breaks and the desire to have as many instructional days as possible before testing, most districts wait until later in the 8-week testing window (which is ridiculously long from a statistical point of view) to administer the tests. This week was the first "heavy" week of testing, though some started earlier. And, as many predicted, the online systems received a failing grade, but it's the students who really lose.
A talented technology coordinator from Milaca, Steve Bistrup, wrote a message to his administration about their experiences with testing thus far. Here is a sample of what he wrote - warning, it is a bit long, but I think well worth the read and sharing with others!!
Long Wait #1 During the mornings in particular, but not exclusively, students have to wait an unacceptably long time for Pearson's main "Self Registration" page to load. Staff are now having to come to work extra early to "pre-load" Pearson's main server page so that students don't have to wait the 5-20 minutes for the login page. This is extremely difficult when we potentially will be testing six sessions simultaneously. (This slowness is NOT a result of our network infrastructure or speeds... these have been ruled out conclusively by technicians and Pearson support).
Long Wait #2 Once students enter their credentials and test code there is another "randomly" long waiting period. We have seen students wait as long as 45 minutes for "the blue bar" to finally load their test questions. (Again, this is NOT a result of our caching server or internal network)
Two to four adults needed to start each session Because we are always having to restart the browser, try a different computer, empty the java cache, reset the browser, etc. until we can finally get a student logged in to the test, an unacceptable number of adults are needed in order to reduce the amount of time students are sitting in front of a spinning wheel while the person next to them is 20 minutes into their test. I can't imagine the amount of anxiety this creates in the students. It certainly creates a high degree of anxiety and frustration in those that desperately want the students to have a fair shot and do their best work. I recall one student who anxiously waited for his test to load....worried that if he had to wait too long he would miss outside recess. The teacher and I had to both reassure him to take his time once he was logged in.
Constant browser plug-ins updates and Pearson's sporadic warnings On 4/14/15 Oracle released a new version of Java. Because Apple computers and TestNAV require you to be on the latest version of Java, staff needed to stay long into the evening that night in order to update Java on every computer that students would be testing on the next morning. For Milaca this is 120 laptops and 90 lab computers. There was also a new version of Flash on this date. Pearson warned us about the Java update but not the Flash update. Luckily we caught it. I can't begin to count how many times Java and Flash have had updates since school started. Some of them Pearson informs us of...more often then not they don't.
Math Test Calculator Issues Pearson acknowledged this as a known issue during the OLPA session. We have now had the same issue during the MCA math sessions. When a student uses the built-in calculator during a math test it randomly locks up their test session. Pearson's only answer has been to "force quit the test and have the student log back into the test." *see points one and two why this is more than inconvenient. Worse, a student has to shut down and restart their testing mindset and momentum.
Server outage and support On 4/14/15 Pearson Access server crashed or "became degraded" across the entire State of Minnesota resulting in testing administrators being unable to manage test sessions. When I called I was put on hold and twice was disconnected with a busy signal. When Pearson eventually sent out a formal message the verbiage was disingenuous. The message made a point to emphasize that "TestNAV and Testing were not effected." Yes, for those kids that were currently in their test they were able to continue. However, no other session could be started or resumed resulting in entire testing sessions being missed or postponed.
Not a fair assessment for kids I have been involved with online testing since the very beginning and I can say that no testing period has been approached with more trepidation by those managing the testing. Unfortunately we have now seen those fears played out. These past two weeks have been frustrating and unfair to kids. If the testing mechanism is supposed to be as transparent as a paper booklet and as technologically easy as working a #2 Dixon Ticonderoga then this has been a colossal failure. I've watched kids put their heads down in frustration as they wait. I've seen students lose their focus as a result of the glitches. I've seen the worry on an instructors face as they watch all the preparation and build-up deflated in a matter of minutes.
Hi - I'm back! I have examples from many other districts too, but Steve says it well, so I wanted to share those experiences. (Congratulations, by the way if you've reached this part of the post!) I can say that not all schools have experienced all of the same issues, but that just goes to reinforce the idea that this is not a fair assessment for kids. In the MN Department of Education Assessment Update - released today 4/15/15 - They included the following message:
To reduce testing disruptions for students, take appropriate actions to reduce noise, such as limiting the use of alarms (not fire alarms), bells, and announcements. Districts should avoid scheduling test sessions during known interruptions, such as during lunch breaks or pre-planned fire or tornado drills.
What about the disruptions caused by all the issues Steve points out? If a district encounters those kinds of issues (NOT a result of anything they did or didn't do) can we really validate that data? Is it a fair comparison to previous years' results if the same issues were not present then? Can anyone feel good about evaluating the success of teachers, students and schools based on such inconsistent testing environments? Heaven forbid we compare districts to each other based on these results. Yet, that is exactly what happens.
The MN Department of Education publishes their "Minnesota School Report Card" on their website each year. Designed for the public to look at progress and demographic information about schools - and built to provide side-by-side comparisons. Front and center on each schools "report card" is proficiency data for the MCA tests for the past 5 years. No where does it indicate that you may be comparing a school where online testing worked flawlessly and one where students were subjected to the kinds of distractions and frustrations that Steve noted. Yes, I firmly believe there would be statistically significant differences. Not to mention you are looking at results from year to year when entirely different groups of students were tested - I'm sure there are no variables there that could influence validity! Plus (though I'll admit this is noted if you look carefully) standards have changed in all three tested areas in the past 5 years, which again, invalidates comparisons from year to year.
I'll stop now - you get the point about how unfair this situation is. Kudos to you if you made it this far! I'll probably break this up, condense it and repackage it in some messages to folks at MDE, legislators, etc. Feel free to share it with anyone you think has the attention span to read it!! And... stay tuned for more updates and even some thoughts about how to improve it.
A talented technology coordinator from Milaca, Steve Bistrup, wrote a message to his administration about their experiences with testing thus far. Here is a sample of what he wrote - warning, it is a bit long, but I think well worth the read and sharing with others!!
Long Wait #1 During the mornings in particular, but not exclusively, students have to wait an unacceptably long time for Pearson's main "Self Registration" page to load. Staff are now having to come to work extra early to "pre-load" Pearson's main server page so that students don't have to wait the 5-20 minutes for the login page. This is extremely difficult when we potentially will be testing six sessions simultaneously. (This slowness is NOT a result of our network infrastructure or speeds... these have been ruled out conclusively by technicians and Pearson support).
Long Wait #2 Once students enter their credentials and test code there is another "randomly" long waiting period. We have seen students wait as long as 45 minutes for "the blue bar" to finally load their test questions. (Again, this is NOT a result of our caching server or internal network)
Two to four adults needed to start each session Because we are always having to restart the browser, try a different computer, empty the java cache, reset the browser, etc. until we can finally get a student logged in to the test, an unacceptable number of adults are needed in order to reduce the amount of time students are sitting in front of a spinning wheel while the person next to them is 20 minutes into their test. I can't imagine the amount of anxiety this creates in the students. It certainly creates a high degree of anxiety and frustration in those that desperately want the students to have a fair shot and do their best work. I recall one student who anxiously waited for his test to load....worried that if he had to wait too long he would miss outside recess. The teacher and I had to both reassure him to take his time once he was logged in.
Constant browser plug-ins updates and Pearson's sporadic warnings On 4/14/15 Oracle released a new version of Java. Because Apple computers and TestNAV require you to be on the latest version of Java, staff needed to stay long into the evening that night in order to update Java on every computer that students would be testing on the next morning. For Milaca this is 120 laptops and 90 lab computers. There was also a new version of Flash on this date. Pearson warned us about the Java update but not the Flash update. Luckily we caught it. I can't begin to count how many times Java and Flash have had updates since school started. Some of them Pearson informs us of...more often then not they don't.
Math Test Calculator Issues Pearson acknowledged this as a known issue during the OLPA session. We have now had the same issue during the MCA math sessions. When a student uses the built-in calculator during a math test it randomly locks up their test session. Pearson's only answer has been to "force quit the test and have the student log back into the test." *see points one and two why this is more than inconvenient. Worse, a student has to shut down and restart their testing mindset and momentum.
Server outage and support On 4/14/15 Pearson Access server crashed or "became degraded" across the entire State of Minnesota resulting in testing administrators being unable to manage test sessions. When I called I was put on hold and twice was disconnected with a busy signal. When Pearson eventually sent out a formal message the verbiage was disingenuous. The message made a point to emphasize that "TestNAV and Testing were not effected." Yes, for those kids that were currently in their test they were able to continue. However, no other session could be started or resumed resulting in entire testing sessions being missed or postponed.
Not a fair assessment for kids I have been involved with online testing since the very beginning and I can say that no testing period has been approached with more trepidation by those managing the testing. Unfortunately we have now seen those fears played out. These past two weeks have been frustrating and unfair to kids. If the testing mechanism is supposed to be as transparent as a paper booklet and as technologically easy as working a #2 Dixon Ticonderoga then this has been a colossal failure. I've watched kids put their heads down in frustration as they wait. I've seen students lose their focus as a result of the glitches. I've seen the worry on an instructors face as they watch all the preparation and build-up deflated in a matter of minutes.
Hi - I'm back! I have examples from many other districts too, but Steve says it well, so I wanted to share those experiences. (Congratulations, by the way if you've reached this part of the post!) I can say that not all schools have experienced all of the same issues, but that just goes to reinforce the idea that this is not a fair assessment for kids. In the MN Department of Education Assessment Update - released today 4/15/15 - They included the following message:
To reduce testing disruptions for students, take appropriate actions to reduce noise, such as limiting the use of alarms (not fire alarms), bells, and announcements. Districts should avoid scheduling test sessions during known interruptions, such as during lunch breaks or pre-planned fire or tornado drills.
What about the disruptions caused by all the issues Steve points out? If a district encounters those kinds of issues (NOT a result of anything they did or didn't do) can we really validate that data? Is it a fair comparison to previous years' results if the same issues were not present then? Can anyone feel good about evaluating the success of teachers, students and schools based on such inconsistent testing environments? Heaven forbid we compare districts to each other based on these results. Yet, that is exactly what happens.
The MN Department of Education publishes their "Minnesota School Report Card" on their website each year. Designed for the public to look at progress and demographic information about schools - and built to provide side-by-side comparisons. Front and center on each schools "report card" is proficiency data for the MCA tests for the past 5 years. No where does it indicate that you may be comparing a school where online testing worked flawlessly and one where students were subjected to the kinds of distractions and frustrations that Steve noted. Yes, I firmly believe there would be statistically significant differences. Not to mention you are looking at results from year to year when entirely different groups of students were tested - I'm sure there are no variables there that could influence validity! Plus (though I'll admit this is noted if you look carefully) standards have changed in all three tested areas in the past 5 years, which again, invalidates comparisons from year to year.
I'll stop now - you get the point about how unfair this situation is. Kudos to you if you made it this far! I'll probably break this up, condense it and repackage it in some messages to folks at MDE, legislators, etc. Feel free to share it with anyone you think has the attention span to read it!! And... stay tuned for more updates and even some thoughts about how to improve it.
Monday, April 13, 2015
Keep Rural Broadband a Bipartisan Priority
Throughout the current Minnesota legislative session, rural broadband development has presented itself as mostly a bipartisan issue. A Republican authored bill in the House (HF593), a Democrat authored bill (SF439) in the Senate and co-authors from both sides of the aisle.
Then, HF843 arrives as an omnibus economic development bill and the Office of Broadband Development (OBD) and Border to Border (B2B) grant program are removed from the original language of the bill. An article appeared on the StarTribune website on Thursday April 9th indicating that the funding had been removed, adding comments from Representative Pat Garofalo (R - Farmington).
I fired off a number of messages to my legislators, those representing the schools I work with, members of the Job Growth and Energy Affordability Policy and Finance Committee, a few others and Representative Garofalo. My note to Rep. Garofalo is shown below. I encourage any of you with an interest in this issue to voice it now. I've received a fair amount of feedback from my messages including some that aim to calm me by saying, "...things in St. Paul aren't always pretty..." and, "Bills are commodities... we trade commodities." I understand all of that, but it doesn't mean I have to like it, and I know that if we stay silent, then no one believes it is important.
*** Below is an email message sent on Friday, April 10th, 2015 ***
Representative Garofalo:
Then, HF843 arrives as an omnibus economic development bill and the Office of Broadband Development (OBD) and Border to Border (B2B) grant program are removed from the original language of the bill. An article appeared on the StarTribune website on Thursday April 9th indicating that the funding had been removed, adding comments from Representative Pat Garofalo (R - Farmington).
I fired off a number of messages to my legislators, those representing the schools I work with, members of the Job Growth and Energy Affordability Policy and Finance Committee, a few others and Representative Garofalo. My note to Rep. Garofalo is shown below. I encourage any of you with an interest in this issue to voice it now. I've received a fair amount of feedback from my messages including some that aim to calm me by saying, "...things in St. Paul aren't always pretty..." and, "Bills are commodities... we trade commodities." I understand all of that, but it doesn't mean I have to like it, and I know that if we stay silent, then no one believes it is important.
*** Below is an email message sent on Friday, April 10th, 2015 ***
Representative Garofalo:
With all due respect, I believe any decision to defund the Office of Broadband Development (OBD) and the Border to Border (B2B) grant program will set rural Minnesota back years and maybe even decades and is a detriment to residents and businesses who wish to locate in rural areas of the state, not to mention schools, healthcare facilities and so on.
I am the Executive Director of a mostly rural cooperative of school districts. We provide technology services to our districts including wide area network support, network services, Internet access and many other instructional and professional development services. I have also been working for four years with the Kanabec Broadband Initiative and the East Central Regional Broadband Committee to bring broadband options to our rural residents. I have testified for Representative Kresha's bills, including that which would have funded the OBD and B2B program.
A Star-Tribune article indicated that you believe wireless options are cheaper. I beg to differ. We have wireless providers in our area including fixed wireless, cellular and satellite. In ALL cases the costs to consumers are significantly higher than wired (copper or fiber) services due to high equipment costs, data caps and generally high monthly fees. The service level is significantly slower than wired options - in most cases barely reaching the minimum definition of broadband at best and usually much slower - and the wireless solutions experience many more service interruptions than any wired option. I think these are easily documented facts and I challenge you to find a significant number of consumers or rural business who are satisfied with wireless services and think it is an affordable solution. Certainly, in our part of that state, they are not affordable options.
I would concede that SOME wireless solutions MAY be cheaper to deploy - at least initially, but I believe that may be offset by the fact that the equipment to drive wireless solutions has a much shorter lifespan than most equipment used in wired solutions. Additionally, the limited amount of spectrum available for wireless solutions makes it difficult to deploy in some areas of the state. It is nothing more than a "Band-Aid" fix.
I encourage you to re-examine this issue and avoid making it a political pawn. Broadband is as important to life and economic development today as electricity and water. Rural Minnesotans will be the losers if this issue dies in the session. If this issue dies, it will effectively kill years of work done by groups around the state to build capacity for public-private partnerships - the kind that the B2B program created.
Please feel free to contact me at any time to discuss. My contact information, including cell phone are below. Thank you for your time and service to the state.
Regards,
Marc Johnson
Monday, February 2, 2015
Border to Border Broadband in Minnesota?
OK... it's been a year, so it's time to once again opine on the pages of the Interweb...
Today, Minnesota's Office of Broadband Development (part of DEED) released the long-awaited list of grant recipients for their $20 million Boarder to Boarder Broadband project. An impressive list of 17 projects scattered across the state of Minnesota. First, congratulations to those behind the funded projects. If you are lucky enough to live in one of the areas that will now receive high-speed broadband, welcome to the 21st century!! You'll love it!
Next, am I disappointed that Kanabec County - and all of East Central Minnesota for that matter - was left out? Yes I am. Lots of people have been working hard for over four years to get better broadband options for our area. Another setback doesn't help, but we can learn from it and strive to present a better case next time. And I think we will.
In the meantime, I've done a little math and present the following information... not to diminish the projects that were funded, but to illustrate just how big this problem still is in the state... and to hopefully help keep the momentum going into this legislative session as we look for continued funding of this program.
So, today's announcement helps the overall broadband conversation/discussion... as long as the legislature doesn't see this as "now we've fixed that problem." It looks to me that the projects chosen were some of the smaller $$ projects among the 40 applicants. They are nicely distributed across the state, and, yes, I know why that matters! There were a couple that were larger, but the average award is just over $1 million and most of the projects serve relatively small geographical and demographical numbers. There were 40 applications totaling $44 million in requests. I know there are lots of un and under served areas that didn't apply because the maximum $5 million amount wasn't enough to make a significant difference in their area, or because they didn't have the necessary partnerships or planning in place to make it happen. So... I hope the legislature doesn't look at this and say... "Well, looks like another $20 million should do the trick and we'll be done." That can't be further from the truth.
Today, Minnesota's Office of Broadband Development (part of DEED) released the long-awaited list of grant recipients for their $20 million Boarder to Boarder Broadband project. An impressive list of 17 projects scattered across the state of Minnesota. First, congratulations to those behind the funded projects. If you are lucky enough to live in one of the areas that will now receive high-speed broadband, welcome to the 21st century!! You'll love it!
Next, am I disappointed that Kanabec County - and all of East Central Minnesota for that matter - was left out? Yes I am. Lots of people have been working hard for over four years to get better broadband options for our area. Another setback doesn't help, but we can learn from it and strive to present a better case next time. And I think we will.
In the meantime, I've done a little math and present the following information... not to diminish the projects that were funded, but to illustrate just how big this problem still is in the state... and to hopefully help keep the momentum going into this legislative session as we look for continued funding of this program.
So, today's announcement helps the overall broadband conversation/discussion... as long as the legislature doesn't see this as "now we've fixed that problem." It looks to me that the projects chosen were some of the smaller $$ projects among the 40 applicants. They are nicely distributed across the state, and, yes, I know why that matters! There were a couple that were larger, but the average award is just over $1 million and most of the projects serve relatively small geographical and demographical numbers. There were 40 applications totaling $44 million in requests. I know there are lots of un and under served areas that didn't apply because the maximum $5 million amount wasn't enough to make a significant difference in their area, or because they didn't have the necessary partnerships or planning in place to make it happen. So... I hope the legislature doesn't look at this and say... "Well, looks like another $20 million should do the trick and we'll be done." That can't be further from the truth.
The latest broadband coverage information put out by the Office of Broadband (DEED) Found here: http://mn.gov/deed/
It's a start... and great news for the people in those communities. If you live in one of those areas, please thank your state representatives and senators. Then, if they happen to be in your area for a press conference or photo-op, or just to have a community forum, please remind them that there is still about 20% of the state's population that do not have the same opportunities. It would help us all if everyone would move that message forward.
Subscribe to:
Posts (Atom)