"We did a lot of experiments to fine-tune the software, then confirmed that the software's assessment correlated very highly with an assessment from a manual rubric developed and applied by Puntambekar's lab."
"In PyrEval-CR, we created the same kind of model that PyrEval would create from a few passages by expert writers but extended it to align with whatever rubric makes sense for a particular essay prompt," Passonneau said. She also provided historical student essay data that was needed to develop PyrEval-CR before deploying it in classrooms. Sadhana Puntambekar, professor of educational psychology at the University of Wisconsin-Madison and a collaborator on both papers, recruited the science teachers and developed the science curriculum.
The researchers tested the functionality and reliability of PyrEval-CR on hundreds of real middle school science essays from public schools in Wisconsin. Starting in 2012, Passonneau and her graduate students worked to automate Pyramid, which led to the creation of the fully automated PyrEval, the precursor of PyrEval-CR. The beginnings of PyrEval-CR date back to 2004, when Passonneau worked with collaborators to develop the Pyramid method, where researchers annotate source documents manually to reliably rank written ideas by their importance. "Simultaneously, the software generates a summary report on topics or ideas present in the essays from one or more classrooms, so teachers can quickly determine if students have genuinely understood a science lesson." "PyrEval-CR can provide middle school students immediate feedback on their science essays, which offloads much of the burden of assessment from the teacher, so that more writing assignments can be integrated into middle school science curricula," Passonneau said. Led by Passonneau, the researchers who worked on the ISLS paper extended the abilities of an existing natural language processing tool called PyrEval to assess ideas in student writing based on predetermined, computable rubrics. Natural language processing is a subfield of computer science where researchers convert the written or spoken word into computable data, according to principal investigator Rebecca Passonneau, Penn State professor of computer science and engineering.
0 kommentar(er)
