Kim Breuer

Text and Citation Analysis of Undergraduate History Methods Papers

Kimberly Breuer, PhD

Assistant Professor in Practice
Coordinator of Online Learning

Department of History, College of Liberal Arts

My project explores the possible use of text and citation analysis as a means of evaluating writing skills and citations in an upper division History research and methods course. HIST 3300 is the only upper division course required of all History undergraduate degree plans and is typically one of the first upper division courses taken by majors. HIST 3300 is also tracked for departmental Unit Effectiveness purposes.

This project will evaluate several text analysis software (free or trial) to answer the following questions: Is text and citation analysis a useful tool for evaluating student writing; can we analyze papers for language skills/grammar, cohesion, sentiment, complexity of argument, and citations? Are the results of text and citation analysis useful for unit effectiveness purposes: Are there any discernable patterns for those students who produce papers with greater complexity and more scholarly citations (hours in major, where students completed lower division history coursework (UTA, transfer, examination credit), GPA (in major and overall)? Is there a “best pathway” for student success (should students take 3300 earlier or later in their coursework)? Does success/failure in 3300 predict subsequent GPA/student success? Do the results point to curricular changes needed to improve research and writing skills text analysis of method papers?

J. T. Dellinger

Hi Kim, I really enjoyed your project and I am very interested to see if there is a correlation between writing development in HIST3300 and future history course GPA/program completion, as well as what interventions could be useful to increase success and retention. For someone with very little experience using tools such as LIWC and Coh-Metrix, do you have some quick suggestions of places to start and/or some resources to use?
- Kim Breuer
  
  Both the LIWC and Coh-metrix websites link to resources and journal articles. I found a quick review of this linked literature to be enough to get me started. After that I began a targeted review of articles using text analysis to assess writing skills to see what types of variables others utilized.
Regina Urban

Kim, thanks so much for your presentation. It was useful to hear a review of these programs and their effectiveness for evaluating student writing (ie. grade level of writing, analytical thinking, and complexity) and for unit effectiveness reporting. I’m looking forward to hearing you and Heather’s combined efforts and the dissemination of your findings regarding how to time the course more effectively for some student groups. I’m thinking there’s some applicability for LIWC or Coh-Metrix in our RN-BSN program to evaluate student papers for change at different points of the program. Thanks for the introduction and comparison of these tools. ~ Regina
- Kim Breuer
  
  I was a bit skeptical myself as to whether these tools would provide quantifiable assessment data for analytical writing, but I am now confident in using them for unit effectiveness assessment. I can definitely using these tools to assess improvement over time for individual students.
Cindy

When I am ready to start looking at our writing samples for placement, I am going to come to you for guidance with these programs!
Peggy Semingson

I like how you compared the different tools. I may still look into Voyant but I think the word clouds can be pretty superficial for data visualization. I didn’t even know citation analysis was a thing until I learned about your and Heather’s project! Thanks, Kim!