Meagan Rogers

A Competency-Based Approach to Program Evaluation

Meagan Rogers, RN-BC,MSN, CPEN

Clinical Assistant Professor
College of Nursing and Health Innovation

The primary purpose of this project is to integrate learning analytics into the curriculum blueprinting and evaluation process through the development of a course and program quality management program. The need to measure learning outcomes and use data from various sources to individualize learning is driven by five problems: increased admissions, challenges in developing a predictive model to identify at-risk students, fragmented measures of learning, and clinical performance indicators that are not clearly linked to learning outcomes. This project is part of a larger initiative aimed at individualizing remediation through a streamlined process of identifying at-risk nursing students. The blueprinting and curriculum evaluation process involves tagging didactic content using a standardized concept taxonomy across the curriculum. Exam items are tagged to evaluate student performance and tailor remediation. Student results on three program assessments are tagged using the taxonomy: course exams, results from an adaptive learning platform, and virtual clinical simulation performance.

J. T. Dellinger

Pulling together all of the data sources is definitely difficult to achieve and you will never capture all of the data points, but it is admirable to try to integrate as many sources as possible. Do you know what software that you might be able to integrate using LTI? Do any of the sources share data with each other?

There is a bit of an echo in a few parts of the video - if you want me to upload a new one, feel free to send me a copy and I will overwrite it.
- MEAGAN ROGERS
  
  Hi Justin!
  I have to admit I get “data-saturated” easily, so I’ve challenged myself to continuously prioritize the data we need. Our LTI efforts are currently focused on three main external sources of data (outside of Blackboard):
  1. Lippincott, which includes electronic textbooks, virtual simulation, and adaptive quizzing for each course in the program
  2. ATI, which is a nationally validated predictor for our licensing exam and includes exams for each course, embedded and individualized remediation, and a comprehensive end of program exam.
  3. Clinical evaluations, which is currently a paper/pencil process!
  
  We have engaged Lippincott and ATI in conversations about report integration, but have not had much luck with blackboard, especially when there is no “grade” or assignment. However, fundamental to moving to an outcomes (or “competency-based”) approach is being able to track students’ learning between courses. ExamSoft offers an excellent LTI solution through the use of “rubrics”, which allow users to tag competencies (or concepts). They are mostly known as a cloud-based testing platform, but many medical schools in the UK and Canada are using ExamSoft rubrics for LTI.
  
  I am sorry I did not catch this in time to re-record the video. Thanks for all of your support!
Maria Trache

I admit that evaluation appears to be a really challenging issue in nursing programs. I guess those who train and evaluate future teachers or engineers experience similar problems. I am still not sure if the challenge comes from how to better measure true learning/understanding or from making sure graduates pass standardized exams for certification or licensing.
- MEAGAN ROGERS
  
  Great point, Dr. Trache!
  I believe pass rates are certainly an implicit motivator for much of this work. I see many of our competency-based efforts in the College of Nursing as a transition to more of an outcomes-based approach and leveraging technology to target high risk students with an intervention without waiting for them to fail a course or perform poorly on an end-of-program predictor exam. Even with a robust student success program, the sheer size of our program makes targeting at-risk students a challenge.
Regina Urban

Meagan, I think that pulling together these data points into a single dashboard would be challenging! However the benefits of obtaining the “big picture” of student performance and being able to hold them accountable for where their retention of knowledge is poor through identification and topic-specific remediation could go a long way towards preparing them for their career in nursing. In your work thus far with the three program assessments, have you found any patterns related to student knowledge and performance that would be considered strengths or weaknesses?
- MEAGAN ROGERS
  
  Hmmm…. that’s a good question. I truly believe we excel at identifying at-risk students and the intervention could be streamlined with technology. For example, time spent on PrepU, national percentile rankings on HESI (or now ATI), and even a few elements of VSim performance (number of level 3 errors) show promising potential as “red flags” for high risk students. But because we generally know who these students are, I think our real opportunity lies in the intervention. ATI’s individualized remediation is a fantastic first-step in getting missing content in front of students based on assessment results. I also think going digital with our clinical evaluations is an important first step in authentic assessments that are used to streamline an intervention for at-risk students. I am fascinated by the notion that there is not always a clear connection between classroom and clinical weaknesses. You have such a unique perspective on all of this, Regina. I wish we still officed together, but you would definitely be sick of me by now!
Jenny Roye

Learner centered…. like it! If we all thought along those lines, our student satisfaction scores would improve, I believe , as well. I echo Meagan’s sentiments- a dashboard spanning the entire program would be most helpful. Tailored remediation would be possible with the tagging system. I envision a push of a button remediation plan.. In a perfect world!!

PS- It might just have been my computer- I think the end or your presentation had two videos at once running..