VISTA POM Reliability

One part of our project uses Video Implemented Script Training for Aphasia (VISTA), a form of rehabilitation in which patients practice with scripts. What is unique about VISTA, though, is that the patients receive video recordings of a native speaker saying their script so that they can they practice along as they (re-)watch it. This ensures that they have the means to consistently train with their scripts rather than needing to wait until their next session with the clinician. The video itself only shows the nose, mouth, and chin of the native speaker being recorded. This is important because it visually demonstrates the movements that are necessary for the successful production of all the sounds of each word in the script.

Another notable aspect of our project is that the scripts are not pre-selected for the patients. Instead, we work with the patients to identify different parts of their life that they would like to practice with (e.g., family, trips, food, etc.). From there, we work with each patient to create a set of scripts that they will practice with for the entirety of the study.

The patients’ ability to produce the scripts will be probed at specific timepoints throughout the course of the experiment. We measure their performance at each probe by having the clinician transcribe what the patient produces for the target script. We then quantify the number of target words that were produced by using a specially designed calculator. The output from this calculation then acts as the primary outcome measure (POM) of the experiment. This provides an in-depth look at the maintenance or loss of language ability over the duration of the study.

However, we must be sure that when we later analyze our data that the measures are accurate and reliable. For this reason, we have developed a system to verify the accuracy of our POM. We have a second rater listen to the audio recording of the probe, and then transcribe and assess the POM for each patient’s probes. It is crucial that this rater does not know any information about the data they are given. That means they do not know which patient it is or the specific session (e.g., pre-treatment or post-treatment). This ensures that they are not biased as they transcribe and assess the output. By including 2 separates assessments, one from the clinician (rater 1) and the new rater (rater 2), we can then compare the transcription (and thus the POM) to assess accuracy and reliability of our data.

This process is critical and must involve careful planning by a reliability supervisor to ensure that the proper steps are taken in the right order. It is the reliability supervisor that will compare the two raters’ transcriptions.

[Work in progress]

The procedure and folder structure can be found below: