...

Once the samples are selected, transcribers will complete the following steps to begin the reliability process.

1. Reliability procedures for transcriber 1:

1. Copy the original transcript from its folder to this folder: 3. Picnic Scene_Rely (transcriber 1)

2. Edit the title of the transcript by adding Coded_Rely AND your initials (e.g. SMK): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_Rely_SMK.cha​

2. Reliability procedures for transcriber 2:

1. Locate the audio files in the folder/link to the audio files: 1. S3_PicnicScene_Picture Description_Audios_Reliability

2. Locate the whisper output files in this folder: 2. PicnicScene_whisper_output_Reliability

3. Create transcription (.cha) file by copying the template that exists within this folder: 4. Picnic Scene_Rely (transcriber 2)​

4. Copy the whisper output CODE001_BACC001_PicnicScene_Spa_Pre_20230914.txt and paste it in the template (.cha)​

5. Fill out some particular fields in the headers of the file (@) (language, participant code, etc.,)​

6. Segment in utterances following the transcription protocol rules CHAT

7. Code with the transcription protocol rules​

8. Use CLAN to detect typos or spelling mistakes (command CHECK and command MOR)

9. Save in 4. Picnic Scene_Rely (transcriber 2) and make sure naming is correct by adding Coded_Rely AND your initials (e.g. AQ): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_Rely_AQ.cha​

3. Running

...

Rely Initial Comparison for Reliability:

1. Run transcriber 1 and transcriber 2 Coded_Rely samples through RELY

2. Add RELY output to this folder: 5. Picnic Scene_Rely output

3. Add score in Smartsheet CS Data Analysis in the 2 columns (report)

4. If score is less than 100% let transcribers know – schedule meeting – consensus final transcription – update in original folder, add date of meeting in CS Data Analysis (example: https://app.smartsheet.com/reports/96Xjp4G54hwCXM3pVVHP5xcrvVFfPXgh9wq49g81)

5. Initial Comparison for Reliability:

The selected samples will be assessed for transcription accuracy using the CLAN software and using Rely (How to use RELY on CLAN). Two key metrics will be evaluated:

• Percentage of Utterances with Matching Codes: Calculated as the proportion of all utterances where the assigned codes match perfectly between transcribers.

• Percentage of Words with Matching Codes: Calculated as the proportion of individual words with identical coding across transcriptions.

4. Discrepancy Resolution:
   If 100% agreement is not achieved:

   • A meeting will be scheduled between the involved transcribers to review discrepancies. It’s recommended to have this meeting after running through Rely all the samples used for reliability.

   • Consensus will be reached on the appropriate transcription for all disputed elements. The date of the meeting will be added in CS Data Analysis (example: https://app.smartsheet.com/reports/96Xjp4G54hwCXM3pVVHP5xcrvVFfPXgh9wq49g81)

5. Finalization of Reliable Transcriptions:
   Following the consensus meeting:

   • Edits will be made to the original transcription located in its original folder (e.g. B--Connected Speech_Data) based on agreed-upon revisions in the same meeting.

   • The transcription will then be considered finalized for reliability and all the samples of that project will be considered reliable enough to extract the linguistic measurements.

6. Documentation of Process:
   All reliability calculations and consensus resolutions will be documented in the Connected Speech Reliability Smartsheet Reports for project records, contributing to transparency and repeatability in the transcription process.

Overview of Reliability Folder Structure:

1. Reliability procedures for transcriber 1:

1. Copy the original transcript from its folder to this folder: 3. Picnic Scene_Rely (transcriber 1)

2. Edit the title of the transcript by adding Coded_Rely AND your initials (e.g. SMK): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_Rely_SMK.cha​

2. Reliability procedures for transcriber 2:

1. Locate the audio files in the folder/link to the audio files: 1. S3_PicnicScene_Picture Description_Audios_Reliability

2. Locate the whisper output files in this folder: 2. PicnicScene_whisper_output_Reliability

3. Create transcription (.cha) file by copying the template that exists within this folder: 4. Picnic Scene_Rely (transcriber 2)​

4. Copy the whisper output CODE001_BACC001_PicnicScene_Spa_Pre_20230914.txt and paste it in the template (.cha)​

5. Fill out some particular fields in the headers of the file (@) (language, participant code, etc.,)​

6. Segment in utterances following the transcription protocol rules CHAT

7. Code with the transcription protocol rules​

8. Use CLAN to detect typos or spelling mistakes (command CHECK and command MOR)

9. Save in 4. Picnic Scene_Rely (transcriber 2) and make sure naming is correct by adding Coded_Rely AND your initials (e.g. AQ): CODE001_BACC001_CatRescue_Spa_Pre_20230914_Coded_Rely_AQ.cha​

3. Running RELY

...

Run transcriber 1 and transcriber 2 samples through RELY

...

Add RELY output to this folder: 5. Picnic Scene_Rely output

...

Add score in Smartsheet CS Data Analysis in the 2 columns (report)

...

How to use RELY on CLAN

RELY Function 4:

...