Notes to the talk

Here is a brief that summarizes the main themes and important ideas discussed in the provided sources: an audio recording ("Data Analysis_042025.mp3") of a thesis seminar session and excerpts from a document titled "Making Sense of Stories: Analyzing Qualitative Data in ELT Teacher Training." The primary focus is on data analysis techniques, particularly qualitative coding, triangulation, and the potential for incorporating quantitative elements.

I. Key Themes and Important Ideas:

A. Importance of Completing Data Collection Before Analysis:

* The seminar leader emphasizes that data analysis should only begin after all data collection is complete. "Today's discussion is about data analysis. All of you have collected or very close to having completed uh collecting all of your data and this is an important requirement to continue the process of data analysis... If you are still trying to collect some information, know that what we talk about today uh you need to wait."

* Starting analysis prematurely, before all data is gathered, is considered a "mistake."

B. Understanding the Purpose of Data Analysis:

* Data analysis is crucial for understanding the collected data and determining what is relevant and significant to report in the results and discussion sections of the thesis.

* It helps researchers move from a large amount of raw data to focused and insightful findings. "Think of it like this. All of you are at this point, you've collected, if not all, most of your data. So you have all this data that you've collected... Ahora con todo esta información which data is not relevant... So you're going to then include this circle represents now only the information that relates to your research questions... Now from your data analysis... you're going to then figure out ok of all this information that now is relevant to my study, what is worth What is including in mys discussion?"

* Not all relevant data needs to be reported; the analysis helps identify the most "important, surprising, insightful, interesting" findings.

C. The Concept and Importance of Triangulation:

* Triangulation involves bringing together different data sources (e.g., interviews, observations, documents) to gain a more comprehensive understanding of the research topic.

* It allows for comparison between what participants say they do/believe, what they actually do (observed), and their planning/reflection processes. "Think of this if it helps to look at it like this. Your um your information here is allowing you to compare different things. For example, what people say they do or believe... What do they actually do? Well, to know that, what do we have to do? Have to observe."

* The seminar leader stresses the importance of having sufficient data to triangulate and encourages participants to address any concerns about this. "If anybody today right now has concerns about whether or not you have the types of data to allow you to triangulate, we need to have a discussion today."

* The "Making Sense of Stories" document provides specific examples of triangulation in ELT teacher training research, such as comparing planned instructions in lesson plans with delivered instructions observed in the classroom. "Compare the planned instructions (document) with the delivered instructions (observation). Were planned ICQs actually used?"

D. Introduction to Qualitative Coding:

* Qualitative coding is defined as a systematic process of labeling and organizing segments of text data (transcriptions, observation notes, documents) to identify patterns, themes, and concepts relevant to the research questions. "The process of coding is the process of labeling text. Coding is a systematic way to make sense of rich, complex, and often messy reality of language."

* All audio and video data must be transcribed into text before coding. Microsoft Word Online's transcription feature is suggested as a tool.

* The coding process involves identifying text segments (words, phrases, sentences, paragraphs) that relate to the research questions and assigning specific labels or "codes" to them. "You're coding things that relate to your research questions... Porque estamos en este proceso distinguendo, tenemos que distinguir qué sirve para nuestro estudio, qué no sirve, vamos a dejarlo fuera."

E. Levels of Qualitative Coding:

* The seminar introduces a three-level inductive coding approach:

* Level One (Initial Codes): Creating very specific labels directly from the text, the literature review, or using in vivo codes (participant's exact words). "The first you create... the code, the label comes from your literature review... Using a label a code directly. If anx dijo eso... Tú puedes seleccionar esta frase. ¿Qué lebo puedes poner? Anxious, anxiety."

* Level Two (Categories): Grouping the initial, specific codes into broader, more conceptual categories. "When we finish, you should have a long list of codes. And so I would do it in something like Excel... Les Segundo nivel es ordenar. Este grupo de códigos initial codes va aquí y voy a crear otra código. Puede ser como en category que representen todos sus códigos que son más específicos."

* Level Three (Themes): Grouping the categories into overarching themes that provide a higher level of understanding and relate directly to the research questions. "Level three, yo voy a poner este themes. Ya vamos a tener categorías, ¿verdad? Cada categoría va a tener sus initial codes. ¿Qué hicimos? ¿Qué hacemos para este nivel level? Categoriz group these categories into these yes."

* The "Making Sense of Stories" document also describes a similar iterative coding process, including immersion, initial/open coding, developing a codebook, focused/axial coding, and identifying themes/selective coding.

F. The Codebook:

* The outcome of the coding process is a codebook, which is a crucial part of the methodology section of the thesis.

* The codebook will list all the codes used, potentially organized by categories and themes, and may include definitions and examples. "Cuando terminen, you're going to have a codebook... you're going to include your codebook that's going to include all of the codes that you used and it's going to be an outline como esema in word. categories initial codes."

* The methodology section will describe the coding process and reference the codebook in the appendix.

G. Incorporating Frequencies and Duration (Quantitative Elements):

* The seminar leader emphasizes that qualitative data can be converted into quantitative data (frequencies, duration) for analysis. "How many of you think you'll need to analyze because we can convert qualitative information into quantitative information..."

* This involves counting the occurrences of specific codes or measuring the length of certain events (e.g., teacher-student exchanges, use of relaxation techniques).

* Examples discussed include tracking the frequency of positive/negative reinforcement, scaffolding, relaxation techniques, and the duration of collaborative work or interactions with specific students.

* The "Making Sense of Stories" document provides detailed examples of how to quantify qualitative data by defining observable behaviors, developing coding rules, and using presence/absence or frequency counts in spreadsheets.

H. Relationship Between Analysis and Reporting:

* The analysis process directly informs what will be reported in the results and discussion sections. "We don't know what to write in the results and discussion until we understand the data. To understand the data, we need to analyze the data."

* The evidence presented in the results section will often consist of direct quotes from the data that have been coded.

* The analysis (coding, identifying themes, considering frequencies) helps determine the structure and content of the results and discussion.

I. Openness to Modifying Research Questions:

* Based on the initial findings during data analysis, it may be necessary to slightly modify the research questions to better align with the emerging answers. "It's very common at this point as you are analyzing your data and when you come back on May 5th that in some cases we may need to modify slightly your research question."

* However, any modifications should remain within the scope of the literature review.

J. Timeline and Expectations:

* Participants are expected to begin the data analysis process (coding, considering frequencies) during the break before the next group session on May 5th.

* This analysis is considered a crucial step that will significantly impact the quality of the thesis.

* The final thesis paper is due on May 22nd, followed by mock presentations starting on May 26th and oral defenses.

K. Utilizing Large Language Models (LLMs) as Research Assistants:

* The "Making Sense of Stories" document introduces the potential of using LLMs to assist with qualitative data analysis.

* LLMs can help with generating initial coding ideas, applying preliminary coding schemes, calculating frequencies of codes, and analyzing Likert scale questionnaires.

* However, it is strongly emphasized that researchers must critically assess, validate, and cross-reference the output from LLMs to avoid bias and inaccuracies. LLMs should be seen as tools for au