Blog post #5- Measurement

My current research problem is “How do the Directed Reading Thinking Activity and Story Mapping comprehension strategies utilized over a 20-week period retain their effectiveness in terms of usage and recall over a 2-year span with elementary students of varying reading ability levels?  Also, how is student motivation about comprehension tasks in the 2-year follow up?"

I have changed my design a bit to make my study more reliable.  The study will still be conducted at 5 elementary schools across 5 different school districts.  However, I will be at one school and four other reading specialists will be at the other schools.  They will be trained on how to conduct each of my measurements.

I have also decided to conduct the intervention over a 20-week period instead of an entire school year based off of many of the articles I have been reading.  The intervention will be done in the second half of the 2nd grade school year.  The teachers implementing the comprehension strategies will also be given significant training beforehand on how to teach each strategy using a specific framework.

Measurements to Collect Data

1.      Survey

a.   I will use a survey in the 2-year follow-up to measure whether students remember the strategies, how to use them and how often they still use them today.

b.   I will be measuring this with a short survey containing four open-opened questions.  The questions will ask students what the strategies were, how to use them, when and why they are useful and how often they still use them today.  The students will have the choice to either write their responses or respond orally. 

c.    This instrument will be valid because it is directly related to what I am trying to measure.  I will also be able to infer from this measurement whether students remember and use the strategies in the follow-up.  This one measurement tool will be used in conjunction with other measurements to confirm the validity of its findings (triangulation of data).  If similar results are found throughout all the measurements the validity will increase.  I will also confer with other literacy experts (reading specialists) about this survey to assure its validity. 

d.   This measurement will be reliable because it is a short survey which is more reliable than a long survey.  It will also be given to a heterogeneous group of students from five classes across five different school districts.  The students will have various reading levels which will make the results more comparable to other students.  In hopes of truthfulness, the students will be told that the survey is anonymous.  All of the reading specialists giving the survey will give students the same directions and the conditions will be similar.  The reading specialists will also all be trained on how to score this open-ended survey and we will compare results at the conclusion.

2.      Interviews

a.   The other reading specialists and I will anonymously interview both teachers and students throughout the study.  We will be trying to measure the teacher’s perspective on the effectiveness of the reading strategies immediately following the intervention and we will be trying to measure students’ attitude and motivation towards comprehension tasks two years after the intervention.

b.   In the interview with the classroom teacher we will ask them if they noticed a positive change in their students’ reading comprehension because of the strategies.  We will also ask if the students’ overall attitudes towards reading and comprehension tasks have changed at all as a result of the strategy intervention.  In the interview with the students we will ask questions such as:

-Did you enjoy using the comprehension strategies in 2nd grade?  Do you think they helped you?

-Do you now find comprehension tasks easy or difficult?

-Do you normally understand a story or book well after you have read it?

-Do you ever use some of those same strategies you learned in 2nd grade?

-Do you feel confident or unsure when asked to complete comprehension tasks?

-Would you rather complete a comprehension assignment or another reading task?

After completing the student interview we will gather their information and complete a short follow-up interview two weeks later to confirm their thoughts and ask additional questions based off of their first interview.

 c.  These interviews will be valid because we will have the perspectives of both the teachers and the students related to the main research question.  Some of the interview questions also overlap with the survey questions which will confirm validity if similar results are found.  Interviewing the teachers immediately following the intervention will allow me to see whether the strategies seemed to work well in order for the follow-up data in two years to be relevant and appropriate.  These interview questions have been specifically designed to help answer my research questions and I will also confer with the other literacy experts on whether to add or delete questions.

d.  These interviews will be reliable based on their stability over time.  We will be conducting two interviews with the students in order to assess whether their answers change or remain stable.  There is also inter-rater reliability as all the reading specialists will be trained on how to interview and interpret the answers.  The reading specialists will take detailed notes during the interviews.  At the conclusion we will compile and compare our data.  Also, the population of students is very heterogeneous which results in better reliability and generalizability. 

3.   Observations

a.   In the informal observations of teachers by the reading specialists we will be trying to measure whether the procedures are being followed and the strategies are being taught correctly in order to provide support where needed.  We would also be measuring student engagement, motivation and attitudes towards the strategies.

b.   In order to measure the above topics the observer will visit the classroom and take detailed field notes.  The observer will be cautious as to not interfere with the normal atmosphere of the class.

c.    These field notes will be valid because they will help ensure that the strategies in my research questions are being correctly taught in order to attain relevant data at the conclusion of the intervention.  Also, we will be able to compare field notes from these observations to follow-up data two years later.  Being literacy experts, the reading specialists will also know what to look for when they are in the classroom, making this measurement valid.  This is also only one source of data that will be combined with other measurement results.

d.   The observations will be reliable because there will be multiple reading specialists taking detailed field notes over an extended time period at various schools.  The schools will be different but the instruction and grade level will be the same.  Each reading specialist will also be trained in the observation procedure and what they should be looking for.  At the conclusion all of the reading specialists will compile and compare their data to look for similar results.  Since there will be multiple observations, the observer will be able to take extensive field notes and observe the class dynamics on different days.  The detailed field notes will eliminate the possible effect of the researcher’s subjective opinions.

4.       Records

a.   I will be trying to measure the characteristics of each school district in order to choose a school to work with that represents that district as a whole.

b.   I will measure the population of the school district by examining ethnicity, number of students using free and reduced lunch, district budget, dropout rates and test scores.  I will use information available to the public through district websites and articles.

c.   Using data collection will be a valid measurement because it provides crucial information to guide my research problem.  Analyzing multiple measures of population statistics will be more valid than simply basing my decisions off of one characteristic.

d.   These records are reliable because they have been tracked and recorded over the years.  The data has also been collected from multiple school officials representing their district.