{"id":458,"date":"2017-04-24T12:43:20","date_gmt":"2017-04-24T19:43:20","guid":{"rendered":"http:\/\/test-inside.ewu.edu\/successfultransitions\/?page_id=458"},"modified":"2024-03-28T16:57:08","modified_gmt":"2024-03-28T23:57:08","slug":"design-cycles","status":"publish","type":"page","link":"https:\/\/test-inside.ewu.edu\/successfultransitions\/mathematics-inquiries\/algebra-a\/design-cycles\/","title":{"rendered":"Design Cycles"},"content":{"rendered":"

The First Design Cycle<\/strong><\/p>\n

 <\/p>\n

Initial design of the observation protocol began by watching videos of high school mathematics classrooms that had high amount of in-class student work. As we watched the video, we began to record student behaviors that we found relevant to the question of measuring students\u2019 use of resources to resolve a problem. This list was then supplemented with behaviors that we had observed in our own experience. This list of behaviors became our open codes:<\/p>\n

Teacher identifies resources<\/p>\n

Teacher provides instruction on use of resources<\/p>\n

Teacher does not create opportunities for other resources (lecture\/quiz\/etc)<\/p>\n

Teacher answers question that could have be redirected to class\/student<\/p>\n

Student asks for more student-student interaction<\/p>\n

Student asks for more work time<\/p>\n

Student requests information on resources<\/p>\n

Student requests instruction on use of resources<\/p>\n

Student asks question of teacher<\/p>\n

Teacher creates opportunities for students to ask questions of each other<\/p>\n

Teacher asks open-ended question<\/p>\n

Students ask questions of each other<\/p>\n

Students share observations with other students<\/p>\n

Students observe and learn from each others’ approaches to problems<\/p>\n

Student builds on another student’s comment<\/p>\n

Student challenges another student’s comment<\/p>\n

Student expresses uncertainty to another student<\/p>\n

Student waits for teacher to come along to solve the problem<\/p>\n

Internet access restricted<\/p>\n

Student passively uses assigned resource (read text\/watch video)<\/p>\n

Internet access made available<\/p>\n

Teacher provides students with control of reference (textbook\/video\/etc).<\/p>\n

Teacher assigns creation of reference (journal\/wiki\/etc).<\/p>\n

Student reviews textbook<\/p>\n

Student reviews video tutorial<\/p>\n

Student flips back in notes\/handout<\/p>\n

Student reviews student created reference<\/p>\n

Students create connections in student created reference<\/p>\n

Student uses website<\/p>\n

Student suggests or brings in resource<\/p>\n

Student asks metacognition\/reflection questions<\/p>\n

Student connects problem to out of classroom experience.<\/p>\n

Student demonstrates intrinsic motivation to learn material<\/p>\n

Student creates a sub-problem for themselves<\/p>\n

Student reorganizes or re-represents the problem (ex: draws a diagram)<\/p>\n

Student categorizes problem by underlying mathematical principle<\/p>\n

Student makes their own observations beyond what is planned by the teacher<\/p>\n

Student makes their own problems beyond what is planned by the teacher<\/p>\n

Student proposes an experiment<\/p>\n

 <\/p>\n

Initial axial coding and first trial coding<\/strong><\/p>\n

 <\/p>\n

Our initial organization of the data involved classifying the open codes into major categories based on who the actors were, and sub-categories based on the type of interaction:<\/p>\n

Student-Teacher<\/p>\n

Teacher identifies resources<\/p>\n

Student requests instruction<\/p>\n

Student-Student<\/p>\n

Teacher opens collaboration<\/p>\n

Students share observations<\/p>\n

Students ask questions of each other<\/p>\n

Students observe each other<\/p>\n

Students (build on\/challenge) each other’s comments<\/p>\n

Student expresses uncertainty to another<\/p>\n

Student-Reference<\/p>\n

Student uses external reference<\/p>\n

Student-Internal<\/p>\n

 <\/p>\n

This initial coding scheme was used to code classroom video from the Trends in International Mathematics and Science Study (TIMSS) website. The videos were coded using a form designed in the iCoda iPad app. iCoda is software tool for field observation in which a form consists of \u201cbuttons.\u201d Whenever an event corresponding to a code is observed to start, a button is pressed to activate it, and then pressed again to deactivate the button. iCode records the start time, end time, and title for all buttons pressed during an observation. Early versions of our form no longer exist, but a sample later version can be seen in Figure 6. The resulting data was studied using analysis software written in-house by the authors.<\/p>\n

\"\"<\/p>\n

Figure 1: Example code histogram of code frequencies (counts)<\/p>\n

\"\"<\/p>\n

Figure 2: Sample timeline showing the time of occurrence of each code during the class video. Vertical axis is code (in the same order as the histogram in figure 1). Horizontal axis is time.<\/p>\n

 <\/p>\n

Second axial coding and first trial coding<\/strong><\/p>\n

\u00a0<\/strong><\/p>\n

During this design cycle and discussion, a second way of organizing the codes emerged, which led to a second axial coding of the codes. We decided that scoring a class based on student actions wasn\u2019t reasonable in situations where the instructor didn\u2019t give the students an opportunity to act. For example, in the timeline above (Figure 1), you can see a long period of time at the beginning of the class when the teacher is introducing the problem (The leftmost black bar representing \u201cstudent-teacher\u201d interaction).<\/p>\n

In this second organizing scheme, we placed the observed behaviors in two dimensions. The dimension was our original top-level category of the actors, while the second dimension was based what kinds of opportunities the students had to use resources, and whether or not they took those opportunities. The new axial coding scheme looked something like (Figure 3).<\/p>\n

 <\/p>\n

\"\"<\/p>\n

Figure 3: Table of results for two-dimensional axial coding<\/p>\n

\"\"<\/p>\n

Figure 4: Count histogram under two-dimensional axial coding scheme<\/p>\n

\"\"<\/p>\n

 <\/p>\n

Figure 5: Example timeline under two-dimensional axial coding scheme.<\/p>\n

 <\/p>\n

The Second Design Cycle<\/strong><\/p>\n

 <\/p>\n

After the initial design cycle, we found that there was a dissonance between focusing on pairs of actors in one dimension, and opportunities in the second dimension because opportunities involved only a single actor. We redesigned the codes to combine these two categories into a single resources dimension.<\/p>\n

Teacher is resource<\/p>\n

Teacher points to resource<\/p>\n

Student uses teacher<\/p>\n

Student uses student<\/p>\n

Student uses object<\/p>\n

 <\/p>\n

However, we still wanted to make a commentary on quality of content, so we added a new second dimension based on Depth of Knowledge (DoK) levels in an attempt to track when students were relying on the teacher for simple recall situations or when their need required more analysis or deeper thinking.\u00a0 The resulting iCoda form can be seen in Figure 6.<\/p>\n

\"\"<\/p>\n

Figure 6: iCoda form for cycle 2.\u00a0 Each button corresponds to a code. The observer presses a button to activate the code, and again to release the button, and iCoda records start and end times.<\/p>\n

\"\"<\/p>\n

Figure 7. Comparison of count and duration histograms for the same class, showing very different pictures of the classroom.<\/p>\n

 <\/p>\n

We began to observe a large number of overlapping event instances; this meant that our initial approach of counting code instances was not viable. Because each button only had an on-off state, overlapping instances would be recorded only as a single event. For example students asking for help often overlapped with each other, leading to a misleadingly low count of student\u2019s asking for help.\u00a0 We compensated for this by making a move to tracking the duration of a code rather than counting instances (Figure 7). This showed us a very different picture of the classroom, where the students spend a lot of time waiting for the teacher to come help them.<\/p>\n

 <\/p>\n

A second problem was the interface of the redesigned form. Decisions identifying actors could be made very quickly, (as soon as the actor began speaking), while decisions about depth of knowledge required more insight into the content. This led to a time lag where the observer would often know which code to choose in the resources dimension long before they knew which DoK level was appropriate to code for.\u00a0 After this iteration and the challenges presented by the additional coding of DoK, the decision was made to remove this feature from the coding.<\/p>\n

\"\"<\/p>\n

Figure 8: Redesigned iCoda form separating the two dimensions to account for time lag.<\/p>\n

 <\/p>\n

The Third Design Cycle<\/strong><\/p>\n

 <\/p>\n

At this point, our codes became relatively fixed. The third design cycle was primarily about negotiating the meanings of individual codes. Due to the limited number of TIMSS videos, we moved to coding classroom video from two high school algebra classes in Arizona.<\/p>\n

 <\/p>\n

\"\"<\/p>\n

Figure 9: Sample inter-rater reliability analysis for code \u201cStudent uses Student.\u201d Blue bars are the codes by the two raters. Black represents times where raters agree there is no code. Green represents times when raters agree there is a code. Orange and magenta represent times of disagreement when only one rater coded. This particular example shows 49.082% agreement between coders.<\/p>\n

 <\/p>\n

Initial inter-rater reliability tests with the form had low ratings (Figure 9). We returned to the video to explore times when the coders disagreed and began negotiating the meanings of codes.<\/p>\n

 <\/p>\n

Teacher is resource:<\/p>\n

The primary disagreement here was whether teacher is resource was only lecture, or whether a teacher responding to a question from a student counted as teacher is resource. Essentially the disagreement was about whether teacher as resource must always be teacher initiated. We decided for the later option: That any time a teacher acted as a source of information, the teacher was acting as a resource, regardless of who initiated it.<\/p>\n

 <\/p>\n

Teacher points to resource:<\/p>\n

The primary disagreement here was whether assigning an activity that required a tool (such as a calculator) counted as teacher points to resource, because students were expected to use (and be introduced to) the tool. Unlike the case above, we decided that teacher points to resource needs to be defined as the teacher suggesting the use of a tool that was not part of a lesson.\u00a0 For example, if a lesson is set up in such as way that the student is expected to use a graphing calculator as a means to solving a problem, we would not code this as \u201cteacher points to resource.\u201d\u00a0 We would, however, code the use of a graphing calculator as \u201cteacher points to resource\u201d if the calculator was not part of the lesson plan.<\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

Fourth Design Cycle<\/strong><\/p>\n

 <\/p>\n

In our third trial, we tested both the electronic form and a new paper form (Figure 10). The paper form was used as a checklist. During observations, the observer kept a timer, and every minute would check the boxes corresponding to any code events during that minute. The paper form also had a section for the observer to write notes on their observations.<\/p>\n

\"\"<\/p>\n

Figure 10. Sample paper form, showing timeline and accompanying notes.<\/p>\n

 <\/p>\n

 <\/p>\n

The team universally preferred the paper form for three reasons. First, coding minute by minute instead of second by second gave the team more time to think and analyze while coding. Second, the paper form allowed for the creation of a timeline without specialized software; and third, the team liked the opportunity to add notes to the timelines. These notes served as both explanations for an observer’s\u2019 particular choice of code, and reminders to the observer of what happened for future discussion. The paper form became the sole tool used to gather date during the final inter-rater reliability run.<\/p>\n","protected":false},"excerpt":{"rendered":"

The First Design Cycle   Initial design of the observation protocol began by watching videos of high school mathematics classrooms that had high amount of in-class student work. As we watched the video, we began to record student behaviors that we found relevant to the question of measuring students\u2019 use of resources to resolve a … Read more<\/a><\/p>\n","protected":false},"author":8580,"featured_media":0,"parent":35494,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"wpo365_audiences":[],"wpo365_private":false,"footnotes":""},"class_list":["post-458","page","type-page","status-publish"],"_links":{"self":[{"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/pages\/458"}],"collection":[{"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/users\/8580"}],"replies":[{"embeddable":true,"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/comments?post=458"}],"version-history":[{"count":1,"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/pages\/458\/revisions"}],"predecessor-version":[{"id":35612,"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/pages\/458\/revisions\/35612"}],"up":[{"embeddable":true,"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/pages\/35494"}],"wp:attachment":[{"href":"https:\/\/test-inside.ewu.edu\/successfultransitions\/wp-json\/wp\/v2\/media?parent=458"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}