I haven't done any inter-rater reliability work in walkthroughs. I think having another administrator do a walk-through can be helpful to get another set of eyes if the teacher is struggling and I have been coaching the teacher through those issues. However if the administrator is using walkthrough forms to check off things they see from a once- or twice- yearly walkthrough without really knowing the teacher's practice, it's not that effective. 

This is not something I have any experience with. I am not sure how I feel about other administrators coming in and evaluating my staff based on no other knowledge of them. It could be really helpful if a set of fresh eyes sees something that you just aren't seeing, or it could be really detrimental if they see something out of context. 

I don't think I'm down for it having any rating scale, the walkthrough having any rating scale or scoring system generally. So again, I'm hoping that the walkthroughs that I provide are very non-judgmental, very concrete. This is exactly what I saw and these are the questions that arise based simply on what I saw. Therefore. It can create a conversation reflective conversation. Hopefully about teaching practices and they can fill in the gaps that
I've missed obviously, when we have discussions, or we respond to each other, an email when a teacher might answer the questions that I've asked just based on the limited bit that I did see. While I wouldn't want it scored, I think certainly it would be interesting to see that whether or not my colleagues other principal saw similar patterns of instruction in a teacher's room and maybe could share some insight or some thoughts based on their walk through that experience.
It's I think it's always good to share practice in that way. So I don't think that would be bad. But certainly creating a scoring or a rubric around a walkthrough that could maybe be used for a more formal evaluation. But the walkthroughs the way we use them and the way I've always understood them. They're not evaluative generally speaking. Obviously. If you see something very egregious you would step in and create a system to support the teacher and support the students around that issue or need.
But it's something that really in my estimation and my hope is that it would create a process for deeper reflective practice and dialogue around teaching, and learning practices and what's best for students and how students are engaging with the curriculum, and the teaching, and instructional practices in our programs, and in our schools.

We do have inter-rater reliability in our formal observation process. I don't think it's fair to judge a teacher based on a 5-10 minute snapshot of their class. I also don't think a walk through should be an evaluative process. It should be a growth process and not something stressful for the teaching staff. 

Yes, just finished an IRR training. There's was so much disagreement, it was insane. As far as others scoring teachers in my building, I feel that it's superficial, although, at times there are clear indicators of best practices vs wasted time/lazy teaching. 

I would not be comfortable with other leaders evaluating my teachers based on a walkthrough. There is so much more to their performance than can be seen in a snapshot. 

I haven't ever used this as we are a one school charter, but I have been to trainings where this was presented. 

I have not had the opportunity to do an inner reliability exercise in a different district that I worked in. We did do walkthroughs together as teams with upper administration, upper leadership. That felt very kind of yucky and hammer down. I think one of the things that we are working with in our school, we've been a k eight, but we've had two separate campuses and we brought two campuses together with two administrative teams. And I think it's tricky. And one of the things I struggle with is having the other administration team give feedback to the teachers, and I've given feedback to the teachers, and then the teachers are getting so much feedback that they don't know who to listen to or what to do, and it can get kind of clunky. And so I think walkthroughs together and just seeing what teachers are doing is great. And then having those conversations to give feedback that would be valuable or conversations that would be valuable based on what was seen within that walkthrough, I think would be a better practice than everyone giving different feedback. However, on the same token, it's good to see different things and someone could see something that I didn't see. And so I think that's very valuable and helpful as well.

For our observation model we have a calibration training for the formal observations. For a walkthrough there is not enough time to see everything that you need to in order to be calibrated 

This was a interesting topic with the inter-rater reliability. The first thing that I think of is in Arkansas, whenever I was getting my credentialing. In order to be able to evaluate teachers. We had to go through the Danielson framework. So we had to watch videos and and ultimately you were looking for two green check marks, so that said that you could reasonably evaluate teachers according to the Danielson framework, but that's not what it's like when you go into the classroom. So again, the relational aspect of being in my Teacher's classroom so often their kids know that I'm going to walk in at any time, but the the comfort level of the teacher knowing that I've been there before and it's not just going to be that five minute, 10 minute block of time that they're being evaluated on because over the course of a year. I'll be in the room 15 times. So 15 observations is much better than in the one or two and then to make the snap judgment based on what could be a terrible day for the teacher.

But again, the the TESS evaluation model, that's kind of what it's tailored on that. You have the four different areas that you get to rate the teacher. And if that's the only time that you are in their classroom, you're really basing that that decision of whatever your rating is based on what could be their best day of teaching, or what could be the worst day of teaching. I don't like the dog and pony show that that fosters.

So, as I was getting into administration and going into classrooms, I had other principals talk about things that you should do whenever you go into the classroom walkthrough and you want to give them a positive, you want to give them something to work on and then you want to give them that positive feel-good vibe before you left as well. What I will say. say that I do in my walkthroughs using Dr. Baeder's material, is I will go in, I'll have probably a set question that I'm honing in on with all of my teachers at the the time that I'm going through that specific stage of the walkthrough. But then if I see something interesting of note that I just want to know more about, I'll ask that kind of question.
Very rarely do I give them the complete negative, "you really need to think about changing this particular dynamic of the class." If I pick up on something like that. I simply ask them, "Explain that to me. What were you? You know, what was your goal? Whenever you had structured, the group this way or what," whatever it may be, but I want them to have that reflective practice of explaining what it was that, you know, they might be able to figure it out on their own, that there's there's a better way to do that.

But again, it's not the the the positive thing backed up by the thing you could change buttered up with another positive. So the relational aspect of being in the classroom so much is the teachers know that whenever you have that concern and you share that concern, you can have a real conversation about it.

Every year in NYC DOE, principals are required to go through this calibration process. A brave principal volunteers her/his school and a group of principals do walk-throughs, rate the visited teachers, and discuss the ratings. Every time I have participate in a group that included the host principal, the host principal spends a lot time defending the teachers and explaining details that the rest of us could not have possibly known due to lack of relationship with the teachers and lack of solid knowledge about the school. Thanks for calling out this myth. 

The notion of feedback and inter-rater reliability often go hand in hand. I have participated in group walkthroughs in which we would swarm a classroom with our notebooks or iPads and make our diligent notes before swarming to the next classroom. Often we would spend up to an hour or more admin bombing classrooms only to take maybe 30 minutes afterward to actually "calibrate".

Every couple of years in Ohio you are required to re-calibrate for our state teacher evaluation system (OTES). I've always wondered how reliable it is. The re-calibration is based off limited information. The scenario would be similar to an outside person evaluating a staff member they don't know. It's hard to give an accurate rating of an educator. 

The terms "interrater reliability" and "calibrating" are familiar to me. These are particularly noted when using the teacher evaluation framework in Texas, which is called T-TESS (Texas Teacher Evaluation and Support System), closely related to Charlotte Danielson's framework. Part of the annual (or sometimes every three years) certification process as an T-TESS evaluator is to watch a full lesson (online) and fully score two domains. Our score is then compared to the reference score done by NEIS (?). Our score must be within one proficiency level of the reference score in each dimension in the two domains in order to be considered for (re)certification. So, in that sense, the idea of being "calibrated" to the instrument is important, because it generates a rating for the teacher.

Dr. Baeder I think that the charming, the beauty of the Walkthrough program you're developing is the idea for the leaders to build a relationship with the teachers. I really like one of the figures you have with the iceberg where the leaders sometimes only see the peak of the iceberg and see nothing below the water. So the holistic part of the evaluation thinking for me is very essential. And how you develop a connection with the teachers are also very important so you understand what the teachers are teaching, how they're assessing and how they are connecting with the kids. Because being there only in front of the kids and teaching in terms of lecturing without engagement, it doesn't mean that much. I think the leaders should supposed to build their relationship and provide authentic feedback. And the only way to do so authentically is to build their relationship and understand what the teachers are saying, what the teachers are doing, and then they're talking and asking questions are very important to understand the whole picture.

I haven't done IRR, and I would think that having someone who doesn't know the teacher could add to the stress level of that teacher being evaluated. It would seem that this person would be looking for a checklist of items that may indicate effective teaching, but may not put the whole picture together for the teacher's actual effectiveness. 

I haven't ever used the IRR myself nor has it been used by me. I know data is important, but I'd rather the data be informative and based on specific actions without any sort of conversion to a number or letter. I'd rather be more similar to standards-based grading with lots of various feedback opportunities and methods. 

I have never done any interreliability scoring of any sort up to this time, and as far as an outsider coming in to do walkthroughs, I do think it would be difficult if they have no understanding of the context or understanding of the team teachers or what the whole premise of kind of the place and space where the school or teaching is headed.

The observation tool that we use has an annual requirement of an inter rating reliability training each summer.It can be challenging but as professionals we all "align" in the way we evaluate. 

STRONGE+ evaluation system teaches inter-rater reliability; it makes sense logically, but I see your point about what it is based on. 

If the focus is on improvement, then we have to be careful with the amount of emphasis we place on scores. Additionally, I believe that a focus on common language, central focus, and frequent dialogue about teaching and learning is more important. 

I have not had to do an inter-related evaluation. I do agree that walkthroughs do no give you a big overview of your teachers. With continuous and fidelity of being in classrooms will help build that big picture we are trying to see by the end of the year. 

This happens all the time. We have calibration walkthroughs with academic coaches and different leaders of the district office. I have a hard time with this aspect of instructional leadership. I feel it is not a true picture of our teachers to base their performance on one 5 or 10 minute visit. Often times the team is very critical of the teachers. It is my opinion that is very important to build relationships with teachers and calibration is not a necessary tool for improving teacher practice or student outcomes. It often feels like a "gotcha" instead of "here to give you support". 

I have done inter-rater calibration training for teacher evaluations. It requires a lot of work to truly calibrate with another person. It would be weird for a leader who does not know a teacher to grade them after a brief walkthrough. 

Our walkthroughs are non-evaluative (per the teacher union agreement). I do two formal evaluations each year and informal evaluations as needed. The district switched to The Rise rubric for staff evaluations. The principal team worked together with district leaders to align evaluative practices. 

To gather data for an accreditation process I went through a calibration exercise to ensure inter-rater reliability. I had to do a few of the exercises repeatedly to ensure that there was complete alignment, especially given that the margin was very specific. While I also agree that there should be alignment, the document to be used did not capture many aspects of the teaching and learning that took place in the class.

Having others come into our classrooms, who had little awareness of our teachers, their styles, our school's culture and context is a bit of a mix-bag. While they can provide valuable information from the lens through which they observe. Their limited one off observation will not give a true reflection of the teachers' skills and capacity. Teaching happens everyday, and so observations and evaluations should occur often enough to observe the true trends and patterns of teaching for the teacher. 

I have used forms in the past to capture and to send feedback. I just found that it didn't prompt a lot of discussion and a lot of the concepts of what was happening within a classroom. What's lost teachers didn't have the opportunity to share what was happening and their rationale behind the moves that they were making with the use of a form. And I even in practice, though, my supervisor know, that's one of the ways that we show that we go, that we are visiting classrooms by capturing these forms, but I agree. It doesn't seem to lead to student achievement by itself. Discussions made so that practice can be pushed and instructional practice can be improved.
So, that's my take on feedback forms. What I've been doing currently is meeting.
Offering feedback and then setting up the time to meet to discuss those feedback notes.

We have done some inter-rater reliability exercises with video and with teacher artifacts. I find them problematic in that the goal becomes focused on the group doing the rating to reach agreement, rather than what the whole purpose should be which is a learner-focused conversation with the teacher.So the "raters" leave feeling successful that they can "rate appropriately," but they have not grown at all in their capacity to have a quality conversation. 

I have participated in group walkthroughs with our content area supervisors - the conversations afterwards were twofold - one goal was for the content lead to share with admin the instructional focus that we should be looking for; the second goal was for us to develop inter-rater reliability so that we could be more consistent. I was often uncomfortable with the discussion - as there were many times that the observers came away with fundamentally different notes and feedback points, despite being in the same class/time period. We also didn't end up with any great "common" points - we would have been much better off discussing language around our instructional framework and what evidence for those standards looks like. 

1.) Our state (OH) mandates inter-rater reliability in order to attain our credential to evaluate teachers. We watch a 30-45 minute lesson and must have our scoring close enough to the "right" answer in order to be credentialed. We also talk about this within our district and at the building level (i.e. who is the "easy" evaluator? What does this word mean? Where would you put this piece of evidence?)

2.) I experienced an alternative evaluator with one of my struggling teachers last year. The bargained evaluation process allows the teachers to request an alternative evaluator for the final observation. It was difficult because her classroom performance, although not great, wasn't actually the concern. The concerns were professionalism related: paperwork, deadlines, compliance, etc. Any observation, even longer than a typical walkthrough, would not have been representative of those concerns. 

Yes, I have done calibrator exercises. I found that the activity depended on individual's prior knowledge and experiences. 

So in a previous District, we actually worked on inter-rater reliability in that we for formal evaluations. We would have to, if at all possible, two qualified evaluators in the room to script and then to score together and that really helped I think for teachers to feel like it was a little more objective. And there was some you know, safety in knowing that they had two different people with eyes on the same lesson and having conversation so that if somebody didn't have, I don't know, they could just balance each other out as far as what they had seen holistically with that instructor. Now for drop-ins however, again with the forms that we were filling out. There were specific metrics that they wanted to track, but I didn't appreciate as an instructional coach, that it gave you a full picture. If you were writing a teacher on student engagement or level of questioning, if you were in for 5-10 minutes of a lesson where those things didn't apply. Because if you didn't observe it you shouldn't be scoring it and yet they were required scores on the feedback form. So I definitely think there's value in having more than one person's eyes on it. So it helps it not be so subjective. But it also is a very subjective practice.

I have done alignment and calibration excercises when being trained on how to evaluate teachers based on our selected evaluation model. I don't feel entirely comfortable with other leaders evaluating the teachers that I directly serve and I don't feel comfortable evaluating other teachers that I don't directly serve as teachers and teaching are very nuanced, therefore require a greater scope of knowledge by the evaluator. 

I have not looked at a "complete picture", only a snap shot. I realize that more goes on during the day that I do not know about when walking through. I think it is good for "outside" leaders to go into classrooms to see what is happening, but I know that I know my teachers better. However, an outside opinion is not always a bad thing. I think they can offer insight and opinions that we might not see because of our relationship to the teacher. 

Yes, I have been involved in various inter-rater reliabilty exercises, we do a yearly calibration with our observation model along with inter-rater observations of teachers throughout the year.As far as my feelings on other people observing my staff and basing their observation and judgement on that small amount of time, I am not a fan. I think while it is okay for one observation to be done by someone not from your building the majority of the observations should be done by the administrators that know the teachers best. The idea of the observation system is to help everyone grow therefore the whole teacher and year needs to be included. 

We don't do this - teacher evaluations are based on multiple visits by multiple people over time and include discussions with the teachers about their practice, a review of unit and lesson plans, assessment tools and other materials that they provide as part of a portfolio of evidence. Multiple people work on the evaluation document to provide their insight and the teacher also has an opportunity to review the document and put in their own feedback. It is meant to be a conversation and meant to be about professional growth. We do use a modification of Danielson's framework to help us decide if what we are observing is "good enough" given where the teacher is in their practice but we don't score teachers. 

I have not performed inter-rater reliability exercises nor have I done calibration exercises- will be interested to learn more. I don't appreciate a one time visit and teacher evaluation but I myself am here taking this course, to improve on this. It is much needed and especially during these times of a pandemic. I am invested in taking more time to work with teachers and on classroom walkthroughs. 

Inter-rater reliability. We really haven't practiced that here in this District. We have gotten together as a team to look at teacher practice and maybe watch a video from teacher channel of a teacher introducing a lesson and maybe do a couple of those. And people sort of fill out the forms that we have. And then we would talk and compared as a way of sort of building our observational skills and to see what other administrators saw and certain clip. So those are some of the things that we have done, but never any kind of have another administrator come into your building and you do that. I think teachers would feel that was beautiful, really odd here, and this smaller District.

We've used calibration techniques to score writing papers and things like that. We have not calibrated as administrators doing walkthroughs. It just hit me that perhaps we should be having debriefing meetings to analyze what we've been seeing in our walkthroughs. Are there trends?I don't believe other leaders should evaluate based on one walkthrough. Everyone has a bad day and it's not fair to that teacher to have an outsider evaluate them based on a 5 minute walkthrough. If they were doing this consistently, I think it could be helpful. 

No two sites are the same, so alignment or calibration would be a futile activity. The goals you have for your teachers and those they have for themselves are likely to be very different from site to site, 

While I have never been part of "aligning" or "calibrating" teacher evaluation, I admit that it is something that I have requested. My building principal and I split the responsibility of evaluating 50+ teachers. We "evaluate" and "rate" teachers very differently - which is confusing to many teachers. In four years of working together as an admin team, we have decided to switch our evaluation roster every two years so that teachers receive feedback from both of us. I feel like this is an area where we could do better!I would not be very pleased if a guest administrator came into my building and rated teachers based upon a brief walkthrough. Especially if they were pointing out areas for improvement for teachers that they do not know - I imagine that I would be very "mama bear" in that situation. Likewise, I would not feel comfortable doing a brief walkthrough in another building and be expected to rate teachers whom I don't know and had only observed briefly. 

I was part of a large school district and we did spend a good chunk of time working on inter-rater reliability, especially, you know, when trying to do long-term evaluations, I think it was helpful to hear other people's perspectives. But as you said, we visited schools, we all went together. Two teams and not all of us, definitely did not have any background information or you know information about the teacher, we were seeing, but they did choose some of the best teachers in the district to visit in terms of having people come in and rate my own teachers, just sort of, you know, on a short-term basis like that. I think that they would miss things. Also, I feel like because of the
Social nature of our school. I would want them to know about our school to be able to know something about online, learning to be able to see face to face, and to know our SEL characteristics to do a better job with with you know, evaluating my teachers.

You know, I have been the participant and recipient of those that have used inter-rater reliability and the inter-rater reliability. And yeah, usually it's connected to classroom walkthrough data. Here's the thing. Usually that has been spearheaded by someone who's very much connected to, maybe they've done their doctorate, in quantitative research using quantitative methods or they are a math person but as one who really believes in a mixed methods approach and more so a qualitative approach as well. There's so much that you miss. When you use just an innovator reliable ability for evaluation with a professional. There's so much more that is connected.
To how we perform. Then just a one snapshot moment. I really appreciate it. Doctor Baeder's connection of this to American Idol, or to America's Got Talent. Any of those judge panel shows because that's such a disservice to professionals, I felt that as a music teacher, I felt that as a musician and you know, ultimately the
forum of public opinion is not where teacher evaluation and really professional evaluation should live. It has to be far, more deeper far, more nuanced, far more complex.

I have never used the calibration exercise or the alignment exercise. I like to get into the classroom and see just what the teachers are doing and everybody, every teacher is different. If you have a veteran teacher who's been around a school for a while, they know my expectations from having that dialogue back and forth with one another. And so just remember never use that. And I don't think that somebody coming off the street and comes in and evaluate your teacher walkthrough to doesn't do that teacher justice.  I think a lot of times when you pop in, do you get to see a little snippet, but you don't get to see the whole class or the whole lesson and I think it's more frequency getting in there. So you can actually see what teachers are doing legitimately not just a 1 or 2 time deal.

Yes, we used Inter-Rater Reliability to score teachers several years ago. If the goal is to improve teachers' instructional practices and have authentic feedback conversations, all leaders must get to know their teachers, students, and the curriculum they teach. A walkthrough is just one data point. As leaders, we should use multiple data points to inform our decisions. 

In my current school and with our admin team. Our feedback is not really about arriving at a score. We have, you know, tried to create a form, the form that we were talking about in the last little bit where we get to, you know, some things we saw, some things that we liked some things that we wondered about those kinds of things. But we've definitely had conversations and there's been some people around the table that have wanted to get at that idea of calibration.

Are we, you know, we going in, are we looking for the same things? I can really relate to this something that occurred to me and is kind of occurred to me more recently in, you know, I would say the last semester of walkthroughs and even formal teacher evaluations that I'm doing is that with different teachers, there's going to be different threads that you're sort of working with them that you're pulling out with them and trying to, you know, kind of get at the heart of some matters. So for example, I working with the teacher, I just did her formal evaluation before the winter break and you know, it's a continuation of conversations we were even having last year about, you know, working with deadlines and managing, you know classroom time and those kinds of things with their students. So I'm able to draw on those experiences that I've had with her and those previous conversations, things that I've seen in other classes things that you know, I might see another areas as well and bring it together, but that totally wouldn't make sense for anybody else because that's a personal conversation within the professional relationship. I've developed with that teacher.

So the idea of calibration I get it, we want to be consistent and maybe we can calibrate other things. But for the nature and the quality of the feedback, I think we need to embrace the idea that it's going to be all different as it should be for our different professional relationships.

As a teacher who just recently left the classroom and a stepping into the role of leadership, the most nerve-wracking thing in the world, is someone coming into my classroom, who doesn't understand me, doesn't understand my kids doesn't understand, you know, the different gaps that my students have doesn't understand the parent student relationships, right? Like, there's a lot of variables there that don't come into play. When someone's coming into my room, who doesn't know these things and yet,

You are evaluating my performance. My kids performance, without any background knowledge, it didn't really seem fair to me, when people would come. Even if I knew ahead of time, right? I never felt 5 to 10 minutes is enough because you never know when the person is coming in or even if they say they're coming in at a certain time, things can happen. So for me it was always just very nerve-wracking and I didn't feel like it ever gave the whole picture as to not only all the things I did well but even some things that I could work on. So for me, I never really was a fan of the five to ten minute window for evaluations.

When I've been in systems that use scoring for evaluations, it has only been as a compilation of data collected from multiple pieces of evidence over the course of the year. Our team would calibrate to check our consistency in "scoring" the rubric with some examples. When we have had certain look fors such as school-wide instructional strategies that all faculty members committed to using, we would emphasize that you're not going to see all elements in a walkthrough and some were more likely to be demonstrated depending on whether it was the beginning, middle, or end of an instructional block. The most helpful aspect of this was helping me see my own patterns to better balance conducting interviews at different times of the day and different phases of the lesson for shorter walkthrough visits. 

I've never seen this in practice, the inter-rater reliability. It's not my districts practice to have outsiders or people from the DEO to come, do walkthroughs and then rate our teachers. So I don't have any personal experience with this.