Download Assessment of Student Learning in IN250 U.S. Studies at Millikin University (2007-08) - Pr and more Exams Sociology of Religion in PDF only on Docsity!
Student Learning in IN250 U.S. Studies Academic Year 2007-08 Assessment Millikin University IN250 Coordinator Larry Troy May 20, 2008 Executive Summary The 2007-08 academic year was again a time of transition. With a new Coordinator, the methodology used in assessing student learning was refined, and an interim report based on just the fall semester was developed to smooth the transition between the two plans. A copy of that report is appended to this document (Appendix C). US Studies faculty were also conscious of the changes to the MPSL that will begin in the 2008-2009 academic year. However, these changes will not have meaningful consequences for US Studies until the 2009-2010 academic year, because incoming first-year students will take US Studies classes in their sophomore year. Nevertheless, pilot IN251 classes will be offered during the 2008- academic year to ensure that transfer students will be able to satisfy their MPSL requirements. Assessment of student learning in IN250 for the 2007-08 academic year demonstrates that learning goals are being satisfied and that IN250 is a strong component of the sequential program of the MPSL. Goals The IN250 learning goals are: By the end of IN250 U.S. Studies, students will be able to
- Critically respond to texts by or about some of the diverse cultures and people living in the United States;
- Use discipline-appropriate sources to inform their critical analyses of diverse individuals’ or cultures’ experiences and perspectives; and
- Write at least one well-developed critical analysis about cultural, structural, or institutional issues specific to the discipline of the course. Snapshot During the 2007-08 academic year, 25 sections of IN 250 were offered by 17 faculty members (see Appendix A): 12 sections by eight History faculty—three full-time tenured or tenure-track faculty; five adjunct faculty without terminal degree 5 sections by three Political Science faculty—two full-time tenure-track; one adjunct faculty 2 sections by one Sociology faculty—full-time, tenured 1 section by one Communication faculty—full-time, tenured 2 by one Theatre faculty—full-time, tenure track 1 by one Exercise Science faculty—full-time, tenured 2 by two English faculty—one full-time tenure-track, one full-time without terminal degree The Learning Story In all sections of IN250, students learn about the experiences and perspectives of some of the diverse cultures and people living in the United States. Courses may focus on particular groups (African Americans, women) or particular issues (poverty
and welfare, Vietnam, Civil Rights, Watergate), but all faculty are committed to inculcating in their students sensitivity and empathy, as well as sharper critical thinking and writing skills. In all sections, students use discipline-specific sources in their writing, which ranges from the formal essay to informal discussion threads in Blackboard. Competencies that promote professional success and understanding of others are emphasized (as per MPSL prepares 1 and 2). Assessment Methods and Data The IN250 Coordinator collected artifacts at the end of the Fall 2006 and Spring 2007 semesters. The IN250 self-study committees (Monroe and Troy in the Fall, and Monroe, McKenna, and Troy in the Spring) randomly selected 5 artifacts each from 6 (out of 8 possible) of the Fall IN250 classes and 3 artifacts each from 11 (out of 13 possible) of the Spring IN250 classes. The artifacts included term papers, essay exams, and assignments. The committees used a single rubric (see Appendix B) to assess all artifacts on a 1-5 scale (1 indicating Nominal, 5 indicating Excellent) on each of the three goals. They then calculated average scores for each learning goal, for all artifacts in total and for each specific class. The following table shows the range of possible scores and interpretation. Green Excellent 3.67-5. Yellow Adequate 2.34-3. Red Nominal 1.00-2. One difference between the Fall and Spring semesters concerns how the committees interpreted the learning goals. The Fall committee noted uncertainty about how the three learning goals should be interpreted. On the first learning goal, for example, the members of the committee were uncertain whether they were to assess how well the student critically responded to texts or how well the student showed an understanding of diverse cultures. Similarly, on the third goal, they were unclear about whether they were to assess a “well-developed critical analysis” or the coverage of “cultural, structural, or institutional issues.” Based on this confusion, the members of the Spring committee decided: On the first learning goal, they would emphasize how well students showed understandings of “diverse cultures and people living in the United States.” On the second learning goal, they would emphasize how well students used “discipline-appropriate sources.” On the third learning goal, they would emphasize how well-written the students’ critical analyses were.
Comparison of AY2006-07 and AY2007-2008 Data Since the methodology used to assess student learning changed across these two years, the assessment scores are not strictly comparable. The IN250 Fall 2007 interim report attempted to recreate the 2006-07 methodology and apply it to the Fall 2007 classes. I attempted to do the same thing here for the entire academic year. Comparison of AY2006-07 and AY2007- % Of Artifacts Judged To Be At Least Adequate Critical Response to Diverse Cultures Use of Sources Critical Analysis 2006- Academic Year
Academic Year
Assuming that the 2006-2007 report used an arbitrary 60%, 70%, and 80% as the 3 cutoffs for Nominal, Adequate, and Excellent, respectively, student learning for the first learning goal, Critical Response, dropped from Excellent to Adequate. There was no change in the Use of Sources, the second learning goal, both scoring in the Excellent range. Finally, the scores rose from Nominal to Adequate on the third learning goal, Critical Analysis. This last finding might indicate greater attention to the quality of writing in IN250 classes. Therefore, regardless of the methodology, students in IN250 during the 2007- academic year met the learning goals established by the faculty. Initiatives and Improvement Plans This year, the faculty voted for a change in the curriculum that will have dramatic effects on IN250. Starting in the 2008-2009 year, two courses will be required in US Studies, rather one: IN250 will be US Cultural Studies, while IN251 will be US Structural Studies. These changes take effect with students entering the University in the 2008-2009 academic year. Since most students take these classes in their sophomore year (2009-2010), new learning goals will have to be developed for these two classes during 2008-2009. The members of the Spring 2008 Committee discussed the results of the Spring assessment, noting that the nature of the assignments from which the artifacts were derived seemed to affect how well student learning could be assessed. Term papers were better than exams, which were in turn better than individual assignments. The 2006-07 report noted the same thing. As the IN250 faculty meet during the 2008- 2009 academic year to discuss the learning goals for the new IN250 and IN classes, this point should be kept in mind. The fact that one of the new threads running through the sequential program will be that the class will be writing intensive should help with this issue.
Appendix A IN250 Courses, AY 2007- Date Course Title Cross-listed Department & Course number FT Tenured Or t-track, W/terminal Degree FT Contract Faculty w/ Terminal Degree FT Contract Faculty w/o Terminal Degree Adjunct Faculty Fall 2007 Criminal Law: 4,5,6th Amendments PO260 ● Civil Rights, Vietnam, & Watergate PO260 ● World Religions in America HI210 ● History of American Workers HI210 ● U.S. History To 1865 HI203 ● U.S. History To 1865 HI203 ● Drama & Representation of US TH331 ● Sport in America ● Spring 2008 Rhetoric-Violence in America CO260 ● Political Participation and Democratic Citizenship PO260 ● Group Influence in America (2 sections) PO260 ● Drama & Representation of US TH331 ● US History Since 1865 (2 sections) HI204 ● US History Since 1865 (2 sections) HI204 ● The Outsider in Film & Fiction EN220 ● Poverty and Welfare (2 sections) SO224 ● Women in America HI210 ● State and Local History HI210 ● Native American Myths and Folklore ● Christianity in the US HI210 ● World Religion in US HI210 ●
Appendix C IN Interim Assessment Results Fall 2007 March 17, 2008 This report outlines the result of an assessment made for IN250 classes taught in Fall
- This interim report is important because it allows us to track changes more closely than simply on a yearly basis, and because the method of analysis has changed from previous years to more accurately reflect a holistic conception of assessment. The Learning Goals The learning goals for IN250, as most recently revised in 2007 are: By the end of IN250 U.S. Studies, students will be able to:
- Critically respond to texts by or about some of the diverse cultures and people living in the United States;
- Use discipline-appropriate sources to inform their critical analyses of diverse individuals’ or cultures’ experiences and perspectives; and
- Write at least one well-developed critical analysis about cultural, structural, or institutional issues specific to the discipline of the course. Snapshot During the Fall 2007 semester, 8 sections of IN 250 were offered by 7 faculty members: 1 section by one Religion faculty--adjunct 3 sections by three History faculty—two full-time tenured or tenure-track; one adjunct 2 sections by one Political Science faculty—adjunct faculty 1 by one Theatre faculty—full-time tenure-track 1 by one Exercise Science faculty member—full-time tenured The Learning Story In all sections of IN250, students learn about the experiences and perspectives of some of the diverse cultures and people living in the United States. Courses may focus on particular groups (African Americans, women) or particular issues (poverty and welfare, Vietnam, Civil Rights, Watergate), but all faculty are committed to inculcating in their students sensitivity and empathy, as well as sharper critical thinking and writing skills. In all sections, students use discipline-specific sources in their writing, which ranges from the formal essay to informal discussion threads in Blackboard. Competencies that promote professional success and understanding of others are emphasized (as per MPSL prepares 1 and 2). Assessment Methods and Data All faculty teaching IN250 classes in Fall 2007, with the exception of those taught by the adjunct faculty members in Religion and History, submitted artifacts, including final exams, term papers, or written assignments, at the end of the Fall 2007 semester. The IN250 self-study committee randomly selected five artifacts from each of the six classes. The committee used a single rubric (see Appendix A) to assess all the artifacts, scoring each artifact on a 1-5 scale (1 indicating Nominal, 5 indicating Excellent) on each of the three goals. We then calculated average scores for each learning goal, for all artifacts in total and for each specific class. The following table shows the range of possible scores and interpretation.
Green Excellent 3.67-5. Yellow Adequate 2.34-3. Red Nominal 1.00-2. Analysis of Assessment Results Mean scores of raters for each learning goal for each class Fall 2007 Instructor Critical Response to Diverse Cultures Use of Sources Critical Analysis Dunn – Civil Rights 3.2 2.8 2. Dunn – Con Law 3.1 3.3 2. Landro - Theatre 4.3 4.2 4. Luy - Sport 2.2 2.0 2. Monroe – US Hist. 3.1 3.5 3. Murphy – US Hist. 2.9 2.9 2. TOTAL 3.14 3.10 2. Overall, the results show that all learning goals were met at an Adequate level in Fall 2007. One class showed a Nominal level, and one class showed an Excellent level, on all three learning goals, respectively. The question of how these compare to previous years’ results is a complex one, because the analytical strategy changed. In the 2006-2007 report, three faculty read 16 of the 205 artifacts submitted, and scored each one on a 0-5 scale. They totaled the scores from the three faculty for each learning goal, arriving at a score within a 0- 15 range. They then reported the percent of artifacts that had at least Adequate total scores (within a 6-10 range, or higher). To the extent possible, we have recreated their methodology to enable us to compare the results from Fall 2007 to those from the 2006-2007 year. The table below reports that comparison. Comparison of Fall 2007 with 2006-2007 Academic Year % Of Artifacts Judged To Be At Least Adequate Critical Response to Diverse Cultures Use of Sources Critical Analysis 2006- Academic Year
Fall 2007 83% 79% 68% We can make two conclusions from these results. First, they are broadly comparable across the two years, suggesting continuity. Second, assuming that the 2006- report used 60%, 70%, and 80% as the 3 cutoffs for Nominal, Adequate, and Excellent, respectively, we see continued Excellence for Critical Response and continued Nominal performance for Critical Analysis, while the scores for Use of Sources dropped from Excellent to Adequate. As we describe below, however, it makes greater methodological sense to use the new strategy, and conclude that students in IN250 classes performed at an adequate level on all three learning goals in Fall 2007.
