Study with the several resources on Docsity
Earn points by helping other students or get them with a premium plan
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community for help and clear up your study doubts
Discover the best universities in your country according to Docsity users
Free resources
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
An assessment framework for evaluating leadership skills in various areas, including presence, innovation, expertise, leading others, and building trust. It includes levels of improvement (Needs Improvement, Satisfactory, Excellent, Outstanding) for each skill and provides descriptions for each level. This document could be useful for military training programs, leadership development courses, or self-assessment for individuals.
Typology: Study notes
1 / 29
This page cannot be seen from the preview
Don't miss anything!
January 2018
United States Army Research Institute for the Behavioral and Social Sciences
Approved for public release; distribution is unlimited.
Research accomplished under contract for the Department of the Army by
Aptima, Inc.
Technical review by
Jayne Allen, U.S. Army Research Institute
DISTRIBUTION: This Research Product has been submitted to the Defense Information Technical Center (DTIC). Address correspondence concerning ARI reports to: U.S. Army Research Institute for the Behavioral and Social Sciences, Attn: DAPE- ARI-ZXM, 6000 6 th^ Street Building 1464 / Mail Stop: 5610), Fort Belvoir, VA 22060-
FINAL DISPOSITION: Destroy this Research Product when it is no longer needed. Do not return it to the U.S. Army Research Institute for the Behavioral and Social Sciences.
NOTE: The findings in this Research Product are not to be construed as an official Department of the Army position, unless so designated by other authorized documents.
ii
Fort Benning Research Unit Jennifer S. Tucker, Acting Chief
January 2018
Approved for public release; distribution is unlimited.
iii
We would like to thank all of the leadership and instructors at the Officer Candidate School (OCS) at Fort Benning for allowing us to conduct this research, providing valuable input and feedback, and serving as proponents of this work.
Development of a Behaviorally Anchored Rating Scale for Leadership
Introduction
U.S. Army leaders must be technically and tactically proficient and capable of leading units that are adaptive and resilient. Leaders who master the attributes and competencies outlined in the Army leadership requirements model are expected to think critically, solve problems, show initiative, and demonstrate character and accountability in their actions (see Figure 1, U.S. Department of the Army, 2013). Assessment of these attributes and competencies is an integral component of many leader development programs. Accurate assessment based on observable behaviors supports formative feedback and contributes to leader self-awareness, learning, and growth (U.S. Department of the Army, 2015).
Figure 1. The Army leadership requirements model (U.S. Department of the Army, 2013).
The mission of the U.S. Army’s Officer Candidate School (OCS) is to develop junior Officers who are capable of addressing future mission challenges and complexities (U.S. Department of the Army, 2014). OCS instructors are tasked with developing and evaluating the future leaders of the Army on a variety of tactical and technical skills, as well as the leadership attributes and competencies described in the Army leadership requirements model.
Several issues exist that make leadership assessment challenging in OCS. Currently, leadership ratings are made using a form that only contains a brief definition of each attribute and competency (see Table 1 for examples). Although these attributes and competencies are conceptually defined within Army doctrine (U.S. Department of the Army, 2012), operational or behavioral definitions have not been developed. As such, different instructors may interpret and evaluate each attribute and competency differently across OCS candidates and over time. For example, one instructor may believe a given behavior is indicative of needs improvement , whereas another instructor may consider the same behavior as satisfactory. Discrepancies stemming from a lack of standardization may lead to inconsistent performance expectations and difficulty in providing consistent formative feedback to OCS candidates. Leader assessments must be objective, consistent, and systematic in order to enable instructors to effectively capture data that allow for (a) a more holistic view of student performance and development; (b) an ability to better discriminate among proficiency levels; and (c) the
provision of more targeted, individualized feedback to boost each candidate’s learning experience.
Table 1
Example Leadership Rating Definitions
Core Categories
Sub- Categories Definition
Attributes
Character Loyalty Bears true faith and allegiance to the U.S. Constitution, the Army, the Unit and other Soldiers Presence Resilience Shows a tendency to recover quickly from setbacks, shock, injuries, adversity, and stress while maintaining a mission and organizational focus Intellect Sound Judgment
Assesses situations and people, and draws feasible conclusions; makes sensible and timely decisions
Competencies
Leads Builds Trust
Establishes conditions that foster a positive command climate Develops Develops Others
Encourages and supports others to grow and succeed as individuals and teams; facilitates the achievement of goals; makes the organization more versatile and productive Achieves Gets Results
Provides guidance and manages resources; ensures tasks are accomplished consistently, ethically, on time, and to standard through supervising, managing, monitoring, and controlling the work Note: Definitions from ADRP 6-22 (U.S. Department of the Army, 2012)
Assessment standardization and consistency can be enhanced with the help of appropriate support tools, such as behaviorally anchored rating scales (BARS). BARS consist of specific, observable behaviors (i.e., behavioral anchors) that exemplify critical performance dimensions or job relevant attributes or competencies at different proficiency levels relevant to the target context (Smith & Kendall, 1963). Many studies have pointed to the benefits of BARS since their inception and their use for performance assessment has become commonplace across a variety of work settings, especially where well-defined criteria are lacking (e.g., Hedge, Borman, Bruskiewicz, & Bourne, 2004; Jacobs, Kafry & Zedeck, 1980; Selvarajan & Cloninger, 2009). Behavioral anchors focus the attention of raters on what to look for when evaluating performance and guide their interpretation of the evidence in a manner befitting the standards and expectations of the training context. BARS are useful for creating a shared mental model about how certain performance dimensions can manifest behaviorally in the target context, thereby reducing ambiguity and increasing rater accuracy (Guion, 2011). Without this frame of reference, raters might be compelled to make a general judgment about an abstract construct, make an inferential leap, or base their evaluation on irrelevant factors. Priming raters to discern relevant observable behaviors and using them as a common reference point for their evaluation of performance results in less bias (e.g., fewer leniency and halo errors) and increases interrater
An iterative development process was used in which focus groups with OCS instructors were followed by content revision. Data Collection 1 was a three-day workshop with OCS instructors ( n = 4) to gather feedback on the current leadership evaluation forms and identify performance indicators for the leadership attributes and competencies currently assessed in OCS. On Day 1, the leadership assessment process was discussed with instructors, including how, when, and why the current leadership evaluation forms are used and specific challenges associated with how assessments are currently made. On Days 2 and 3, instructors were asked to provide examples of observable behavioral performance indicators for the leadership attributes and competencies. Specifically, instructors verbally described key elements of leadership and provided example behaviors (e.g., treats others fairly and with respect) they look for when assessing each leadership attribute/competency across relevant OCS training events. Once critical themes and elements were identified, observable behaviors were specified for each of the four performance levels/rating categories (i.e., needs improvement , satisfactory , excellent , and outstanding ). For example, instructors indicated that an OCS candidate who needs improvement in confidence may waver, second guess, not make a decision, or ask others to make a decision for him/her, whereas an outstanding candidate would make a sound decision in a timely manner and be able to articulate his/her reasoning while seeking feedback to refine the plan as needed. The workshop yielded a list of behavioral examples for nine leadership attributes (under presence and intellect) and 10 competencies (under leads, develops, and achieves). Due to time constraints, the attributes under character were not discussed in this workshop. Following Data Collection 1, iterative revision of the behavioral anchors were made to enhance the clarity, consistency, comprehensiveness, and observability of the behavioral statements across the rating scale categories.
Following these revisions, the measure was piloted with two OCS companies. The primary goal of this pilot was to identify an initial workflow and gather feedback on the behavioral anchors developed to date. Paper and electronic (PDF) copies of the measure were distributed to OCS instructors to use in their regularly scheduled training events over the course of several weeks. Instructors were briefed on the fundamental assumptions behind the measure including the caution that the provided behavioral anchors are not exhaustive nor are they a checklist; rather, the anchors should serve as a guide on what to look for when evaluating leadership in the context of OCS. The instructors were allowed to decide where, when, and how to use the measure.
Separate focus groups were conducted with instructors from each company that participated in the pilot. As part of the first post-pilot focus group (Data Collection 2), instructors ( n = 5) were asked to help refine the anchors for a prioritized set of leadership attributes and competencies including fitness, sound judgment, leads by example, creates a positive environment, and gets results. These attributes and competencies were prioritized because they had the least detail from previous data collection sessions. Feedback was also gathered on the prospective utility and usability of the measure for evaluating leadership performance in the OCS context. According to the instructors, the main benefits of the measure was to (a) serve as a guide for new instructors; (b) provide justification for certain ratings if questions/concerns arise; (c) help with evaluation in ambiguous situations; and (d) help with composing the evaluative and formative comments that accompany the leadership assessment ratings. Overall, this focus group resulted in minor wording changes to some of the behavioral anchors, deletions of irrelevant and
low priority behaviors, and transition of some behavioral anchors to different rating categories in order to more accurately reflect performance expectations and standards in this training context.
As part of the second post-pilot focus group (Data Collection 3), instructors ( n = 3) were asked to provide feedback on the following: (a) if, when, and how they used the measure during the pilot; (b) the utility and usability of the leadership attribute/competency behavioral anchors; (c) the measure development strategy for the attributes under character; and (d) the accuracy of the themes that had been identified by the research team for the Army Values and Warrior Ethos. Instructors were also asked to provide behavioral examples for each Army Value and comment on the relevance of the behavioral examples developed by the research team. Feedback regarding the utility and usability of the measure for performance evaluation in this context was largely consistent with that received during the first post-pilot focus group. Specifically, instructors indicated the behavioral anchors were helpful when giving developmental feedback to OCS candidates and provided the instructors with additional ideas on what to coach. Consistent with Data Collection 2, these instructors suggested that the main prospective benefit of the measure would be to serve as a guide for new instructors during training. Interestingly, one instructor suggested that the measure may be more useful when evaluating leadership in a field setting rather than in garrison given the wide range of behaviors that can be observed in the field. The most critical feedback received pertaining to the character measure was to reduce the behavioral examples to two rating categories (go/no-go). The concern was that the four-point rating scale was unnecessarily complex and examples of wrong and right would be preferable; reducing the content to a few simple, key points of what to observe would be more valuable in this context.
Based on this feedback, the anchors for the character measure were transitioned from the four-point rating scale to a dichotomous rating format. A series of internal working groups and iterative content revisions were conducted to continue refining the character measure, and go/no- go behavioral examples were developed for empathy and discipline. A rating approach for the Army Value honor was also conceptualized in a manner that aligned with the provided Army definition (i.e., if rating on any Army Value equals no-go, then rating on honor should also be no-go).
As part of Data Collection 4, instructors ( n = 2) were asked to (a) evaluate the dichotomous behavioral indicators for the character portion of the BARS; (b) provide feedback on the relevance and accuracy of the go/no-go behavioral examples in the context of OCS; and (c) discuss the prospective utility of the measure within OCS. The instructors confirmed utility of the dichotomous rating approach and suggested that the measure may be helpful to OCS candidates during the peer evaluation process. Specifically, the anchors would help contextualize and define character for OCS candidates, and may enhance the quality of peer commentary. Instructors also discussed the prospective utility of the measure for self-assessment; namely for promoting introspection in OCS candidates. Following this focus group, the character measure was refined based on feedback from instructors, resulting in minor wording and content changes. Subsequently, the behavioral anchors were reviewed by the research team to improve clarity, relevance, completeness, and consistency.
To explore the potential application of the BARS to a field training environment, field training observations were conducted. Five members of the research team observed squad-level field training exercises where OCS candidates were rotated into squad leader positions and
Future research should focus on validating the developed leadership measure, as well as examining the impact on formative feedback, learning, and training outcomes. Furthermore, the use of the measure for evaluating leadership in the field should continue to be explored. The attributes and competencies delineated in the Army leadership requirements model are critical for successful performance in field training exercises.
References
Borman, W. C. (1991). Job behavior, performance, and effectiveness. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 2, pp. 271- 326). Palo Alto, CA: Consulting Psychologists Press.
Campbell, T., & Cairns, H. (1994). Developing and measuring the learning organization: From buzz words to behaviors. Industrial and Commercial Training, 26 , 10-15. doi:10.1108/
Guion, R. M. (2011). Assessment, measurement, and prediction for personnel decisions (2 nd^ ed.). New York, NY: Routledge.
Hedge, J. W., Borman, W. C., Bruskiewicz, K. T., & Bourne, M. J. (2004). The development of an integrated performance category system for supervisory jobs in the U.S. Navy. Military Psychology, 16 , 231-243. doi:10.1207/s15327876mp1604_
Jacobs, R., Kafry, D., & Zedeck, S. (1980). Expectations of behaviorally anchored rating scales. Personnel Psychology, 33 , 595-640. doi:10.1111/j.1744-6570.1980.tb00486.x
Selvarajan, R. & Cloninger, P. A. (2009). The influence of job performance outcomes on ethical assessments. Personnel Review, 38 , 398-412. doi:10.1108/
Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47 , 149-155. doi:10.1037/h
U.S. Department of the Army, Headquarters (2012). Army leadership (ADRP 6-22). Washington, DC.
U.S. Department of the Army, Headquarters (2013). Army leader development program (PAM 350-58). Washington, DC: Author.
U.S. Department of the Army, Headquarters (2014). Officer Candidate School Standard Operating Procedures. Retrieved from https://www.benning.army.mil/infantry/199th/ocs/content/pdf/OCSOP.pdf
U.S. Department of the Army, Headquarters (2015). Leader development (Field Manual No. 6- 22). Washington, DC: Author.
Needs Improvement Satisfactory Excellent Outstanding
but may be slow or waver when pressed
ambiguity but may be slow, overly cautious, or hesitate when pressed (e.g., unnecessarily seeks validation for decision or permission to take action)
when mission/task conditions change
Resilience
(^) • Is unable to bounce back
after a negative event; loses composure or becomes flustered when a mistake has been made; fails to course-correct or continue with task/mission
Needs Improvement Satisfactory Excellent Outstanding
INTELLECT Mental Agility
(^) • Fails to identify the main problem or does not act to implement a solution
Interpersonal Tact
(^) • Fails to adjust tone and interaction style for different contexts; does not respond to non-verbal signals from others (e.g., eye rolling)
Needs Improvement Satisfactory Excellent Outstanding
Expertise
(^) • Is unaware or unable to articulate tactical/technical procedures; parrots back objectives discussed at the beginning of week
Leads Others
(^) • Hinders subordinates' ability to accomplish task
Needs Improvement Satisfactory Excellent Outstanding
Extends Influence
(^) • Lets teammate fail in
leadership role by either taking over, undermining, or doing nothing
Leads by Example
(^) • Participates in some but
not all training activities
