Download Development of Practical Item Analysis Program for Indonesian Teachers: AnBuso and more Study Guides, Projects, Research Learning processes in PDF only on Docsity!
International Journal of Instruction April 2017 ● Vol.10, No.
e-ISSN: 1308-1470 ● www.e-iji.net p-ISSN: 1694-609X
pp. 199-
Citation: Muhson, A., Lestari, B., Supriyanto & Baroroh, K. (2017). The Development of Practical Item Analysis Program for Indonesian Teachers. International Journal of Instruction , 10(2), 199 - 210. https://doi.org/10.12973/iji.2017.10213a
Received: 17/10/ Revision: 02/12/ Accepted: 09/12/
The Development of Practical Item Analysis Program for Indonesian
Teachers
Ali Muhson Yogyakarta State University, Indonesia, alimuhson@uny.ac.id
Barkah Lestari
Yogyakarta State University, Indonesia, barkahlestari@uny.ac.id
Supriyanto
Yogyakarta State University, Indonesia, supriyanto@uny.ac.id
Kiromim Baroroh Yogyakarta State University, Indonesia, kiromim_b@uny.ac.id
Item analysis has essential roles in the learning assessment. The item analysis program is designed to measure student achievement and instructional effectiveness. This study was aimed to develop item-analysis program and verify its feasibility. This study uses a Research and Development (R & D) model. The procedure includes designing and developing a product, validating, and testing the product. The data were collected through documentations, questionnaires, and interviews. This study successfully developed item analysis program, namely AnBuso. It is developed based on classical test theory (CTT). It was practical and applicable for Indonesian teachers to analyse test items.
Key Words: feasibility, item-analysis program, AnBuso , development, practical item, teachers
INTRODUCTION
Learning processes determine the quality of education. This implies that teachers have a very central role in improving the quality of education. They have crucial roles not only in designing and implementing but also in assessing the learning processes and their outcomes. Improving the quality of learning and assessment system is one of the efforts to improve the quality of education.
Learning and assessments can be said to be ‘two sides of the same coin’ (Guliker, Bastianes, & Kirschner, 2004). All forms of assessment are essential components of teaching (Russell & Airasian, 2012) and they have strong influences on the quality of
200 The Development of Practical Item Analysis Program …
learning (Raymond, Homer, Smith, & Gray, 2012). Assessments involve not only finding out what students have learned but also improving the processes of learning and teaching (Bers, 2008). This implies that assessments have essential roles in improving not only the learning processes but also the learning outcomes. Good assessments will encourage teachers to determine appropriate instructional strategies and motivate students to study harder.
An assessment is a process of gathering evidence of student learning to inform instructional decisions (Stiggins & Chappuis, 2012). It means that teachers have to gather accurate evidence of the achievements of their students as the quality and impacts of their instructional decisions depend on it. The development of assessment instruments should therefore measure all aspects of learning objectives. Good students should attain better scores than the poor ones.
The implementation of the new curriculum, known as the 2013 curriculum, in the secondary education levels in Indonesia is facing many problems (Republika, 2014), for example the difficulty of changing teachers’ mind-set (Metronews, 2014; Tempo, 2013). A study reported that 87 percent of Indonesian teachers had difficulties in understanding the implementation of authentic assessments (Susilowati, 2013) as the assessment system includes too many aspects that must be measured (Tempo, 2014). It means that their ability to perform good assessments has to be improved.
An assessment is a systematic process of providing credible evidence of resources, implementation, and learning outcomes with the purpose of improving and developing effective instruction, programs, and services in higher education (Banta & Palomba, 2014). Thus, it is a process of gathering information about the students and in turn the information is employed to make instructional decisions and improve learning processes.
An assessment should be able to assess students' progress. Such information can be used as a basis for identifying the students’ status in the group and determining the next steps of learning. Therefore, the assessment has to be able to assess their competence accurately. Assessments help teachers clarify the goals and the purposes of their teaching, create learning experiences in real-life contexts, and provide many different ways for students to demonstrate their abilities and skills (Darling-Hammond, 2014). Properly used assessment procedures can contribute directly to the improvement of learning, such as (1) clarifying the nature of the intended learning outcomes, (2) providing short-term goals to work toward, (3) providing feedback concerning learning progress, (4) providing information for overcoming learning difficulties and for selecting future learning experiences, and (5) identifying the next instructional goal (Miller, Linn, & Gronlund, 2009).
The purposes of assessments include establishing a classroom that supports learning, planning and conducting instruction, placing students, providing feedback, diagnosing student problems and disabilities, and summarizing and grading academic learning and progress (Russell & Airasian, 2012). Thus, assessments are used to monitor the learning progress, provide information as a basis for giving feedback, and improve learning.
202 The Development of Practical Item Analysis Program …
correlation between item score and total scores. This index is the equivalent of a point- biserial coefficient in this application. Arbitrary interpretations have been made on ranges of values of the point-biserial correlation. They range from Very Good items (>0.40), Good items (<0.39, >0.30), Fair/marginal items (<0.29, >0.20), Non- discriminating/poor items (<0.19, >0.00), Need attention (<0.00) (Alagumalai & Curtis, 2005: 8; Ebel & Frisbie, 1991: 232; Crocker & Algina, 2008: 315). It means that the item with correlation coefficient less than 0.2 has to be rejected. Based on these criteria, an item is considered good if it has difficulty index between 0.3 to 0.7 and item discriminant coefficient more than 0.2. AnBuso uses these criteria to judge a good or poor item.
METHOD
This study uses a model of Research and Development (R & D). The procedure includes designing and developing a product, validating, and trying the product. A preliminary study was carried out to determine the ability and willingness of teachers to use item analysis and identify their real needs. The next step was designing a model of application program and its guidebook, which were validated by experts and revising them in order to obtain validated products.
The product validation was conducted using the Delphi method. This validation processes also involved some experts in education, measurements and computer programming. The testing of the trial products were conducted twice: limited and extended testing. The test was to obtain comprehensive information and input from users. The results show that all the instruments, the program and its guidebook are feasible to use although there are some suggestions for revision. The results of this test were also used to improve the quality of the products so that they are ready to be disseminated widely.
The study involved 65 respondents comprising teachers and school supervisors from five districts in Yogyakarta (DIY). Purposive sampling was carried out to select the respondents with computer literacy, especially the ability to operate Microsoft Excel program.
Observation, documentation, questionnaires, and interviews were employed to elicit the research data. Observation was conducted to obtain data or information about the teachers’ ability to use the application program that has been developed. The data of instruments used to measure students' abilities, such as formative and summative tests were obtained by documentation techniques. The elicited documents took the forms of final examination tests, daily tests, remedial programs, and the results of the previous item analysis used by the teachers.
Muhson, Lestari, Supriyanto & Baroroh 203
Table 1 Feasibility Categories No Score Category 1 More than M + 1,8 SD Very feasible 2 M + 0,6 SD to M + 1,8 SD Feasible 3 M – 0,6 SD to M + 0,6 SD Fair 4 M – 1,8 SD to M – 0,6 SD Less feasible 5 Less than M – 1,8 SD Not feasible
The questionnaire elicited feedback or suggestion from teachers, supervisors, education department officers, and education experts. The questionnaire was also developed to measure the feasibility of the program concerning its visual display, contents, usefulness, practicality, and convenience. Interviews were conducted to teachers, supervisors, education department officers, and experts to obtain information regarding the strength and drawbacks of the AnBuso program.
The data were analysesed descriptively and supported by tables and graphs. Five scales of categories were employed to measure the feasibility of the program (see Table 1).
FINDINGS
The study involved 65 participants comprising teachers, supervisors and education actors. Most of them (72%) were teachers of high schools (SMA, SMK, and MA). Many of them (68%) were teachers of public schools and 57% were government employees. Respondents came from various fields of study, including Economics, Accounting, Mathematics, Social Studies, English, Indonesian, Information Technology, Chemistry, Physics, Biology, Geography, Arabic, and so on.
The willingness of the teachers to analyze test items was low (57%). Only 11% of them regularly conducted test item analysis, 20% of them often did it and 12% of them never did it. It would be important to assess why the teachers had a low willingness to analyze test items.
In general, the teachers would perform item analysis if the supervisors ordered them to do so. It means that their awareness to perform item analysis was very low. This is because they were unable to use the available program. Of course this contributed to the poor quality of test items they have developed.
This study successfully developed AnBuso program and its manual book. AnBuso was developed on Microsoft Excel program. It has been improved based on the feedback from the respondents of the present research. The AnBuso was accompanied with a manual book in Indonesian language that will help teachers take advantage of the program. It provides step-by-step instructions which are very practical for the teachers to operate the program. The book contains an introduction, contents, input and data report.
The program has undertaken some important modifications in order to be compatible with the assesments system of the 2013 curriculum. Some revisions were conducted to
Muhson, Lestari, Supriyanto & Baroroh 205
needs. Some amendments were also made on the assessment scales. In the previous model, the grading scales were 1 - 10 and 1- 100. To cater for the assessments system of the 2013 curriculum, the new model provides a 1- 4 grade scale (see Figure 2 ).
Basically the purpose of the item analysis is to determine the quality of the test items and obtain information about the assessment results so that teachers can make a follow- up plan such as remedial and enrichment teaching. The program also provides information about the students who are included in the remedial and enrichment group.
Figure 3 shows the general overview of the AnBuso system. Basically the program is divided into three components: input, process and output. Data input consists of Input 01 which takes general data, and Input 02 which contains the identity of students and their answers. The data entered into the Input 01 spreadsheet include the School Name, Subject, Class/Program, Name of test, SK/KD, Name of the Teacher, Teacher’s NIP, Semester, Academic Year, Test Date, Inspected Date, name of the school Principal, Principal’s NIP, Reporting Place, Reporting Date, Grades (10 or 100), the score of Minimum competence criteria (KKM), Objective Test Data, and Essay Test Data. The data entered into the Input 02 spreadsheet include Students Name, Gender (M = Male, F = Female), Multiple Choice Answer Sheet, and Essay Score. Whenever the data are entered in the input 01 and input 02 spreadsheets, the program will automatically generate a complete report consisting of:
Data Input 01 (Identity)
Data Input 02 (Response Data)
Data Processing
Student Score Report
Item Analysis Report (Multiple Choice Test)
Item Analysis Report (Essay Test)
Response Pattern Report (Multiple Choice test)
Remedial Material Report
Remedial Grouping Report
Graphical Report (Score Distribution)
Graphical Report (Learning Mastery)
Figure 3 Flowchart Diagram Revision
206 The Development of Practical Item Analysis Program …
- Student Report Sheet which displays list of test scores. The list contains the name of student and gender, number of items answered correctly and incorrectly, objective test score, essay test score, final score, and description (passed and failed), The other information in this sheet is the total number of the students, the number (percentage) of students who passed the tests, the minimum scores, maximum scores, average scores, and the standard deviation.
- Item Report Sheet which shows the result of Multiple Choice Analysis. It contains Discriminant Index, levels of difficulty, Malfunction Distractors, and Conclusion of item in multiple choice test.
- Response Pattern Report Sheet which contains Distribution of the answers of Multiple Choice tests.
- Essay Report Sheet that contains Discriminant Index, levels of difficulty, and Conclusion of item in essay test.
- Remedial Material Reports Sheet which contains materials for Individual and Classical Remedial tests.
- Remedial Group Reports Sheet which contains the grouping of participants who take remedial tests.
- Graphics Sheet which contains charts of minimum competence criteria (KKM) score distributions, and the proportions of the learning mastery.
Practically and convenience
Content Screen display Usefullness Male Female
Figure 4 The results of the feasibility test
The results of the feasibility test showed that AnBuso program, developed in this study, was considered very feasible by the respondents. Of the total respondents, 51% said that it is feasible, 46% stated it is very feasible and the rest stated it is quite feasible. This shows that the program has a high usefulness in helping teachers perform item analysis.
208 The Development of Practical Item Analysis Program …
Guliker, J.T., Bastianes, T.J., & Kirschner, P.A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52 (3), 67-86.
Hambleton, R.K. & Swaminathan, H. (1985). Item response theory. Boston, MA : Kluwer Inc.
Kaplan, R.M. & Saccuzzo, D.P. (2005) Psychological testing: Principles, applications, and issues, 6th edition. Belmont, CA: Thomson Wadsworth.
Kubiszyn, T., & Borich, G.D. (2013). Educational testing and measurement: classroom application and practice (10th Ed.). New Jersey: John Wiley & Sons, Inc.
Mehrens, W.A. & Lehmann, I.J. (1991). Measurement and Evaluation in Education and Psychology, (4th Ed. ), Orlando, FL: Harcourt Brace Jovanovich.
Metronews. (2014). Ini delapan masalah dalam implementasi kurikulum 2013. Retrieved October 19, 2014, from Metronews: News.metronews.com.
Miller, M.D., Linn, R.L., & Gronlund, N.E. (2009). Measurement and assessment in teaching (10th edition). New Jersey: Pearson Education Inc.
Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3th edition). New York: McGraw-Hill, Inc.
Puskur. (2008). Model penilaian kelas kurikulum berbasis kompetensi. Jakarta: Balitbang Depdiknas.
Raymond, J.E., Homer, C.S., Smith, R., & Gray, J.E. (2012). Learning through authentic assessment: An evaluation of a new development in the undergraduate midwifery curriculum. Nurse Education in Practice, 13 (5), 471-476.
Republika. (2014). Implementasi kurikulum 2013 masih dibayangi banyak masalah. Retrieved November 18, 2014, from Republika: www.republika.co.id.
Russell, M.K., & Airasian, P.W. (2012). Classroom assessment: concepts and applications (7th Ed.). New York: McGraw-Hill.
Stiggins, R.J., & Chappuis, J. (2012). An introduction to student involved assessment for learning (6th Ed). Boston: Pearson assessment training institute.
Susilowati. (2013). Kurikulum 2013, 87 persen guru kesulitan cara penilaian. Retrieved November 18, 2014, from UNNES: http://unnes.ac.id.
Tempo. (2013). Problematika implementasi kurikulum 2013. Retrieved July 10, 2013, from Tempo: www.tempo.co.
Tempo. (2014). Kurikulum 2013, Apa Saja Kendalanya? Retrieved August 16, 2014, from Tempo: www.tempo.co.
Wiberg, M. (2004). Classical test theory vs. Item response theory: an evaluation of the theory test in the Swedish driving-license test. UMEA Universitet.
Muhson, Lestari, Supriyanto & Baroroh 209
Turkish Abstract Endonezyalı Öğretmenler İçin Uygulamalı Madde Analizi
Madde analizi öğrenmenin değerlendirilmesinde önemli bir role sahiptir. Madde analiz programı eğitimin etkililiğini ve öğrenci başarısını ölçmek için uygulanır. Bu çalışma bir madde analiz programı geliştirerek programın uygulanabilirliğini öçmeyi amaçlamıştır. Çalışmada, Araştırma ve Geliştirme (R&D) modeli kullanılmıştır. İşlem basamakları, ürün dizaynı ve geliştirilmesi, ürünün onaylanması ve test edilmesi aşamalarından oluşmaktadır. Veriler belgeleme, anketler ve görüşmeler yoluyla toplanmıştır. Bu çalışmada, başarılı bir şekilde AnBuso adı verilen klasik test teorisine (CTT) dayalı bir madde analiz programı geliştirilmiştir. Program Endonezyalı öğretmenlerin madde analizi yapmaları için uygun ve uygulanabilir bulunmuştur.
Anahtar Kelimeler: uygunluk, madde analiz programı, AnBuso, öğretmenler, öğrenmenin değerlendirilmesi, öğrenci başarısı, eğitimin etkililiği
French Abstract Le Développement de Programme d'Analyse d'Article Pratique pour Professeurs indonésiens
L'analyse d'article a des rôles essentiels dans l'évaluation apprenant. Le programme d'analyse d'article est conçu pour mesurer l'accomplissement d'étudiant et l'efficacité d'instruction. Cette étude a été visée à se développer l'analyse d'article programme et vérifie sa faisabilité. Cette étude utilise une Recherche et développement (la R et D) le modèle. La procédure inclut la conception et le développement d'un produit, une validation et un test du produit. Les données ont été rassemblées par des documentations, des questionnaires et des entretiens. Cette étude a avec succès développé le programme d'analyse d'article, à savoir AnBuso. Il est développé basé sur la théorie classique de test (CTT). C'était pratique et applicable pour des professeurs indonésiens pour analyser des articles de test items.
Mots Clés: faisabilité, programme d'analyse d'article, AnBuso, professeurs, apprenant évaluation, accomplissement d'étudiant, efficacité d'instruction
Arabic Abstract تطوير برنامج تحليل البند العملي ا للمعلمين االندونيسي تحليل العنصر لديه دورا أساسيا في تقييم التعلم. تم تصميم برنامج تحليل البند لقياس التحصيل العلمي للطالب وفعالية التعليم. (R &دراسة نموذج البحث والتطوير وتهدف هذه الدراسة إلى تطوير برنامج البند التحليل والتحقق من جدواه. تستخدم هذه ال ويشمل اإلجراء تصميم وتطوير المنتج والتأكد من صحتها، واختبار المنتج. تم جمع البيانات من خالل الوثائق، D). يتم تطويره على أساس نظريةAnBuso. واالستبيانات، والمقابالت. هذا نجحت في تطوير برنامج تحليل البند الدراسة، وهي .وكانت عملية وقابلة للتطبيق للمعلمين االندونيسي لتحليل بنود االختبار(CTT).الكالسيكية االختبار والمعلمين وتقويم التعلم، تحصيل الطالب، والفعالية التعليميةAnBuso الكلمات الرئيسية: الجدوى، برنامج البند التحليل،
