This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by McWILLIAM, R. A. Articles by WARE, W. B. Search for Related Content Social Bookmarking                         What's this?

The Reliability of Observations of Young Children's Engagement

An Application of Generalizability Theory

University of North Carolina at Chapel Hill

WILLIAM B. WARE

University of North Carolina at Chapel Hill

The reliability of measurement of how young children spend their time has traditionally been computed in terms of interobserver exact agreement This study sought to apply generalizability theory to the measurement of engagement Forty-seven young children, 15 of whom had disabilities, were observed four times in their child care setting. Types and levels of engagement were coded by three raters. Using ANOVA procedures for determining the relative contribution of different sources of error, a fully crossed (with subjects) two-facet (sessions, raters) generalizability (G) study was employed. The nine outcome measures were four types and five levels of engagement Results showed that raters accounted for less than 2% of the variance in the error of the scores, while sessions accounted for most of the variance other than between subject variance. The outcome measures proved to vary in their reliability robustness. The G study was followed by a decision (D) study to determine the levels of the facets that would be required to achieve a generalizability coefficient of .80. The number of sessions could be realistically increased to achieve this aim, but the increase in "dependability" achieved with the addition of each rater was minimal. Conclusions are drawn about the importance of assessing more than one source of error, raters, in observational research Overall, molecular methods were determined to be relatively unstable for measuring the molar construct of engagement, but certain engagement outcomes were stable across sessions and raters.

Journal of Early Intervention, Vol. 18, No. 1, 34-47 (1994)
DOI: 10.1177/105381519401800104


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				N. Watt, A. Wetherby, and S. Shumway Prelinguistic Predictors of Language Outcome at 3 Years of Age J Speech Lang Hear Res, December 1, 2006; 49(6): 1224 - 1237. [Abstract] [Full Text] [PDF]

				C. T. Bruckner, P. J. Yoder, and R.A. McWilliam Generalizability and Decision Studies: An Example Using Conversational Language Samples Journal of Early Intervention, January 1, 2006; 28(2): 139 - 153. [Abstract] [PDF]

				C. J. Dunst, M. B. Bruder, C. M. Trivette, D. Hamby, M. Raab, and M. McLean Characteristics and Consequences of Everyday Natural Learning Opportunities Topics in Early Childhood Special Education, January 1, 2001; 21(2): 68 - 92. [Abstract] [PDF]

				R. B. McCathren Teacher-Implemented Prelinguistic Communication Intervention Focus on Autism and Other Developmental Disabilities, January 1, 2000; 15(1): 21 - 29. [Abstract] [PDF]

				R. B. McCathren, P. J. Yoder, and S. F. Warren Testing Predictive Validity of the Communication Composite of the Communication and Symbolic Behavior Scales Journal of Early Intervention, January 1, 2000; 23(1): 36 - 46. [Abstract] [PDF]

				P. M. Brown, M. D. Remine, S. J. Prescott, and F. W. Rickards Social Interactions of Preschoolers With and Without Impaired Hearing in Integrated Kindergarten Journal of Early Intervention, January 1, 2000; 23(3): 200 - 211. [Abstract] [PDF]

				R. B. McCATHREN, P. J. YODER, and S. F. WARREN Prelinguistic Pragmatic Functions as Predictors of Later Expressive Vocabulary Journal of Early Intervention, January 1, 1999; 22(3): 205 - 216. [Abstract] [PDF]

				P. J. YODER, S. F. WARREN, and L. HULL Predicting Children's Response to Prelinguistic Communication Intervention Journal of Early Intervention, January 1, 1995; 19(1): 74 - 84. [Abstract] [PDF]

				R. A. McWilliam and D. B. Bailey Jr. Effects of Classroom Social Structure and Disability on Engagement Topics in Early Childhood Special Education, January 1, 1995; 15(2): 123 - 147. [Abstract] [PDF]