Invalidating the Full Scale IQ Score in the Presence of Significant Factor Score Variability: Clinical Acumen or Clinical Illusion?
Abstract
Within the professional literature, it is frequently suggested that significant variability in lower-level factor and index scores on IQ tests renders the resulting FSIQ an inappropriate focus for clinical interpretation and diagnostic decision-making. To investigate the tenability of this popular interpretive heuristic, the present study examined the structural and predictive validity of the KABC-II for participants in the normative sample who were observed to have significant variability in their factor scores. Participants were children and adolescents, ages 7-18, (N = 2,025) drawn from the KABC-II/KTEA-II standardization sample. The sample was nationally stratified and proportional to U.S. census estimates for sex, ethnicity, geographic region, and parent education level. Using exploratory factor analysis and multiple factor extraction criteria, support for a five-factor extraction was obtained consistent with publisher theory. As recommended by Carroll (1993; 1995) hierarchical structure was explicated by sequentially partitioning variance appropriately to higher- and lower-order dimensions. Results showed the largest portions of total and common variance were accounted for by the second-order general factor with meaningful residual variance accounted for by Short-Term Memory at ages 7-12 and 13-18. As a result, the Fluid-Crystallized Index (FCI) accounted for large predictive effects across measures of academic achievement whereas the five first-order CHC factor scores consistently accounted for trivial proportions of incremental predictive variance beyond the FCI. Implications for clinical practice and the correct interpretation of the KABC-II and other related measurement instruments in the presence of significant scatter are discussed.References
American Educational Research Association, American Psychological Association, & National Council on Measurement on Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Bain, K., & Gray, R. (2008). Test reviews: Kaufman, A. S., & Kaufman, N. L. (2004). Kaufman Assessment Battery for Children, Second edition. Circle Pines, MN: AGS. Journal of Psychoeducational Assessment, 26, 92-101. doi: 10.1177/0734282907300461
Baron, J. (2008). Thinking and deciding (4th ed.). New York: Cambridge University Press.
Beall, A. L., Holdnack, J. A., Saklofske, D. H., & Weiss, L. G. (2016). Practical considerations in WISC-V interpretation and intervention. In L. G. Weiss, D. H. Saklofske, J. A. Holdnack, & A. Prifitera (Eds.), WISC-V assessment and interpretation: Scientist-practitioner perspectives. (pp. 66-93). San Diego, CA: Academic Press.
Beaujean, A. A. (2015). John Carroll’s views on intelligence: Bi-factor vs. higher-order models. Journal of Intelligence, 3, 121-136. doi: 10.3390/jintelligence3040121
Beaujean, A. A., Parkin, J., & Parker, S. (2014). Comparing Cattell-Horn-Carroll factor models: Differences between bifactor and higher order factor models in predicting language achievement. Psychological Assessment, 26, 789-805. doi: 10.1037/a0036745
Braden, J. P. & Ouzts, S. M. (2005). Review of Kaufman Assessment Battery for Children, Second Edition. In R. A. Spies & B. S. Plake (Eds.), The sixteenth mental measurements yearbook (pp. 517-520). Lincoln: University of Nebraska.
Canivez, G. L. (2008). Orthogonal higher-order factor structure of the Stanford-Binet Intelligence Scales for children and adolescents. School Psychology Quarterly, 23, 533-541. doi: 10.1037/a0012884
Canivez, G. L. (2011). Hierarchical factor structure of the Cognitive Assessment System: Variance partitions from the Schmid-Leiman (1957) procedure. School Psychology Quarterly, 26, 305-317. doi: 10.1037/a0025973
Canivez, G. L. (2013a). Incremental validity of WAIS-IV factor index scores: Relationships with WIAT-II and WIAT-III subtest and composite scores. Psychological Assessment, 25, 484-495. doi: 10.1037/a0032092
Canivez, G. L. (2013b). Psychometric versus actuarial interpretation of intelligence and related aptitude batteries. In D. H. Saklofske, C. R. Reynolds, & V. L. Schwean (Eds.). The Oxford handbook of child psychological assessment (pp. 84-112). New York: Oxford
University Press.
Canivez, G. L. (2014). Construct validity of the WISC-IV with a referred sample: Direct versus indirect hierarchical structures. School Psychology Quarterly, 29, 38-51. doi: 10.1037/spq0000032
Canivez, G. L., & Watkins, M. W. (2010). Investigation of the factor structure of the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV): Exploratory and higher-order factor analyses. Psychological Assessment, 22, 827-836. doi: 10.1037/a0020429
Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press.
Carroll, J. B. (1995). On methodology in the study of cognitive abilities. Multivariate Behavioral Research, 30, 429-452. doi: 10.1207/s15327906mbr3003_6
Carroll, J. B. (1998). Human cognitive abilities: A critique. In J. J. McArdle & R. W. Woodcock (Eds.), Human cognitive abilities in theory and practice (pp. 5-23). Mahwah, NJ: Erlbaum.
Carroll, J. B. (2003). The higher–stratum structure of cognitive abilities: Current evidence supports g and about ten broad factors. In H. Nyborg (Ed.), The scientific study of general intelligence: Tribute to Arthur R. Jensen (pp. 5–21). New York, NY: Pergamon.
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276. doi: 10.1207/s15327906mbr0102_10
Chapman, L. J., & Chapman, J. P. (1967). Genesis of popular but erroneous psychodiagnostic observations. Journal of Abnormal Psychology, 72, 193-204. doi: 10.1037/h0024670
Chen, F. F., Hayes, A., Carver, C. S., Laurenceau, J. P., & Zhang, Z. (2012). Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. Journal of Personality, 80, 219-251. doi:10.1111/j.1467–6494.2011.00739.x
Child, D. (2006). The essentials of factor analysis (3rd ed.). New York: Continuum Publishing.
Cohen, J. (1988). Statistical power analysis for the behavior sciences (2nd ed.). New York: Psychology Press.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum
Courville, T., Coalson, D. L., Kaufman, A. S., & Raiford, S. E. (2016). Does WISC-V scatter matter? In A. S. Kaufman, S. E. Raiford, & D. L. Coalson, Intelligent testing with the WISC-V (pp. 209-228). Hoboken, NJ: John Wiley.
Dana, J., & Dawes, R. (2007). Comment on Fiorello et al. Interpreting Intelligence Test Results for Children with Disabilities: Is Global Intelligence Relevant? Applied Neuropsychology, 14, 21-25. doi: 10.1080/09084280701280379
Daniel, M. H. (2007). “Scatter†and the construct validity of FSIQ: Comment on Fiorello et al. (2007). Applied Neuropsychology, 14, 291-295. doi: 10.1080/09084280701719401
Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgement. Science, 243, 1668-1674. doi: 10.1126/science.2648573
Decker, S. L., Hale, J. B., & Flanagan, D. P. (2013). Professional practice issues in the assessment of cognitive functioning for educational applications. Psychology in the Schools, 50, 300-313. doi: 10.1002/pits.21675
DeMars, C. E. (2013). A tutorial on interpreting bifactor model scores. International Journal of Testing, 13, 354-378. doi: 10.1080/15305058.2013.799067
Dombrowski, S. C. (2013). Investigating the structure of the WJ-III Cognitive at school age. School Psychology Quarterly, 28, 154-169. doi: 10.1037/spq0000010
Dombrowski, S. C., & Gischlar, K. L. (2014). Ethical and empirical considerations in the identification of learning disabilities. Journal of Applied School Psychology, 30, 68-82. doi: 10.1080/15377903.2013.869786
Dombrowski, S. C., Canivez, G. L., Watkins, M. W., & Beaujean, A. A. (2015). Exploratory bifactor analysis of the Wechsler Intelligence Scale for Children-Fifth Edition with the 16 primary and secondary subtests. Intelligence, 53, 194-201. doi:
1016/j.intell.2015.10.009
Drozdick, L. W., Wahlstrom, D., Zhu, J., & Weiss, L. G. (2012). The Wechsler Adult Intelligence Scale-Fourth Edition and the Wechsler Memory Scale-Fourth Edition. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary Intellectual Assessment: Theories, Tests, and Issues (3rd ed.; pp. 197-223). New York: Guilford Press.
Fabrigar, L. R., & Wegener, D. T. (2012). Exploratory factor analysis. New York: Oxford University Press.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi: 10.1037/1082-989X.4.3.272
Faust, D. (1990). Data integration in legal evaluations: Cab clinicians deliver on their premises? Behavioral Sciences and the Law, 7, 469-483. doi: 10.1002/bsl.2370070405
Feifer, S. G., Nader, R. G., Flanagan, D. P., Fitzer, K. R., & Hicks, K. (2014). Identifying specific reading subtypes for effective educational remediation. Learning Disabilities: A Multidisciplinary Journal, 20, 18-30. Retrieved from http://js.sagamorepub.com/ldmj/article/view/5151
Fiorello, C. A., Hale, J. B., Holdnack, J. A., Kavanagh, J. A., Terrell, J., & Long, L. (2007). Interpreting intelligence test results for children with disabilities: Is global Intelligence relevant? Applied Neuropsychology, 21. 2-12. doi: 10.1080/09084280701280338
Fiorello, C. A., Hale, J. B., McGrath, M., Ryan, K., & Quinn, S. (2002). IQ interpretation for children with flat and variable test profiles. Learning and Individual Differences, 13, 115-125. doi: 10.1016/S1041-6080(02)00075-4
Fiorello, C. A., Hale, J. B., & Wycoff, K. L. (2012). Cognitive hypothesis testing. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (3rd ed., pp. 484-496). New York: Guilford.
Flanagan, D. P., & Kaufman, A. S. (2004). Essentials of WISC-IV assessment. Hoboken, NJ: John Wiley.
Flanagan, D. P., Alfonso, V. C., & Mascolo, J. T. (2011). A CHC-based operational definition of SLD: Integrating multiple data sources and multiple data-gathering methods. In D. P. Flanagan & V. C. Alfonso (Eds.), Essentials of specific learning disability identification
(pp. 233-298). Hoboken, NJ: John Wiley.
Flanagan, D. P., Ortiz, S. O., & Alfonso, V. C. (2013). Essentials of cross-battery assessment (3rd ed.). Hoboken, NJ: John Wiley.
Flanagan, D. P., Ortiz, S. O., Alfonso, V. C., & Dynda, A. (2006). Integration of response-to-intervention and norm-referenced tests in learning disability identification: Learning from the Tower of Babel. Psychology in the Schools, 43, 807-825. doi: 10.1002/pits.20190
Fletcher, J. M., Steubing, K. K., Morris, R. D., & Lyon, G. R. (2013). Classification and definition of learning disabilities: A hybrid model. In H. L. Swanson, K. R. Harris, & S. Graham (Eds.), Handbook of learning disabilities (2nd ed., pp. 33-50). New York: Guilford Press.
Frazier, T. W., & Youngstrom, E. A. (2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? Intelligence, 35, 169-182. doi: 10.1016/j.intell.2006.07.002
Freberg, M., Vandiver, B. J., Watkins, M. W., & Canivez, G. L. (2008). Significant factor score variability and the validity of the WISC-III full scale IQ in predicting later academic achievement. Applied Neuropsychology, 15, 131-139. doi: 10.1080/09084280802084010
Gambrill, E. (2012). Critical thinking in clinical practice: Improving the quality of judgements and decisions (3rd ed.). Hoboken, NJ: John Wiley.
Gignac, G. E. (2007). Multi-factor modeling in individual differences research: Some recommendations and suggestions. Personality and Individual Differences, 42, 37-48. doi: 0.1016/j.paid.2006.06.019
Gignac, G. E. (2016). Residual group-level factor associations: Possibly negative implications for the mutualism theory of general intelligence. Intelligence, 55, 69-78. doi: 10.1016/j.intell.2016.01.007
Gignac, G. E., & Watkins, M. W. (2013). Bifactor modeling and the estimation of model-based reliability in the WAIS–IV. Multivariate Behavioral Research, 48, 639-662. doi:10.1080/00273171.2013.804398
Glutting, J. J., McDermott, P. A., Watkins, M. W., & Kush, J. C. (1997). The base rate problem and its consequences for interpreting children’s ability profiles. School Psychology Review, 26, 176-188. Retrieved from http://www.nasponline.ord
Glutting, J. J., Watkins, M. W., & Youngstrom, E. A. (2003). Multifactored and cross-battery assessments: Are they worth the effort? In C. R. Reynolds & R. W. Kamphaus (Eds.), Handbook of psychological and educational assessment of children: Intelligence aptitude, and achievement (2nd ed., pp. 343-374). New York: Guilford.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
Gorsuch, R. L. (2003). Factor analysis. In J. A. Schinka & F. F. Velicer (Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (pp. 143-164). Hoboken, NJ: John Wiley.
Gottfredson, L. S. (1997). Why g matters: The complexity of everyday life. Intelligence, 24, 79-132. doi: 10.1016/S0160-2896(97)90014-3
Groth-Marat, G. (2009). Handbook of psychological assessment (5th ed.). Hoboken, NJ: John Wiley.
Hale. J. B., & Fiorello. C. A. (2001). Beyond the academic rhetoric of 'g': Intelligence testing guidelines for practitioners. The School Psychologist, 55, 113-139. Retrieved from http://www.apadivisions.org/division-16/publications/newsletters/school-psychologist/2001/11-issue.pdf
Hale, J. B., & Fiorello, C. A. (2004). School neuropsychology: A practitioner’s handbook. New York: Guilford.
Hale, J. B., Fiorello, C. A., Kavanuagh, J. A., Holdnack, J. A., & Aloe, A. M. (2007). Is the demise of IQ interpretation justified? A response to special issue authors. Applied Neuropsychology, 14, 37-51. doi:10.1080/09084280701280445.
Hale, J. B., Fiorello, C. A., Dumont, R., Willis, J. O., Rackley, C., & Elliott, C. (2008). Differential Ability Scales-Second Edition (neuro)psychological predictors of math performance for typical children and children with math disabilities. Psychology in the Schools, 45, 838-858. doi: 10.1002/pits.20330
Haynes, S. N., Smith, G. T., & Hunsley, J. D. (2011). Scientific foundations of clinical assessment. New York: Routledge.
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychomerika, 30, 179-185. doi: 10.1007/BF02289447
Horn, J. L. (1991). Measurement of intellectual capabilities: A review of theory. In K. McGrew, J. K. Werder, & R. W. Woodcock, WJ-R technical manual (pp. 197–232). Itasca, IL: Riverside Publishing.
Hunsley, J. (2003). Introduction to the special section on incremental validity and utility in clinical assessment. Psychological Assessment, 15, 443-445. doi:10.1037/1040-3590.15.4.443
Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36. doi: 10.1007/BF02291575
Kaufman, A. S. (1994). Intelligent testing with the WISC-III. New York: John Wiley.
Kaufman, A. S., & Kaufman, N. L. (2004a). Kaufman Assessment Battery for Children (2nd ed.). Circle Pines, MN: American Guidance Service.
Kaufman, A. S., & Kaufman, N. L. (2004b). Kaufman Assessment Battery for Children-Second Edition manual. Circle Pines, MN: American Guidance Service.
Kaufman, A. S., & Kaufman, N. L. (2004c). Kaufman Test of Educational Achievement (2nd ed.). Circle Pines, MN: American Guidance Service.
Kaufman, A. S., & Kaufman, N. L. (2004d). Kaufman Test of Educational Achievement-Second Edition manual. Circle Pines, MN: American Guidance Service.
Kaufman, A. S., Lichtenberger, E. O., Fletcher-Janzen, E., & Kaufman, N. L. (2005). Essentials of KABC-II assessment. Hoboken, NJ: John Wiley.
Kotz, K. M., Watkins, M. W., & McDermott, P. A. (2008). Validity of the general Conceptual Ability score on the Differential Ability Scales as a function of significant and rare interfactor variability. School Psychology Review, 37, 261-278. Retrieved from http://www.nasponline.org
Kranzler, J. H., & Floyd, R. G. (2013). Assessing intelligence in children and adolescents: A practical guide. New York: Guilford Press.
Lichtenberger, E. O., Sotelo-Dynega, M., & Kaufman, A. S. (2009). The Kaufman Assessment Battery for Children-Second Edition. In J. A. Naglieri & S. Goldstein (Eds.), Practitioner’s guide to assessing intelligence and achievement (pp. 61-94). Hoboken, NJ: John Wiley.
Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012). Neuropsychological assessment (5th ed.). New York: Oxford University Press.
Lilienfeld, S. O., Ammirati, R., & David, M. (2012). Distinguishing between science pseudoscience in school psychology: Science and scientific thinking as safeguards against human error. Journal of School Psychology, 50, 7-36. doi:
1016/j.jsp.2011.09.006
Luria, A. R. (1966). Human brain and psychological processes. New York: Harper Row.
Macmann, G. M., & Barnett, D. W. (1997). Myth of the master detective: Reliability of interpretations for Kaufman’s “intelligent testing†approach to the WISC-III. School Psychology Quarterly, 12, 197-234. doi: 10.1037/h0088959
Marley, S. C., & Levin, J. R. (2011). When are prescriptive statements in educational research justified? Educational Psychology Review, 23, 197-206. doi: 10.1007/s10648-011-9154-y
McDermott, P. A., Fantuzzo, J. W., & Glutting, J. J. (1990). Just say no to subtest analysis: A critique on Wechsler theory and practice. Journal of Psychoeducational Assessment, 8, 290-302. doi: 10.1177/073428299000800307
McDermott, P. A., Fantuzzo, J. W., Glutting, J. J., Watkins, M. W., & Baggaley, A. R. (1992). Illusions of meaning in the ipsative assessment of children’s ability. The Journal of Special Education, 25, 504-526. doi: 10.1177/002246699202500407
McDermott, P. A., Watkins, M. W., & Rhoad, A. (2014). Whose IQ is it?—Assessor bias variance in high-stakes psychological assessments. Psychological Assessment, 26, 207-214. Doi: 10.1037/a0034832
McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
McFall, R. M. (2000). Elaborate reflections on a simple manifesto. Applied and Preventive Psychology, 9, 5-21. doi: 10.1016/S0962-1849(05)80035-6
McGill, R. J. (2015). Interpretation of KABC-II scores: An evaluation of the incremental validity of Cattell-Horn-Carroll (CHC) factor scores in predicting achievement. Psychological Assessment, 27, 1417-1426. doi: 10.1037/pas0000127
McGill, R. J., & Busse, R. T. (2015). Incremental validity of the WJ III COG: Limited predictive effects beyond the GIA-E. School Psychology Quarterly, 30, 353-365. doi: 10.1037/spq0000094
McGill, R. J., & Spurgin, A. R. (2015). Exploratory higher order analysis of the Luria interpretive model on the Kaufman Assessment Battery for Children-Second Edition (KABC-II) school-age battery. Assessment. Advanced online publication. doi: 10.1177/1073191115614081
McGill, R. J., Styck, K. M., Palomares, R. S., & Hass, M. R. (2015). Critical issues in specific learning disability identification: What we need to know about the PSW model. Learning Disability Quarterly. Advance online publication. doi: 10.1177/0731948715618504
Meehl, P .E. (1956). Wanted-a good cook-book. American Psychologist, 11, 263-272. doi: 10.1037/h0044164
Naglieri, J. A. (2000). Can profile analysis of ability test scores work? An illustration using the PASS theory and CAS with an unselected cohort. School Psychology Quarterly, 15, 419-433. doi: 10.1037/h0088798
Nelson, J. M., & Canivez, G. L. (2012). Examination of the structural, convergent, and incremental validity of the Reynolds Intellectual Assessment Scales (RIAS) with a clinical sample. Psychological Assessment, 24, 129-140. doi:10.1037/a0024878
Nickerson, R. S. (2004). Cognition and Chance: The psychology of probabilistic reasoning. Mahwah, NJ: Erlbaum.
Pfeiffer, S. I., Reddy, L. A., Kletzel, J. E., Schmelzer, E. R., & Boyer, L. M. (2000). The practitioner’s view of IQ testing and profile analysis. School Psychology Quarterly, 15, 376-385. doi: 10.1037/h0088795
Prifitera, A., & Dersh, J. (1993). Base rates of WISC-III diagnostic subtest patterns among normal, learning-disabled, and ADHD samples. Journal of Psychoeducational Assessment Monograph Series, 43-55.
Rapaport, D., Gil, M., & Schafer, R. (1945). Diagnostic psychological testing: The theory, statistical evaluation, and diagnostic application of a battery of tests (Vol. 1). Chicago: Yearbook Medical.
R Development Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. doi:10.1080/00273171.2012.715555
Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129-140. doi:10.1080/00223891.2012.725437
Reschly, D. J., Myers, T. J., & Hartel, C. (Eds.). (2002). Mental retardation: Determining eligibility for social security benefits. Washington, DC: National Academy Press.
Revelle, W. (2012). Psych: Procedures for psychological, psychometric, and personality research (version 1.2.4) [Computer Software]. Evanston, IL: Northwestern University
Schmid, J., & Leiman, J. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF02289209
Schmidt, F. L., & Hunter, J. (2004). General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality and Social Psychology, 86, 162-173. doi: 10.1037/0022-3514.86.1.162
Schneider, W. J. (2008). Playing statistical Ouija board with communality analysis: Good questions, wrong assumptions. Applied Neuropsychology, 15, 44-53. doi: 10.1080/09084280801917566
Schneider, W. J. (2013). Principles of assessment aptitude and achievement. In D. H. Saklofske, C. R. Reynolds, & V. L. Schwean (Eds.). The Oxford handbook of child psychological assessment (pp. 286-330). New York: Oxford University Press.
Schneider, W. J., & McGrew, K. S. (2012). The Cattell-Horn-Carroll model of intelligence. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (3rd ed., pp. 99-144). New York: Guilford.
Strickland, T., Watkins, M. W., & Caterino, L. C. (2015). Structure of the Woodcock-Johnson III cognitive tests in a referral sample of elementary school students. Psychological Assessment, 27, 689-697. doi: 10.1037/pas0000052
Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.
Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 31, 321-327. doi: 10.1007/BF02293557
Wasserman, J. D., & Bracken, B. A. (2013). Fundamental psychometric considerations in assessment. In J. R. Graham & J. A. Naglieri (Eds.), Handbook of psychology: Assessment psychology (2nd ed., Vol. 10, pp. 50-81). Hoboken, NJ: John Wiley.
Watkins, M. W. (2000). Cognitive profile analysis: A shared professional myth. School Psychology Quarterly, 15, 465-479. doi: 10.1037/h0088802
Watkins, M. W. (2003). IQ subtest analysis: Clinical acumen or clinical illusion? The Scientific Review of Mental Health Practice, 2, 118-141. Retrieved from http://www.srmhp.org/0202/iq.html
Watkins, M. W. (2004). MacOrtho. [Computer Software]. State College, PA: Ed & Psych Associates.
Watkins, M. W. (2006). Orthogonal higher order structure of the Wechsler Intelligence Scale for Children-Fourth Edition. Psychological Assessment, 18, 123-125. doi: 10.1037/1040-3590.18.1.123
Watkins, M. W. (2013). Omega. [Computer Software]. Phoenix, AZ: Ed & Psych Associates.
Watkins, M. W., Glutting, J. J., & Lei, P. W. (2007). Validity of the full scale IQ when there is significant variability among WISC-III and WISC-IV factor scores. Applied Neuropsychology, 14, 13-20. doi: 10.1080/09084280701280353
Watkins, M. W., & Smith, L. G. (2013). Long-term stability of the Wechsler Intelligence Scale for Children-Fourth Edition. Psychological Assessment, 25, 477-483. doi:
1037/a0031653
Wechsler, D. (2008). Wechsler Adult Intelligence Scale-Fourth Edition technical and interpretive manual. San Antonio, TX: NCS Pearson.
Wechsler, D. (2014). Wechsler Intelligence Scale for Children-Fifth Edition technical and interpretive manual. San Antonio, TX: NCS Pearson.
Weiner, I. B. (1989). On competence and ethicality in psychodiagnostic assessment. Journal of Personality Assessment, 53, 827-831. doi:10.1207/s15327752jpa5304_18
Wiggins, J. S. (1988). Personality and prediction: Principles of personality assessment. Malabar, FL: Krieger Publishing.
Wood, J. M., Tatryn, D. J., & Gorsuch, R. L. (1996). Effects of under- and overextraction on principal axis factor analysis with varimax rotation. Psychological Methods, 1, 354-365. doi: 10.1037/1082-989X.1.4.354
Yates, B. T., & Taub, J. (2003). Assessing the costs, benefits, cost-effectiveness, and cost-benefit of psychological assessment: We should, we can, and here’s how. Psychological Assessment, 15, 478-495. doi: 10.1037/1040-3590.15.4.478
