Development of the APA College National Examination in Psychopharmacology

by Janet Ciuccio

Because the College was developing an exam that it believes will eventually be used in state licensure, it was necessary to adhere to procedures that are widely considered protective against legal challenges involving validity and fairness. The specific standards that are most frequently cited in this context are the technical guidelines described in the Standards for Educational and Psychological Testing of the AERA/APA and the NCME and relevant sections of the Uniform Guidelines on Employee Selection Procedures of the EEOC, Civil Service Commission, U. S. Dept. of Labor and the U. S. Dept. of Justice.

Briefly, compliance with these standards involves the use of exam development methodology that results in the measurement of knowledge that is truly related to the job being performed and is not discriminatory against protected groups. In other words, the exam has validity for its intended purpose and use and is not discriminatory. The exam's content accurately reflects what you need to know to do the job.

In beginning our work, we had two initial tasks, both of which, if performed properly, would build the foundation for a valid and defensible exam. We needed to contract with a testing firm with a proven track record of adhering to each and every required step in developing a quality examination that was both valid and defensible. We also needed to constitute a representative panel of experts to work with our selected contractor.

First, the panel of experts:

We made extensive efforts to obtain nominations for what is now our 19-member expert panel. We sought a broad representation in terms of geographic location, gender, race, practice setting, years of experience, and area of expertise. Practitioners, scientific researchers and psychologists with dual credentials such as psychologist/physician, psychologist/pharmacologist or psychologist/nurse were sought to provide the necessary diversity of expertise, experience, and perspective.

467 letters were sent inviting nominations. Groups receiving invitations included:

  • State and Provincial Psychological Association Presidents (N=59) and Executive Directors (N=58)

  • APA Practice Divisions and Division 28 (N=32)

  • APA accredited educational program directors (N=303)

  • Department of Defense Fellows (N=10)

  • Association of State and Provincial Psychology Boards (ASPPB)

  • Society of Psychologists for Prescriptive Authority

  • Prescribing Psychologists' Register

  • Association of Medical School Psychologists

  • American College of Neuropsychopharmacology

Fifty-four nominations were received. Nine nominees declined and the remaining CVs were carefully reviewed with regard to education and training in psychopharmacology, neuroscience, pharmacology, medicine and nursing and with regard to experience.

Our original expert panel in early 1998 consisted of 15 individuals, including a representative from ASPPB, Dr. Lynn Rehm, who is with us today on this panel. Four more experts were added in 1999 as we moved along in the process. Seven of the 10 military prescribing psychologists are participating along with pharmacists, physicians, pharmacologists, researchers, practitioners and a nurse anesthetist. The breadth of experience and expertise represented on the panel is unquestionably representative.

While we were involved in constituting our expert panel, we were also fully involved in selecting a testing firm that would guide the development of the first versions of the PEP and that we could count on to continue to work with us over the years to analyze candidate and item performance and to update the exam to reflect changing knowledge as well as to develop additional forms of the exam. We solicited proposals from several highly regarded national testing firms. We asked each firm to describe its proposed methodology for determining the knowledge domain to be tested, its proposed methodology for obtaining multiple-choice items in sufficient number and quality to produce two original forms of the examination, its proposed methodology to be used for demonstrating the validity and fairness of the examination, and for setting an appropriate passing score. We required a description of each firm's itembanking capabilities and procedures, including procedures for ongoing monitoring and periodic reporting of exam statistics, as well as a description of procedures for protecting and maintaining test security, both during development and on an ongoing basis. Finally, we looked at each firm's proposed staffing plan and at the cost proposal.

After carefully evaluating each proposal, we selected Professional Examination Service of New York. With over 50 years of experience, PES is nationally regarded for the quality of its many licensing and certification examinations including examinations for licensure in physical therapy, marriage and family therapy, veterinary medicine, medical laboratory personnel, certification examinations for critical care nurses, nurse practitioners, clinical social workers, eye surgeons, opticians, dental hygienists and many more, especially of note is PES's national licensing examination for psychologists, the EPPP.

Dr. Sandy Greenberg, PES's Director of Research Programs, and Dr. Robert Lipkins, Program Director at PES, are here with us today to describe our practice analysis and resulting content delineation and the development of the first two forms of the PEP along with their passing score. Thank you very much for your attention and interest.