Training the trainers: teaching clinician educators to provide information literacy skills feedback*[dagger] (EC)

Publication: Journal of the Medical Library Association
Author: Maggio, Lauren A
Date published: July 1, 2011


"Competency-based education focuses on learner performance in reaching specific objectives goals and objectives in the curriculum" [1]. At the time of this writing, the Stanford University School of Medicine (SOM) was in the process of developing a comprehensive competency-based curriculum with an emphasis on providing students with feedback on their performance. Feedback is defined as "information about previous performance that is used to promote positive and desirable development" [2]. As part of developing the new curriculum, a librarian-medical doctor (MD) team at Lane Medical Library at Stanford University Medical Center created and implemented an interactive, competency-based information literacy (IL) curriculum that included opportunities for students to attain and practice IL skills and to receive feedback on their efforts. The curriculum trains both students and faculty in an attempt to synchronize the IL skills of both groups and to encourage the faculty to act as information literate role models.

There is an extensive literature on the importance of feedback for medical students and trainees [3, 4]; however, these concepts are absent in the literature of medical librarianship. This report describes an effort to train clinician educators to provide medical students with training and feedback on IL skills. Medical students' perceptions of the impact of faculty feedback on their overall development was also measured.


At Stanford, the IL curriculum was jointly created and led by the authors, one of whom is Lane's medical education librarian and liaison to the SOM and the other of whom is chief of inpatient medicine at the Palo Alto Veteran's Hospital and clinical assistant professor of medicine at Stanford SOM and dedicates 10% of his time as Lane's chief clinical informationist. In spring 2008, the authors began to integrate an IL curriculum into "Practice of Medicine" (POM), the SOM's 2-year pre-clerkship clinical skills curriculum. While designing the formal IL curriculum, the powerful role of the "informal curriculum" - defined as "an unscripted, predominantly ad hoc, and highly interpersonal form of teaching and learning among and between faculty and students" [5] - was also considered. By increasing the number of faculty able to role model IL skills, the hope was to harness the informal curriculum to improve students' IL skills.

The Stanford SOM Educators-4-CARE (E4C) faculty was selected as the group to provide the desired feedback to medical students. The E4C members, 15 clinician educators from a variety of medical specialties, are paired with each of Stanford's 82 incoming medical students to serve as faculty role models who can foster their development into skilled and compassionate physicians. E4C faculty members dedicate 20% of their time to mentoring students and are provided ongoing faculty development opportunities, such as workshops on communication skills, the physical exam, and IL. Therefore, training EC4 faculty to be information literate role models should enable them to positively and longitudinally instruct and provide feedback to students in this domain.

Stanford University School of Medicine Educators-4-CARE faculty training

In the summer and fall of 2008, the authors led two nmety-minute IL training sessions for E4C faculty. The purpose of the first session was to define IL for the faculty and help them understand the relationship between IL and evidence-based practice (EBP). This session provided the instructors with information on faculty's current IL skill level and helped to focus future faculty development in this area. Prior to this session, faculty had completed an online case-based assessment designed to gauge their skills in IL. In the case, faculty were challenged to articulate a caserelated question, locate and appraise information related to their questions, and briefly discuss the application of the found information. In the first inperson, faculty-development session, the original case was discussed; relevant information resources, such as the Cochrane Library, were highlighted; and, through interactive discussion, faculty members were encouraged to consider this exercise in the context of IL.

Two months later, the second nmety-rninute, inperson, faculty-development session took place. This session presented an abbreviated version of the "Introduction to IL" session that SOM first-year students would receive in the following weeks; more importantly, the session introduced E4C faculty to the upcoming in-class IL exercise that students were to complete as part of the "Introduction to IL" class. The exercise to be completed in class and reported using an online survey included a case and series of questions designed to capture the strategies that students used to locate information to answer a case-based question. The exercise required students to report (1) search tools and terms used, (2) the title and uniform resource locator (URL) of the resources used to answer the question, and (3) their judgment of the quality of their chosen resources, coupled with rationale for their decision. Upon completion, student answers were electronically submitted to the instructors.

The second E4C faculty development session began with faculty individually completing the original student assignment. Using two of the faculty's responses, the instructors modeled how to provide feedback on the assignment. Next, E4C faculty received a handout featuring a sample student response. The faculty formed dyads to practice giving feedback in the modeled style. This exercise was followed by a group debriefing of the feedback process, specifically as it pertained to this assignment. Finally, faculty were coached in the use of an online form designed to provide feedback to students for the upcoming student assignment. Forms were designed to facilitate the modeled feedback style, and faculty were asked to rate and comment on: (1) if the example student search was flexible, (2) if the chosen search terms were applicable, (3) if the information the student found answered the question, and (4) if the faculty agreed with the student's judgment on the quality of the identified resource. For example, when asked to rate the flexibility of the student's search, faculty were provided the choices: flexible (student tried several search resources and terms to find the best information); somewhat flexible (student used more than one search resource and term); and not flexible (student used a single search resource and term). Each question also featured a text box for further comments.

Student training

The student tiaining session, "Introduction to IL" (Table 2, online only), was held on October 27. Seventy-nine of the 82 students (96%) attended the session and submitted in-class assignments. Each student received 115 minutes of hands-on instruction and had access to a computer. Students were first presented the case of a patient concerned about breast cancer and, without instructional intervention, were asked to locate information to answer a related question using whatever information resources they felt appropriate to answer their question. Students recorded their search process on a provided form, indicating information resources used, their search terms, and the final citation. Students were asked to judge whether or not the information source selected for the final citation was a "quality" resource and to briefly defend their decision. Upon completion of this exercise, students electronically submitted their assignments to the instructors and were provided a summary of their answers (Appendix, online only).

Following this first case assignment, the instructors engaged the students in a discussion of the exercise and demonstrated related library resources, such as etextbooks, Lane Library's website, and PubMed. Next, students repeated the exercise with a second case on anemia with instructions to utilize the resources that their instructors and classmates introduced in the session. Upon completing the assignment, their data were transmitted to the instructors and the session concluded.


On November 5, the fifteen faculty mentors received their students' submissions to the second case via email. These materials were sent by the E4C administrative staff with a note from the E4C director stating that it was expected that all E4C faculty would complete feedback for their students. At this time, the handouts from the earlier October training session were also sent to faculty who were invited to contact the instructors with any questions.

Fourteen of the faculty (93%) returned feedback on their students' submissions on November 28. Of the faculty who submitted feedback, 13 (87%) answered all 4 of the multiple choice questions on the feedback forms. The single facilitator who skipped answering the multiple choice questions provided only free-text comments for students. Ten of the responding faculty (71%) provided their students with free-text comments. A single E4C faculty member did not provide feedback; in that instance, feedback was completed by the librarian instructor and students were notified of the feedback's origins. Two faculty members asked the librarian to review their feedback. Both faculty members were assured that their feedback was appropriate. Faculty feedback was reviewed and transmitted within 48 hours to the students via an email that included the students' responses to the inclass assignment and their faculty members' related feedback. All feedback was sent to students by November 30, ensuring adequate time to review faculty feedback before the POM exam on December 9, which included 4 IL-related questions.

Following the final exam, all students were emailed a 4-question survey to gauge students' perceived efficacy of the initial "Introduction to IL" session and the related faculty feedback. The survey included the 2 questions: "Regarding your E4C facilitator's feedback, was it helpful in preparing for the final exam?" and "Regarding your E4C facilitator's feedback, was it helpful overall?" Twenty-four students (29%) responded to the survey. However, only 22 students (27% overall) answered the questions related to faculty feedback. The following data are for survey respondents only (not including those who missed the session or received feedback from the librarian). In relation to preparing students for the exam, 4 (18%) students reported that the faculty feedback was helpful, whereas 9 (41%) students felt that it was somewhat helpful and 9 (41%) students not helpful. Five (23%) students found the feedback to be helpful overall; however, 9 (41%) found it to be somewhat helpful overall and 8 (36%) not helpful overall (Table 1).


Employing a train-the-trainer model, this project attempted to use faculty development to improve the feedback that students received regarding their IL skills. The project achieved moderate success, with 59% of students finding the feedback at least somewhat helpful in preparing them for the exam, and 64% finding it to be at least somewhat helpful overall. However, faculty feedback was variable. For example, although all students received feedback based on the multiple choice options, only 10 of the faculty provided their students with personalized comments, which ranged from a few words to a paragraph. Several factors may have accounted for the varied level of faculty feedback, including busy schedules and lack of confidence in providing feedback in this domain, which may be alleviated through additional feedback trainings. Future studies may examine whether the form feedback takes (multiple-choice feedback versus free-text comments) influences students' perception of helpfulness. Lastly, the faculty were not surveyed regarding their perception of the usefulness of the instruction in relation to providing feedback to students in this domain. Investigating faculty perceptions might provide information that could improve future faculty training.

Overall the students' survey responses, which indicated that most students found the feedback to be only somewhat helpful, demonstrated that they did not find the feedback component of the IL curriculum to be of significant benefit, and this curricular activity has been discontinued. Two main reasons might account for this lack of benefit: (1) Effective feedback is linked to goal setting [6]. The sessions reported here did not state concrete competencies or goals. Since this training, Stanford IL clinical competencies have been created, and these could potentially serve as reference points in the future. (2) Working with our faculty, it was clear that many of them were not experts in EBP or IL. Ninety minutes of faculty development was clearly insufficient to fully train faculty in IL skills, meaning that faculty were not fully prepared to provide optimal feedback. This finding illustrates a major barrier to building a truly integrated IL/EBP curriculum in a school of medicine with a large and diverse faculty and demonstrates the need for more robust and effective faculty development.


Several factors influenced the decision to engage Stanford E4C faculty in providing students feedback on their IL skills, including the importance of feedback in medical education and the desire to provide tailored timely feedback. Over a five-month period, a cohort of clinician educators was trained to provide feedback on their application of IL skills to an online assignment. A brief survey of the students found this faculty feedback to be moderately helpful, both overall and in preparing for their final exam. Looking to the future, we will continue to train and work closely with the E4C faculty to improve their IL and related feedback skills.


The Lane Medical Library acknowledges the efforts of the Educators-4-CARE. Special thanks are given to Kristin Fabbro and Drs. Preetha Basaviah and Lars Osterberg.

* Based on a presentation at MLA '10, the 110th Annual Meeting of the Medical Library Association; Washington, DC; May 24, 2010.

[dagger] Usage of the following data is covered under a protocol approved by the Stanford University School of Medicine Institutional Review Board.

(EC) A supplemental appendix and Table 2 are available with the online version of this journal.


1. Accreditation Council for Graduate Medical Education. Outcome project, 2002 [Internet]. The Council [2002; cited 10 Nov 2010]. < proHome.asp>.

2. Archer JC. State of the science in health professional education: effective feedback. Med Educ. 2010 Jan; 44(1):101-8.

3. Ende J. Feedback in clinical medical education. JAMA. 1983 Aug 12;250(6):777-81.

4. Hewson MG, Little ML. Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med. 1998 Feb;13(2):111-6.

5. Hafferty FW. Beyond curriculum reform: confronting medicine's hidden curriculum. Acad Med. 1998 Apr;73(4): 403-7.

6. Locke E, Gary L. A theory of goal setting & task performance. Englewood Cliffs, NJ: Prentice Hall; 1990.

Author affiliation:

Lauren A. Maggio, MS (LIS), MA, AHIP; Keith A. Posley, MD, MS

See end of article for authors' affiliations.

DOI: 10.3163/1536-5050.99.3.014

Author affiliation:


Lauren A. Maggio, MS (LIS), MA, AHIP (corresponding author),, Medical Education Librarian; Keith A. Posley, MD, MS,, Chief of Inpatient Medicine, Veteran's Affairs Palo Alto Health Care System, and Chief Clinical Informationist; Lane Medical Library and Knowledge Management Center, Stanford University Medical Center, 300 Pasteur Drive L-109, Stanford, CA 94305-5123

Received September 2010; accepted January 2011

The use of this website is subject to the following Terms of Use