Evaluate programs and services on measurable criteria.
“Libraries will get you through times of no money better than money will get you through times of no libraries.”
–Anne Herbert
EXPLICATION
The value of any library to its community is largely based on the programs, services, and resources it offers. Programs, services, and resources offered by any library are established based on an explicit set of goals and objectives. However, as a library’s community and/or resources change, changes in programs, services and resources may be necessary for the library to remain a vital part of the community. Evaluation is a critical and continual process necessary to discover the “success, impact, results, costs, outcomes,” etc. of library programs, services, or resources as well as to identify shortcomings and determine areas for improvement (McClure, 2008, p. 179). Evaluation of programs, services, and resources is intrinsically tied to the planning process. Programs and services must be framed on clearly established goals and objectives that reflect the mission and vision of the organization, whether those of the library itself or those of the parent organization. Goals and objectives provide the basis for evaluation. Evaluation results may be used to inform and/or justify continuation, termination, or budgetary decisions associated with programs, services, and resources as well as for selecting and developing future programs and services. Evaluation results may be presented to internal and/or external stakeholders.
Evaluation planning should establish specific assessment criteria. In other words, it is important to “understand the scope of the evaluation” in order to ensure that appropriate data is collected (McClure, 2008, p. 183). Evaluation planning should answer such questions as:
- What staff will take part in the evaluation? Do any of the staff have prior experience in evaluating programs, services, or resources?
- Who will be the chair or point-person for the evaluation process?
- What will be evaluated and why?
- What type of data will be collected? How will it be collected?
- To whom will results be presented? How will results be presented?
Evaluation planning should also determine resource and budgetary requirements as well as establish a time line for assessment. All of these criteria potentially effect the degree to which an evaluation is successful. Measures may consider a program or service’s performance, output (results), and/or outcome (impact).
Evaluation may be formative or summative. Formative evaluation typically occurs during the program planning stage but may also take place on an ongoing basis throughout the program (McClure, 2008). Formative evaluation may involve soliciting feedback from other staff members to detect shortcomings or flaws and provide suggestions for improvement before a new program or service is implemented. It may involve “test-runs” of new online tools or software in order to determine what type of instruction for use may be helpful to patrons. Formative evaluation also may be used to provide preliminary data against which summative results can be compared. For example, evaluation of information literacy instruction sessions may require pre-testing of participants in order to establish the level of information literacy of individuals at the beginning of the session. This can be compared to post-session results in order to get an idea of the effectiveness of the information literacy session. It may also involve continual testing throughout the program in order to determine what parts of a learning session are most effective. Summative evaluation occurs at the end of a program or service and helps determine the degree to which goals and objectives have been met. Data used for summative evaluation may be qualitative or quantitative; ideally it should be a combination of both.
Data may be collected in a variety of ways including surveys, one-on-one interviews, live or videotaped observation, and quizzes/tests demonstrating learning. Whenever possible, evaluations from patrons should be anonymous in order to elicit the most honest responses and feedback. Surveys may be provided in print form or online using programs such as Survey Monkey. Survey and interview questions should be worded in a way so as to elicit appropriate data for evaluation. Evaluators should be aware of the quality and limitations of data. McClure (2008) suggest that evaluators “pretest the data collection instruments” and consider what “specific threats might limit the validity, reliability, and usefulness of the data” (p. 187). Effective evaluation is reliant on effective data collection. Whenever possible, libraries should proceduralize data collection, i.e., functions occurring in the library should include procedures for collection and storage of data associated with that function, e.g., maintaining data related to collection usage. Ongoing data collection helps establish a “culture of assessment,” underscoring to staff the importance of data in evaluation of programs and services that provide value to the information organization with regard to the needs and demands of users.
COMPETENCY DEVELOPMENT
My classes in the MLIS program have done an excellent job in conveying the importance of evaluation programs, services, and resources in library settings. I have learned about evaluating technology—hardware, software, discovery tools, websites, etc. (INFO-243, Systems Analysis and INFO-244, Online Searching). I have learned about evaluating services and programs (INFO-210, Reference and Information Services; INFO-250, Design and Implementation of Instructional Strategies for Information Professionals; INFO-282, Correctional Library Management; and INFO-285, Research Methods in Library and Information Science—Evaluating Programs and Services). I have learned how to evaluate information instruction sessions (INFO-250, Design and Implementation of Instructional Strategies for Information Professionals, and INFO-254, Information Literacy and Learning). I have learned about how an institution’s mission and vision statements inform program and service goals, how to design new programs, and how to establish evaluation criteria for those programs (INFO-204, Information Professions; INFO-250, Design and Implementation of Instructional Strategies for Information Professionals; and INFO-285, Research Methods in Library and Information Science—Evaluating Programs and Services). I believe that depth and breadth of what I have learned about evaluation has instilled in me the purpose and value of evaluation as a means to ensuring that the programs, services, and resources offered by any library meet the needs of the service community.
EVIDENCE
1. Instruction Session Observation and Evaluation
Information instruction is an important service that supports and furthers the goal of fostering and instilling information literacy. Information instruction may be as simple as online tutorials offering step-by-step instructions about how to use a specific information discovery tool or how to evaluate information. At the other end of the spectrum information instruction may consist of structured sessions that cover a range of topics relating to information literacy. Information instruction is offered in many types of libraries, particularly academic libraries. The most common form of information instruction offered in academic libraries is the so-called one-shot library orientation session. These sessions are meant not only to introduce students to library resources but also an introduction to the concepts inherent in academic research. Unfortunately, most one-shot sessions “use prepared searches to demonstrate a database, inadvertently leaving students with no idea how common it is for a search to fail and with few clues as to how to handle common search problems in electronic databases” (Keyser, 2000, p. 35). Sessions presented in this manner do little or nothing to teach students anything and often leave them feeling even more anxious and unsure about conducting research. The first item of evidence is from an INFO-254 assignment that required observation of an instruction session and evaluating it based on a set of criteria. I observed a library orientation session conducted at the university I attended as an undergraduate. I evaluated the session using the Association of College and Research Library’s (ACRL) Information Literacy Competency Standards for Higher Education. Evaluation made it clear to me where improvements in the session could be made. The limitations inherent in this evaluation pertain to evaluating the actual effectiveness of the session by measuring in some way the learning that took place.
2. Formative Evaluation
Formative evaluation generally takes place during the creation of a program or service. Formative evaluation allows program designers to detect shortcomings or flaws in a proposed program or service. The next item of evidence is an assignment completed for INFO-250. The culminating project for that class was designing a unit of instruction. For this portion of the assignment, I evaluated instruction tools and materials that would be used in conjunction with the instruction session. This type of formative instruction is important to ensuring that materials and tools used are appropriate and will support and enhance the instruction unit.
3. Strategic Plan: Goals, Objectives, Outcomes, and Measures of Success
A strategic plan “identifies and sets the overall direction of an organization” (Evans and Ward, 2007, p. 149). Strategic plans establish “meaningful, achievable, and quantifiable objectives” (Hirsch 2015, p.406) based on the purpose of the library and define how to implement those objectives as well as establish criteria for measuring progress and evaluating achievement. The final item of evidence provided is a strategic plan designed as the culminating project for INFO-204. The strategic plan is based on a hypothetical library, which in this case is a special library that is part of a children’s museum. Appendix A provides evidence of my ability to craft realistic goals and objects, set a timeline for achieving those goals, and set evaluation criteria for determining when and how well those goals and objectives are met.
CONCLUSION
Stenstrom (2015) asserts, “in an era of increasing scrutiny, and diminishing resources for some organizations, assessment plays a critical role” (p. 272). Any library’s primary function is to serve the members of its community through programs, services, and resources. Programs, services, and resources should be framed around specific goals and objectives. Evaluation must take place to ensure that goals and objectives are met. The discussion and evidence presented demonstrates my understanding and mastery of the concepts involved in evaluating programs and services on measurable criteria.
References
Evans, G. E., & Ward, P. L. (2007). Management basics for information professionals (2nd ed.). NewYork, NY: Neal-Schuman.
Hirsch, S. (Ed.). (2015). Information services today. New York, NY: Rowman & Littlefield.
Keyser, M. W. (2000). Active learning and cooperative learning: Understanding the difference and using both styles effectively. Research Strategies, 17, 35-44.
McClure, C. R. (2008). Learning and using evaluation: A practical introduction. In K. Haycock & B. E. Sheldon (Eds.), The portable MLIS: Insights from the experts (pp. 179-192). Westport, CT: Libraries Unlimited. (Original work published 2008)
Stenstrom, C. (2015). Demonstrating Value. In S. Hirsch (Ed.), Information services today (pp. 271-277). New York, NY: Rowman & Littlefield.
Kristin Lehigh 