Assessment:  Measuring Services and Resources in Academic Libraries

5th Biennial Conference for Academic Libraries in New York State

 

October 2-3, 2003

Sheraton Syracuse University Hotel and Conference Center

 

Panel on LibQUAL+™ Assessment Tool

Pat Schafer, Cornell University

Theresa Maylone, St. Johns University

Denise Stephens, Syracuse University

 

Pat Schafer, Director of Information Services Instruction, Research, and Information Services, Cornell University Library:  The Third Time Around:  What Have We Learned?

 

Cornell has done LibQUAL+™ three times. They participated from the beginning of the research project to design the survey. The experience has been very worthwhile.  Cornell had three reasons to volunteer the first year:  gain experience in survey administration, get useful planning info, and have a chance to affect the outcome. 

 

Survey administration:  The survey team is very important, as wide cooperation is needed throughout the institution.  Also needed is effective communication with disgruntled users.  Project management skills are crucial, and statistical skills are needed.  They communicated with resources on campus for statistical analysis.  ARL website now provides more useful information, but early on institutions needed to do their own analysis.  Skills in assessment analysis, etc., are still useful.

 

Selection of the survey sample is critical. They used the patron database. Library staff skewed response tremendously in the first year, so they were not used in other years. 

 

Communicating with users helps with the survey but also is wonderful PR.  They set up special email addresses for responses, and this resulted in a number of dialogs and questions.  A comments box allowed survey users to vent.

 

Documenting decisions is crucial.  E.g.,  which departments are put in the humanities or social sciences group.  This makes it much easier if surveys are done in subsequent years. 

 

Survey results:

Know your audience.  Don't make decisions based on LibQUAL+™ solely.  The results point you to areas for further study. This type of big survey is good for trend analysis when done over multiple years.  Presentation of data can be good for PR also.  In addition, you can parse out various groups to determine if they are similar or where differences lie.

 

Cornell focused on the desired-perceived gap.  They tried to identify biggest gap areas and do something to get further information or make improvements in these areas.

 

Cornell is leaning more towards data-driven decisions. They are experiencing an attitude shift in the library and find that the fear of doing service surveys is decreasing. 

 

LibQUAL+™ is like a weathervane -- pointing you in the right direction for future decisions.

 

Theresa Maylone, Executive Director, University Libraries, St. Johns University:  LibQUAL:  So What Now?

 

Ms Maylone focused on surveys as a planning tool. 

 

St. Johns University LibQUAL+™ survey results show some problems.  She noted that she had just received a call saying that three new positions have been approved for the library (after loss of 13 positions over the last few years).  The new positions are in information literacy and information technology.  So -- something positive has happened as a result of LibQUAL.

 

2003 was their second year in LibQUAL, and they have signed up for 2004.

 

Each year of a survey may occur in a different environment (new ILS, more computers, wireless, culture changes, substantial enrollment changes, etc.).  In addition, the wider environment may have effects on information seeking behaviors of their consumers.  We need to identify a suite of our own information seeking behaviors to inform changes, or we will be responding to those other external factors.

 

LibQUAL+™ is a powerful resource for a local instrument, comparative measures, and [??one more].  It is multi-method approach, which expands the suite of tools for assessment. 

 

Other types of surveys don't measure expectations of users, while this one is based on the fact that only users can judge quality.  LibQUAL+™ for them is now beginning to have a high level of validity and reliability. Previously they did not ask what students and faculty want to achieve and what outcomes they wanted.  LibQUAL+™ asks these questions and shows what kinds of transformations need to take place.  ARL gives an enormous amount of support for a bargain price. 

 

The 'so what' is the library's to determine.  Use this tool in conjunction with others to help the institution meet its goals.  We need to create a culture of assessment.  This is a major shift for the institution as a whole and a leadership opportunity for libraries. 

 

LibQUAL+™ as a total market survey allows comparison with competitors in provision of information services (e.g. Google).  E-QUAL is under development by ARL for digital libraries. 

 

LibQUAL+™ has value in providing a common language for assessment.  It gives libraries a voice for leadership in the field of assessment.

 

 

 

Denise Stephens, Associate University Librarian for Public Service, Syracuse University:  Introducing Library Assessment As a Control for Change

 

2003 was Syracuse University’s first LibQUAL+™, and it was the first significant library assessment ever done.  It was important because we need to know what users expect and value in order to meet their information needs.  Assessment gives us a framework for changes that need to be made.

 

Why LibQUAL+™ as a tool?

It is tested, authoritative, library-focused (focused on what libraries do), broad in scope, has a user perspective, manageable administration, and is an effective springboard for change.

 

Sometimes the survey confirms what librarians already know, but it is helpful to take the results to the administration, as they are from the user perspective.

 

LibQUAL+™ facilitates the measurement of quality. It can also be a tool for validating what we think we know.  They are finding there are more similarities between user groups than there are differences.  That also means that problems that are identified apply to all groups, thus the problems are especially significant.

 

They have an emerging cultural impact of LibQUAL+™ in the library:

·        The four dimensions are common knowledge

·        Outcomes are integrated into planning

·        Expanded applications of assessment are growing

·        Change is expected

 

They still need more discussion with users, but the broad trends already identified are useful.

 

Institutional impact:

·        Library is increasingly visible

·        Library is gaining credibility

·        Expectations are on the rise.

 

What's next?

Objective is to reinforce the growing culture of assessment by

·        Translating data to measurable goals

·        Translating goals to action with accountability

·        Facilitating participation

·        Documenting and communicating progress

·        Providing and ongoing mechanism to perform and management assessment.  One outcome will be a standing committee.

 

LibQUAL is the first of many steps to integrate assessment into the way we do things.  We expect to learn quite a lot about ourselves and our clientele along the way.

 

Syracuse will not do the survey in 2004 but probably will do it in 2005 after some changes have been made.  In making those changes, we must recognize what we can do ourselves and identify those areas in which we must rely on others for assistance.

 

Questions for the LibQUAL+™ Assessment Tool Panel:

·        Are participants putting results out on the web?  McGill is doing that, saying they had a commitment to improvement.  Syracuse will highlight key areas with what they plan to do.  Some results have been published in their library publication to faculty.  Cornell would like to do something and will be discussing how to do it this year.  St. Johns needs to communicate internally first - they don't have the internal communication that Syracuse seems to.  The red in survey results communicates important information.  It must not be regarded as “bad.”  Institutions must listen to what users say is important to them.  The University at Albany put the entire report up as a whole.

·        The lack of bibliographic instruction questions is a fundamental flaw of LibQUAL+™.  Kathy Miller noted that the committee in New York raised this.  ARL reaction was that bibliographic instruction was included in earlier surveys, and it wasn't that important.  Kathy believes that in the summer meeting ARL indicated they might address this issue again.  Pat noted that this type of survey can't address individual areas.  Theresa commented that it is broad survey looking at user perception and expectations.  Pat reported that Cornell did a specific survey on instruction issues this summer. 

·        Why would a person ever mark any part as less than 9?  Pat doesn't like that aspect of the design.   Others commented that it should show relative importance for respondents

·        (Addressed to Mr.  Ratteray) Has LibQUAL+™ come up in Middle States self study yet?  No, but they are looking forward to seeing them come up.

·        Theresa noted that until this time this has been a pilot study.  Feedback to them has helped to refine the study.  Data gives normative scales to make comparisons. 

·        One director noted the enthusiastic response of a professor who teaches statistics on their campus.  The professor wants students to work on this.  This is good, as their small school does not have an institutional research office.

·        How can institutional data be compared without more knowledge of the groups that participated in the survey?  Pat:  Cornell is not really doing such comparisons except for occasional PR.  Theresa:  the normed data assists with this.

 

 

Notes by Jean Sheviak