Information Society and Library Evaluation Transitions in Portugal: A Meta-evaluation Model and Frameworks (1970–2013)
Leonor Gaspar Pinto, Universidade Nova de Lisboa, Portugal lgpinto@sapo.pt
Paula Ochôa, Universidade Nova de Lisboa, Portugal poc.paula@gmail.com
Abstract

The need for greater understanding of assessment practices and models highlights a deficit of an up-to-date meta-evaluation model, whilst articulating with new phases in Information Society (IS) development. This paper aims to discuss the meta-evaluation model and frameworks that were created to explain the relations between IS transitions and the development of library performance evaluation models in Portugal (1970–2013).

The research is based on a qualitative methodology supported by a combination of literature review with the construction and application of conceptual models and frameworks.

The meta-evaluation model of the impact of transitions on library performance evaluation provides an adequate representation and explanation of relationships between IS transitions and library performance evaluation models. The CLPET (Categorizing Library Performance Evaluation Typologies) Matrix, as well as the relational framework that was developed proved to be useful analytical tools.

This paper highlights the current transition of performance models into a more holistic performance management, clarifying that diverse uses and components of performance are strongly linked to IS dynamics. It will also enable readers to discuss the impact of meta-evaluation models as a strong instrument to support the challenges of visions, strategies and best practices shifted over time.

Key Words
information society transitions; library performance evaluation; meta-models
1. Introduction

In the last 30 years, both library assessment and Information Society (IS) developments have experienced a great and long period of theory building and conceptualization. The most striking feature of performance management is its continuing expansion over the last two decades making this period its international apogee. The 1990s are considered to be the rise of performance, with measurements becoming more intensive, extensive, external (Bouckaert & Halligan, 2006), professionalized and institutionalized (Demmke, 2006). There have been many debates over the value of performance, measurement and its management, specifications of its components and their relationships and applications in different countries, characterized by a constructive period of consolidation and refinement of measurement approaches and instruments. Several performance issues have been researched – performance appraisal, policy planning, external reporting, performance based budgeting and audit – and the crucial question has been how to develop performance measurement in a usable and functional way as a tool for policy and management (Van Dooren, 2005). Another emerging approach is the discussion of theoretical insights into how information is processed and the reasons of its non-use (Van Dooren, 2011). Special attention has also been paid to the way in which decision makers use performance measurement information vis-à-vis their prior understanding, experience and evidences, aligning performance models and evidences of institutional uses, namely ‘good practices’. Recognizing the political nature of performance management and assuming that high quality dialogue will lead to improved judgment and decision-making may strengthen the role of performance in complex and ambiguous contexts (Van Dooren, Bouckaert, & Halligan, 2010).

Among the Portuguese Library and Information Services (LIS) professional and academic community some performance evaluation approaches and models became almost predominant, while other international trends failed to break through. The discussion about evaluation in Portuguese libraries has evolved from pioneering experiences and isolated cases into the notion of best practices with interest to the entire library sector. Libraries are social organizations and, therefore, they can never be static entities. As the environment changes, so must libraries. This means that library evaluation models must also keep pace with the rate of change and, consequently, they have multiple interactions with policy making transitions, suffering several moments of feedback, adjustment, learning and adaptation, all interacting in a variety of ways, risks and opportunities.

According to Van Dooren (2005), a performance measurement policy is the missing link in reforms, but, in general, cultural organizations have not developed advanced performance measurement systems that highlight both cultural/economic performances and their impacts. Bonet and Donato (2011) explain this attitude by different factors: the intrinsic difficulties in measuring a symbolic value; the frequent inability to define consistent performance measurement systems, which results from difficulties in setting the mission and the strategic goals; and the existence of governance systems that are less oriented to the stakeholders, resulting in less attention to the external communication of performances. So, these authors conclude that there are responsibilities, as well as big challenges, for cultural politicians and for cultural management researchers: “There is an ongoing tension between contrasting forces: centralization versus decentralization; organizations’ dependence versus autonomy; general objectives versus specific objectives; extrinsic goals versus intrinsic goals. In that context, the key factors to successfully mastering the crisis are: a) building decision-making processes that should be fluent, non bureaucratic and participatory, and at the same time able to combine responsibility with autonomy; b) moving towards a long term strategic approach defining policy and organizational priorities” (Bonet and Donato, 2011, p. 9).

Understanding libraries’ evaluation related fields cannot ignore the concept of Excellence (European Foundation for Quality Management – EFQM, Business Excellence Model, 2003), closely tied to the development of quality management theories and models and to performance evaluation dynamics in European organizations. The main concepts are focused on result orientation principles, leadership, process management, development and involvement of people, continuous training, partnerships and social responsibility. Organizational maturity and sustainability of results posed new issues related to new meanings and perspectives to convey to another step. Integrated, balanced, multidimensional, comprehensive and holistic models are frameworks used as tools to achieve the excellence stage (ISO 9000; Investors in people; EFQM Excellence Model; Balanced Scorecard; SERVQUAL).

Discussions about evaluation variety are growing, with a particular interest in several research themes:

  1. Quality management models and their forms of development and implementation in organizations (Cullen, 2005; Oakland, 2003; Wilson & Town, 2006).

  2. Main tendencies about economic value[1] (Aabø, 2011; Almauk, 2012).

  3. Leadership styles, benchmarking initiatives and good practices (Poll, 2007; Staes & Thijs, 2010).

  4. Public policies about performance evaluation (Bouckaert, 2012).

  5. Organizational learning.

  6. Critical factors of digital success that promote Excellence (Gonçalves, Moreira, Fox, & Watson, 2007; Huijboom & Van der Broek, 2011; Plum, Franklin, Kyrillidou, Roebuck, & Davis, 2010).

  7. Evidence-based management (Rankin, 2012; Tanner & Deegan, 2012; Town, 2011).

  8. Results and impacts to consolidate new and convergent contexts of Excellence (Poll, 2012; Streatfield & Markless, 2009; Tenopir, 2012).

Library assessment has been mainly about quality rather than about value (Town, 2011) and excellence (Ochôa & Pinto, 2012). In many countries, after the first performance measurement revolution, a second one is needed and will happen soon. New measures linked to institutional objectives are needed and must be communicated: holistic and coherent long-term measures that assess societal benefit, educational impact and intangible assets resources, like relational capital, organizational capital and human and intellectual capital, transferable skills, collections, services, research management, research information, meta content (Town, 2011) and user-defined value metrics of electronic resources (Chew, Stemper, Lilyard, & Schoenborn, 2012). Knowledge management and Competency management are considered important elements of organisational strategy and pillars of success and they are a welcome shift in thinking about these issues.

Measures of success are faceted across several categories (Tanner & Deegan, 2012, p. 42):

  • Users, audience and stakeholders – how libraries benefit them.

  • Innovation and development – evaluating the value of new products and services in terms of how it enables users and providers to do new and innovative actions (new research and teaching methods, web 2.0 type of interactions, new modes of collection development and curation)

  • Internal processes – evaluating how strategy provides value to the way in which the organization operates.

  • Financial benefits and outcomes – performance evaluation.

Value maps, relating library behaviours to end user benefits, form one of the proposals to highlight value. Value mapping goes beyond performance measurement and acts as a performance management of excellence, using and supporting assessments, planning, decision-making, communication, facilitation and evaluation.

This move, from measurement towards value and excellence, is at the centre of concomitant changes for librarians. Research will be required to determine which factors are relevant to different types of libraries in making management decisions.

Influenced by eGovRTD2020 recommendations (Codagnone & Wimmer, 2007) to contemplate holistic research themes, actions and actors, doctoral research carried out by each of the two authors[2] led to a joint meta-evaluation[3] study on the impact of main IS transitions on performance evaluation models implemented in Portuguese libraries, providing evidence for the exploration of new scenarios for theoretical development. As in other countries, in Portugal research related to the impacts of libraries is uncoordinated and the nonexistence of a national coordinating body for libraries and a national research agenda to encourage consistency in evaluation methodologies characterize the major difficulties in establishing regular performance practices. The Information Society (IS) has been developed mainly since 1997 with several governmental strategies focused on qualifications, education and training, social inclusion and accessibility; simplification and better public services, digital contents and services and ITC use on a daily basis. Many strategies followed the European Union agenda, trying to accomplish benchmarking indicators and outcomes. Some transitions in European strategy underline several impacts in institution management that must be examined. This is the case of libraries that developed models in specific contexts related to specific IS factors, using the theories and instrumentalities of evaluation, resulting in a cumulative body of knowledge and skills that must be highlighted.

This project’s core research question was ‘How did major Information Society transitions impact on library performance models in Portugal?’ Based on a qualitative methodology supported by a combination of literature review with the construction and application of conceptual models and frameworks, the research was anchored in two core elements: Information Society and library performance models. Digging further into potential interrelationships and exploring interactions, the following steps were taken:

  1. Identification of the main IS Transitions Cycles in Portugal and their relation to the library context (academic, governmental, specialized, school and public libraries);

  2. Identification of main library performance models – mapping using a categorization matrix;

  3. Analysis and identification of IS impacts on the adoption of models in a holistic reference framework, mapping four main categories.

Figure 1 presents the conceptual (meta-evaluation) model that was designed as an explicative hypothesis.

Fig. 1:

Meta-evaluation model of the impact of transitions on library performance evaluation models.

This research is intended to inform the debate by policy formers of four main kinds: those concerned about the overall quality of services; those concerned with institutional and sector comparisons; those within institutions concerned with interpreting their own performance data appropriately, and those who study information society developments.

The project was presented at the 3rd Qualitative and Quantitative Methods in Libraries International Conference (QQML, 2012) and has already contributed to a series of academic discussions.

2. Information Society and library performance evaluation models in Portugal

Evaluation can be considered as an autonomous scientific discipline. In fact, it is a “transdiscipline” whose subject matter is the study and improvement of tools for other disciplines (Scriven, 1991, p. 365). Decades of Evaluation Research applied to Library and Information Science have resulted in a multiplicity of theoretical and empirical studies, from the pioneer work of Bradford (1948) to the publication of the first international standard on library performance indicators (ISO 11620, 1998) and recent developments in the emerging arena of library impact evaluation (Imholz & Weil, 2007; Markless & Streatfield, 2006; Mays, Tenopir, & Kaufman, 2010; Streatfield & Markless, 2009; among others). Within the library evaluation research corpus, meta-evaluation is still an underexplored area (Calvert, 2008; Pinto, 2012; White, 2002).

As Calvert (2008) points out, “Some authors have described the diversity of library evaluation methods in some detail (...) yet few have attempted to evaluate the variety of evaluation methods or to model library evaluation” (p. 2). Based on Systems Theory, this author and, before him, Nicholson (2004) built a cell matrix to categorize the types of library performance evaluation. Research carried out for the doctoral degree by one of the authors led to the development of their proposals and to the creation of a new and enhanced conceptual framework for categorizing types of library performance evaluation models/approaches – the CLPET (Categorizing Library Performance Evaluation Typologies) Matrix. The categorization is framed by the Matrix’s axes: the y-axis, which uses three possible Perspectives on evaluation (Internal, External and Holistic) and the x-axis, which reflects the evaluation Topic – the Library itself or the Use made of its products and services.

This meta-evaluation approach was employed to identify main performance evaluation models and practice types used in Portuguese libraries from the 1970s to 2010, as shown in Table 1. With the exception of statistics, which in its most rudimentary form can be traced back to the late 19th century, all library performance evaluation models that were categorized in the CLPET Matrix present their creation/start date.

Table 1:

Main performance evaluation models and practice types used in Portuguese libraries (1970–2010).

Pinto (2012) [adapted from Nicholson (2004) and Calvert (2008)]

Legend: PLNN – Public Libraries National Network LMLN – Lisbon Municipal Libraries Network

KRCN – Knowledge and Resources Centres Network

The longitudinal analysis of the context of library performance evaluation in Portugal highlights four main tendencies, which are reflected in the CLPET Matrix content (see Table 1):

2.1. Emergence of user primacy

In Portugal, the first reflections and studies on library use and users’ profile can be traced back to the 1970s. In the next decade, albeit with less vigour than before, User Studies[4] continue to dominate the small Portuguese library performance evaluation corpus. Although there was an active interest in studying the information use and its users in the 90s, it was only with the advent of the next century that user primacy was enhanced, boosted by theoretical and empirical research on users’ digital environment interactions, bibliometrics and impact assessment. The ENTITLE-LMNL Impact Assessment Framework Model reflects this increased attention to library impact on individuals, bringing an extended user perspective into the assessment process. This Model was tested and implemented in the Lisbon Municipal Libraries Network (LMLM) in the context of its participation in the ENTITLE (Europe’s New libraries Together In Transversal Learning Environments) Project, from 2008 to 2009 (ENTITLE, 2009; Pinto, 2009).

2.2. Valorising library performance information

For quite a long time, Portuguese library performance measurement corresponded to irregular collections of input and output statistics, which were considered to be end results with a very vague connection to the management functions. This situation, which characterizes the primordial stage of library service assessment (Shi & Levy, 2005), started to be reversed in the early 1990s, dragged by two initiatives: the development in 1991 of the LMLN System for collecting and reporting performance information and the creation in 1993 of the Public Libraries National Network (PLNN) Model of Performance Information Management System. Both models were anchored in an internal perspective on library performance evaluation centred on both Library and Use topics. The model, as well as the development process and results of the second initiative, were presented and discussed in 1997 at the IFLA Satellite Meeting on Performance measurement and quality management in public libraries (Pinto, 1998) and at the 2nd Northumbria International Conference on Performance Measurement, and were the main responsible for the score 3 (“Government-backed performance measures”) given to Portugal in a world-wide survey on Government interest in library performance measures (Winkworth & Gannon-Leary, 2000).

The first performance evaluation model specifically built for the LIS sector – the Knowledge Resources Centre Network (KRCN) Performance Evaluation Model – was conceived in 2001. Developed and implemented by OQIKS – Observatory of Quality in Information and Knowledge Services, this model was based on four types of references, one of them being the international library standards ISO 11620 and ISO 2789.

This transition period is characterized by the special attention given to the form and format of assessment by a typology of libraries.

2.3. Quality

The introduction of Quality Management in Portuguese libraries is directly related to the evolution of quality in Public Administration, which was strongly conditioned by the accession of Portugal to the European Union in 1986. This quality path was pioneered in 1996 by the Information Unit of the General Secretariat of the Portuguese Ministry of Education (IU-ME) through the development of three main management strategies focused on clients’ needs within the context of an enduring Quality Programme (Ochôa & Pinto, 2006):

  • 1996–2000: Fostering and adapting quality models and methods, namely EFQM – European Foundation for Quality Management’s excellence model and CAF – Common Assessment Framework, the self-evaluation framework recommended for European Public Administration Services. During this period, the IU-ME team carried out two administrative modernization research projects, one of which resulted in the creation of a tailored self-assessment model for public administration information units – MonitorDoc.

  • 2001–2002: Developing and implementing a quality concept linked to Knowledge Management.

  • 2003–2006: Developing and implementing a quality concept linked to competency management.

If we exclude the distinctive case of the IU-ME, only at the dawn of the 21st century Portuguese LIS academics and practitioners started paying more attention to the external perspective on library performance evaluation, under the growing impetus of Quality. This trend was primarily rooted in two initiatives: the first, pursued by OQIKS between 2001 and 2003, was aimed at building KRCN’s quality concept and led to the development of the KRCN Quality Observation Model; the second one was part of a wider strategy targeted at the construction of an evaluation culture in LMLN (2004–2006) and implemented through the LMLN Service Quality Model, a model adapted from Parasuraman, Zeithaml and Berry’s Gap Model (Vinagre, Pinto, & Neves, 2011).

2.4. Integrated models

By capitalizing on previous experiences and reflections on the application of various models and management tools to the LIS field, a new trend centred on a holistic perspective on library performance evaluation came up during the first decade of the 21st century. In fact, the groundbreaking appearance of MonitorDoc (1999) was followed by the design and implementation of several integrated performance evaluation models: LMLN Performance Evaluation Integrated System (2003), SIADAP+B Model (2004), Mix-model CAF-BSC (-AHP) (2005 – Melo & Pires, 2008), Digital Library Integrated Evaluation Model (2006 – Pinto, Ochôa, & Vinagre, 2009) and School Libraries Self-assessment Model (2008 – Portugal. Ministério da Educação e Ciência. Gabinete da Rede Bibliotecas Escolares, 2011).

It was only after 1997 that Portugal developed the main measures towards IS, with political attention to libraries gathered from strategic initiatives in Europe, library programmes and national developments of library automation. Over time and with guidance from e-Europe Action Plans, library services were measured and integrated in sectorial actions to develop citizenship, informational and digital literacy, e-science and digital libraries. Different phases were developed in order to implement an IS agenda, with cycles of technological investment and of modernization initiatives. In periods of technological changes, libraries were a fertile area for generating new services for citizens. In periods of modernization, libraries were faced with strategic changes and some uncertainty and risk in their outcomes. Some researchers see modernization cycles as exponential transformations in evaluation practices and a predominant feature of public services crossroad with New Management, influencing the choice of models and pathways.

The focus on meta-evaluation implies that researchers not only examine library actions, but also their alignment with European orientations and the meanings that such actions have for LIS professionals in underpinning their choices, interactions and use of models. This implies that an integrated micro/meso/macro performance focus is needed due to the reported difficulties for integrating this information in a comprehensive, coherent, consistent and functional way in policy and management cycles (Bouckaert & Halligan, 2006; Demmke, 2006; Van Dooren, 2011).

Our research was designed to meet these challenges and led to the identification of four major transitions in IS library environmental scanning:

  • Transition I – started with the Schwencke Resolution and its impact on national library policies;

  • Transition II – started with the Bangemman Report and its impact on national IKS policies;

  • Transition III – started with the Lisbon Strategy and its impact on quality strategies and on the e-Europe agenda;

  • Transition IV – started with the European Digital Agenda, focused on new trends for services to citizens and enterprises.

3. Transitions and impacts on library performance evaluation models in Portugal

Aiming at a better understanding of the relationship between transitions and performance evaluation models, the CLPET Matrix was combined with another approach – Bouckaert and Halligan’s Four models of performance (2006) – that represents the historical development of performance and management as a basis for analysing and comparing country orientations to performance in four components: measuring, incorporating, using and limitations. According to Bouckaert and Halligan (2006), contextual analysis of different performance management systems is a significant element since it enables comparisons across countries, while acknowledging the importance of their cultural and administrative traditions.

These authors identified three country approaches: “those that have made a high commitment to performance management, where the interest is in how they draw up their performance management systems handle the limitations to this approach; countries that have sought to balance performance management with other features and how that plays out in practice; and countries that have sought to selectively draw on performance management techniques while operating within another type of system” (p. 14).

This framework proved to be useful in gathering evidences of several impacts of IS strategies on four major transitions in libraries’ evaluation models grouped into four cycles of recognition, reinforcement, uncertainty and weakness (see Table 2), summarizing a list of dimensions that define Portugal as a country that has sought to balance performance management with other priorities and political and economic influences.

Table 2:

IKS landmarks and major transitions v/s main Portuguese performance evaluation models.

Legend: Perspectives on library performance evaluation:I-Internal,E External,HHolistic.

The national and European contexts influence the four transition cycles selected:

3.1. Transition I (1984–1994)

Cycle 1986–1994transition to EU performance trends. – We see a phase of recognition of evaluation processes inspired by European libraries. All types of libraries are listed and participate in several plans. The National Library has a particular role as leader of the change process. User needs and satisfaction methods are recognized as management practice and several diagnostic studies of Portuguese libraries are carried out in order to collect information.

One example is the LMLN System for collecting and reporting performance information in public libraries (1991). However, uncertainty and lack of training is assumed by the generality of LIS professionals.

3.2. Transition II (1994–2000)

Cycle 1994–2000transition to Information Society national agenda. – This process has its major evidence in the recognition of libraries as partners in the process of building the Information Society in Portugal: public libraries have a strong participation in this phase; governmental libraries try to explore quality models and benchmarking through the Monitordoc self-evaluation model (1999). Quality models like the Common Assessment Framework (CAF) are disseminated mainly after 2000 and many academic and specialized libraries developed successful experiences of self-evaluation.

3.3. Transition III (2000–2010)

Cycle 2000–2005transition to e-government policy. – Libraries reinforce their role in the conceptualization and development of performance evaluation models. This emphasis is due to the national policy on public services assessment and the creation of the SIADAP+B model (2004) is one of its sectorial outcomes. This model is intended to be applied in all types of Portuguese libraries. This transition is characterized by the growing interest of academic libraries in quality management and benchmarking.

Cycle 2005–2010transition to the national Technological Plan and strong alignment with i2010 strategy – Libraries are considered mainly as information resources for usage. School libraries play an important role in the national pool of performance evaluation of best practices.

3.4. Transition IV (2010-)

Cycle 2010-transition to a digital agenda in a time of financial crises. – The sector is facing a very relevant challenge. There is a need to re-think the cultural system today in the face of a global and national economical crisis. The role of libraries is uncertain and is dependent upon the performance evaluation orientation provided by national governmental structures. At the same time, opportunities to explore, develop and exchange experiences are scarce. Measuring quality is not as important as before and services are interested in developing approaches centred on resources to prove efficiency. In contrast to previous decades, Excellence is now a new and emerging phenomenon in libraries, not only with organizational impacts but also with a strong emphasis on information society transition moments. This involves an interdisciplinary view (Management, Sociology, Information Science, Communication, and Marketing) that examines the confluence of different variables on evaluation methods and practices. One of those studies is concerned with convergence culture and information environments, considering hybridization as a process of integration and fragmentation, particularly during periods of unusual transition, contingency and negotiability. We believe like Hobohm (2012, p. 8) that “we find a concept of quality which indeed stems from the warehouse paradigm and is not rooted in modern customer-oriented quality management”.

The construction of the Single Information Market, the Digital Agenda and the Agenda for Culture requires a different understanding of how the new cycles work and how evaluation models can help libraries to survive and advance. Impact metrics are also a new theme related to information management developments. However, due to financial priorities, information-documentation qualifications and skills are less valued in the market and some signs of weakness in the application of evaluation models become progressively more apparent. One of these signs is visible when in governmental libraries managers are chosen from other professional fields, ignoring librarians’ performance, leadership and importance. This trend is new in Portugal and reflects a transition to invisibility and loss of career among librarians.

4. Discussion and final remarks

Information Society and library performance evaluation pathways can be seen as being separate areas of interest and separate lines of research. Our meta-evaluation model and frameworks aim at linking and integrating different views and practices over time, as tools to analyse the different cycles of IS public policy transitions and at outlining an agenda for joint research. They do not attempt to cover all the phenomena but, on the other hand, by adopting a longitudinal approach to library performance evaluation, at national or international levels, LIS academics and practitioners gain a better understanding of the present situation, as well as a better chance of making in the future more informed decisions regarding library performance management and evaluation policy. These dimensions provide comprehensive coverage for future multi-dimensional IS constructs concerning library management models, assessment trends and improvement tools. The data collected from this framework are the prerequisite for further modelling and for the implementation gap analysis of performance models.

There is plenty of scope for further study. To complement this examination, we suggest the use of SWOT analysis to clarify and evaluate the importance and relevance of IS developments identified in regard to assessment implementation modalities and PEST analysis to complement the cross-border contexts and evidence-based practices, making visible the multitude of excellences that can be found in multiple evaluation models.

The way forward to achieving the dream of a modern society implies that the library sector highlights the significant performance improvements to create and sustain the evaluation environment that is needed for enabling and encouraging libraries to become best practice organizations.

References
Aabø, S. (2011). The value of public libraries: a socio-economic analysis. In Belloti, M. (Ed.), Verso un’economia della biblioteca: Finanziamenti, programmazione e valorizzazione in tempo di crisi (pp. 169–176). Milano: Bibliografia.
Almauk (2012). Economic impact toolkit. Retrieved November 3, 2012, from http://almauk.org/working-together/our_activity/economic-impacts.
Bonet, L., & Donato, F. (2011). The financial crisis and its impact on the current models of governance and management of the cultural sector in Europe. ENCATC Journal of cultural management and policy, 1, 4–11. Retrieved November 13, 2013, from http://www.encatc.org/pages/fileadmin/user_upload/Journal/JOURNAL_ART_BONET_DONATO_2011.pdf.
Bouckaert, G., & Halligan, J. (2006, September). A framework for comparative analysis of performance management. Paper presented to the Study Group on Productivity and Quality in the Public Sector at the Conference of European Group of Public Administration, Universitá Bocconi, Milan. Retrieved November 13, 2013, from http://webh01.ua.ac.be/pubsector/milan/bouckaert_halligan.pdf.
Bouckaert, G. (2012). Managing performance: international and national trends. Forum AP, Roma, May 17th. Retrieved September 27, 2012, from http://www.qualitapa.gov.it/fileadmin/mirror/i-valperf/materiali/2012-05-17_VdP_Slide_Bouckaert.pdf.
Bradford, S. C. (1948). Documentation. London: Crosby Lockwood.
Calvert, P. J. (2008). Assessing the effectiveness and quality of libraries. Wellington: Victoria University of Wellington. Retrieved July 7, 2011, from http://researcharchive.vuw.ac.nz/bitstream/handle/10063/1045/thesis.pdf?sequence= 1.
Chew, K., Stemper, J., Lilyard, C., & Schoenborn, M. (2012) User-defined valued metrics for electronic journals. Library Assessment Conference: Building Effective, Sustainable, practical assessment. In: Proceedings of the 2012 Library Assessment Conference, Charlottesville, VA, Oct 29–31. Retrieved November 13, 2013, from http://eprints.rclis.org/18727/1/UserDefinedValueBasedMetricsForElectronicJournalsPDF.pdf.
Codagnone, C., & Wimmer, M.A. (2007). Roadmapping eGovernment research: visions and measures towards innovative governments in 2020. Clusone: MY Print. Retrieved November 13, 2013, from http://air.unimi.it/bitstream/2434/171379/2/FinalBook.pdf.
Cullen, R. (2005). Operationalising the Focus/Values/Purpose matrix: a tool for libraries to measure their ability to deliver service quality. Presentation at the 6th Northumbria Performance Measurement Conference, August 2005, Northumbria University. Also (2006) Performance Measurement and Metrics, 7 (2), 83–99. doi:10.1108/14678040610679470.
Demmke, C. (2006). Governmental, organisational and individual performance. Performance myths, performance ‘hype’ and real performance. EIPASCOPE, 1, 5–12. Retrieved November 13, 2013, from http://aei.pitt.edu/6370/1/Scop06_1_1.pdf.
ENTITLE (2009). Impact Assessment Framework. Retrieved March 12, 2011, from http://www.entitlelll.eu/eng/Assessment-Framework.
Gonçalves, M.A., Moreira, B.L., Fox, E.A. & Watson, L.T. (2007). What is a good digital library? A quality model for digital libraries. Information Processing & Management, 43 (5), 1416–1437. doi:10.1016/j.ipm.2006.11.010.
Hobohm, H.-C. (2012). Can digital libraries generate knowledge? Contribution to the Cologne Dialogue on Digital Humanities. Retrieved November 13, 2013 from http://www.cceh.uni-koeln.de/files/Hobohm_final.pdf.
Huijboom, N., & van den Broek, T. (2011). Open data: an international comparison of strategies. European Journal of e-practice, 12 (March/April), 1–13. Retrieved November 13, 2013, from http://www.epractice.eu/files/European%20Journal%20epractice%20Volume%2012_1.pdf.
Imholz, S., & Weil, J. (2007). Worth their weight: an assessment of the evolving field of library valuation. New York: Americans for Libraries Council. Retrieved July 19, 2012, from http://www.ala.org/ala/research/librarystats/worththeirweight.pdf.
Markless, S., & Streatfield, D. (2006). Evaluating the impact of your library. London: Facet Publishing.
Mays, R., Tenopir, C., & Kaufman, P. (2010). Lib-Value: measuring value and return on investment of academic libraries. Research library issues: a bimonthly report from ARL, CNI, and SPARC, No. 271 (August 2010). Retrieved June 2, 2012, from http://works.bepress.com/cgi/viewcontent.cgi?article=1135&context=carol_tenopir.
Melo, L.B., & Pires, C. (2008). Performance evaluation of academic libraries: implementation model. Paper presented at the 17th Hellenic Conference of Academic Libraries, Ioanina (2008), Retrieved May 2, 2012, from: http://eprints.rclis.org/bitstream/10760/12374/1/b2.Melo_Pires.pdf.
Nicholson, S. (2004). A conceptual framework for the holistic measurement and cumulative evaluation of library services. Journal of documentation, 60 (2), 164–182. doi:10.1108/00220410410522043. Retrieved November 13, 2013, from http://www.emeraldinsight.com/journals.htm?issn=0022-0418&volume=60&issue=2&articleid=864243&show=pdf.
Oakland, J.S. (2003). Total quality management: text with cases. 3rd ed. Oxford: Butterworth-Heinemann.
Ochôa, P. (2012). Transições profissionais na Sociedade de Informação em Portugal: percursos identitários e ciclos de competências de bibliotecários portugueses (1973–2010). Alcalá de Henares: Universidad de Alcalá. PhD thesis. Retrieved November 13, 2013, from http://dspace.uah.es/dspace/bitstream/handle/10017/17109/TESIS%20PAULA%20OCHOA.pdf?sequence=1.
Ochôa, P., & Pinto, L.G. (2006). Quality - an on-going practice and reflection in a governmental library (1996–2006). In IFLA General Conference and Council, 72, Seul. Retrieved November 13, 2013, from http://archive.ifla.org/IV/ifla72/papers/078-Ochoa_Pinto-en.pdf.
Ochôa, P., & Pinto, L.G. (2012). Contextos emergentes da excelência. In Congresso Nacional de Bibliotecários, Arquivistas e Documentalistas, 11, Lisboa, 2012, Integração, Acesso e Valor Social: actas. Lisboa: B.A.D. Retrieved July 3, 2013, from http://www.bad.pt/publicacoes/index.php/congressosbad/article/view/299.
Pinto, L.G. (1998). Portugal. In IFLA Satellite Meeting, Berlim, 1997. Performance measurement and quality management in public libraries: proceedings (pp. 110–119). Berlin: Deutsches Bibliotheksinstitut.
Pinto, L.G. (2009). Testing the Impact Assessment Framework in Lisbon Municipal Libraries Network. Presentation at the Budapest Entitle Conference on ‘Libraries for Lifelong Learning’. Presentation can be retrieved from http://entitlelll.eu/hun/Meetings/Final-Conference/Presentations/Testing-the-Impact-Assessment-Framework-Lisbon.
Pinto, L.G. (2012) – Dimensões estratégicas e emergentes da avaliação do desempenho: modelos, dinâmicas e impactos na Rede de Bibliotecas Municipais de Lisboa (1989–2009). Alcalá de Henares: Universidad de Alcalá. PhD thesis. Retrieved November 13, 2013, from http://dspace.uah.es/dspace/bitstream/handle/10017/17141/TESIS%20LEONOR%20GASPAR%20PINTO.pdf?sequence=1.
Pinto, L.G., Ochôa, P., & Vinagre, M.H. (2009). Integrated approach to the evaluation of digital libraries: an emerging strategy for managing resources, capabilities and results. In M. Heaney (Ed.), Library statistics for the 21st century world (pp. 273–288). München: K. G. Saur.
Plum, T., Franklin, B.R., Kyrillidou, M., Roebuck, G., & Davis, M. (2010). Measuring the impact of networked electronic resources: developing an assessment infrastructure for libraries, state, and other types of consortia. Performance measurement and Metrics, 11 (2), 184–198. doi:10.1108/14678041011064098.
Poll, R. (2007). Benchmarking with quality indicators: national projects. Performance Measurement and Metrics, 8 (1), 41–53. doi:10.1108/14678040710748076.
Poll, R. (2012). Can we quantify the library’s influence? Creating an ISO standard for impact assessment. Performance Measurement and Metrics, 13 (2), 121–130. doi:10.1108/14678041211241332 Retrieved November 13, 2013, from http://www.emeraldinsight.com/journals.htm?issn=1467-8047&volume=13&issue=2&articleid=17041642&show=pdf.
Portugal. Ministério da Educação e Ciência. Gabinete da Rede Bibliotecas Escolares (2011). Modelo de avaliação da biblioteca escolar. Lisboa: Gabinete RBE. Retrieved September 19, 2011, from http://www.rbe.min-edu.pt/np4/83.html.
Rankin, C. (2012). The potential of generic social outcomes in promoting the positive impact of the public library. Evidence from the National Year of Reading in Yorkshire. Evidence Based Library & Information Practice, 7 (1), 7–21. Retrieved November 13, 2013, from http://ejournals.library.ualberta.ca/index.php/EBLIP/article/view/11727/13250.
Scriven, M. (1991). Evaluation thesaurus, London: Sage.
Shi, X., & Levy, S. (2005). A theory-guided approach to library services assessment. College & research libraries, 66 (3), 266–277. Retrieved November 13, 2013, from http://crl.acrl.org/content/66/3/266.full.pdf+html.
Staes, P., & Thijs, N. (Eds.) (2010). Growing Towards Excellence in the European Public Sector: A decade of European collaboration with CAF. Maastricht: EIPA. Retrieved November 13, 2013, from http://www.aeval.es/caf2006/docs/10_years_of_CAF.pdf.
Streatfield, D., & Markless, S. (2009). What is impact assessment and why is it important?. Performance measurement and metrics, 10 (2), 134–141. doi:10.1108/14678040911005473. Retrieved November 13, 2013, from http://www.emeraldinsight.com/journals.htm?issn=1467-8047&volume=10&issue=2&articleid=1819366&show=pdf.
Tanner, S., & Deegan, M. (2012). Inspiring research, inspiring scholarship. London: JISC. Retrieved November 13, 2013, from http://www.kdcs.kcl.ac.uk/fileadmin/documents/Inspiring_Research_Inspiring_Scholarship_2011_SimonTanner.pdf.
Tenopir, C. (2012) Beyond usage: measuring library outcomes and value. Library Management, 33 (1/2), 5–13. doi:10.1108/01435121211203275. Retrieved November 13, 2013, from http://www.carl-abrc.ca/uploads/library_articles/Tenopir.pdf.
Town, S. (2011). Value, impact and the transcendent library: progress and pressures in performance measurement and evaluation. Library Quarterly, 81 (1), 111–125. doi:10.1086/657445. Retrieved November 13, 2013, from http://www.jstor.org/stable/10.1086/657445.
Van Dooren, W. (2005). What makes organisations measure? Hypotheses on the causes and conditions for performance measurement. Financial accountability & management, 21 (3), 363–383. doi:10.1111/j.0267-4424.2005.00225.x. Retrieved November 13, 2013, from http://steunpuntbov.be/rapport/s0406007.pdf.
Van Dooren, W. (2011). Why is performance information not used? In search for some common barriers. Paper presented at the 2011 ECPR general Conference (Reykjavik), University of Antwerp, Belgium.
Van Dooren, W., Bouckaert, G., & Halligan, J. (2010). Performance management in the public sector. London: Routledge.
Vinagre, M.H., Pinto, L.G., & Neves, J. (2011). Giving voice to children and young adults: a methodological approach to the evaluation of service quality in libraries. In Northumbria International Conference on Performance Measurement in Libraries and Information Services (PM8), 8, Florence (2009). Libraries plus: adding value in the cultural community: proceedings (340–358 pp.). Newcastle upon Tyne: Northumbria University Press.
White, L.N. (2002). Does counting count: an evaluative study of the use and impact of performance measurement in Florida public libraries. Tallahassee: Florida State University.
Wilson, F., & Town, J.S. (2006). Benchmarking and library quality maturity. Performance Measurement and Metrics, 7 (2), 75–82. doi: 10.1108/14678040610679461. Retrieved March 6, 2012, from: http://bura.brunel.ac.uk/bitstream/2438/408/1/e-print%20of%20PMM72.pdf.
Wilson, T.D. (2000). Recent trends in user studies: action research and qualitative methods. Information research, 5 (3). Retrieved March 3, 2011, from http://informationr.net/ir/5-3/paper76.html.
Winkworth, I., & Gannon-Leary, P. (2000). Library performance measures: government perspectives. In Northumbria International Conference on Performance Measurement in Libraries and Information Services, 3, Northumberland, 1999 – Value and impact: proceedings (pp. 61–65). Newcastle: Information North.
Notes

Cost efficiency, cost effectiveness, cost comparison, financial allocation, contingent valuation and value added return on investment.

L.G. Pinto (2012) investigated in her doctoral research performance assessment impact on Portuguese Libraries and P. Ochôa (2012) investigated in her PhD research professional trajectories in information society transitions. In earlier research, the authors have categorized quality dimensions in Portuguese libraries, which formed the foundation of their current research.

Meta-evaluation is the evaluation of an evaluation or cluster of interconnected evaluations (Scriven, 1991).

According to Wilson (2000), “...’user studies’ is a term which covers a very wide range of potential research, from the study of users’ choices of books from a university library, through reactions to on-line search outputs, to the in-depth analysis of the underlying needs that result in information-seeking”.



Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.