EVALUATION OF INTER- AND TRANSDISCIPLINARY RESEARCH
Experiences and reflections on best practice

This is the final of a series of four annual conferences dedicated to practices, methodologies and epistemologies of inter- and transdisciplinary research and teaching. The first conference focussed on the theme of PROBLEM FRAMING as a decisive and determining initial phase of the research process (see td-conference 2008). At the second conference we addressed issues of INTEGRATION that cut across the whole research process (see td-conference 2009), from problem framing and problem analysis to the IMPLEMENTATION of research in a life-world context, which was the topic last year (see td-conference 2010). This year, national and international experts are invited to share their knowledge about evaluation models for inter- and transdisciplinary research proposals, processes and outcomes in their larger scientific, socio-political and cultural contexts.

In contrast to most disciplinary research practices, interdisciplinary and transdisciplinary projects are characterised by a variety of disciplines and various practitioners in professional fields outside academia. As a consequence, general evaluation methods that have been developed for disciplinary work can only be used to assess specific disciplinary contributions but are hence not appropriate for assessing specific inter- or transdisciplinary goals, processes and achievements. Major challenges for evaluators of inter- and transdisciplinarity include making decisions about which dimensions to select and possibly rank when judging the quality of a project – is it the ‘original’ contribution to a particular field, the potential for innovation, the quality of integration, the broader impact or rather the learning process itself, for example. In general, what is missing are established frames of reference and bench marks against which performance and outcomes are measured.

To map and examine these issues is a necessary first step for a systematic overview and critical review of the current methodologies as well as for setting an agenda for the evaluation of inter- and transdisciplinarity (ID and TD) as a tool for learning, improvement, innovation and excellence of this type of research.

Programme

Abstracts can be found further down after the information about the speakers

Location: room 002 in Hallerstrasse 11 (behind University Main Building, at the end of Gesellschaftsstrasse)

IASH Workshop:
Competencies in/for Inter- and Transdisciplinary Research and Learning

A workshop with presentations by Veronica-Boix-Mansilla, Julie Thompson Klein and Willi Haas/Barbara Smetschka (powerpoint pdf)
Chair: Manuela Rossini

Partners

We thank the following partners for their collaboration and support:

SNSF, institutional partner (and financial support)

Stiftung Mercator Schweiz, financial support

CSID, travel grants for speakers from the USA

artnersartnerartner

Constructing 'Success' in Interdisciplinary Collaborations: When Cognition, Emotion, and Interaction Matter at Once

With few exceptions, available theories concerning interdisciplinary collaborations tend to focus heavily on either cognitive, social, or institutional dimension of such interchange. Drawing on an empirical study of nine established interdisciplinary research networks, this presentation examines how collaborators construe notions of success (or lack thereof) and the conditions that enable or impede their joint accomplishments.

A theoretical construct “shared socio-emotional-cognitive (SSEC) platforms” is advanced to capture markers of successful interdisciplinary collaboration and the conditions that enable them. The talk elaborates on this theoretical concept illustrating it with examples from our data. Three key dimensions of SSEC platforms will be described addressing the theoretical assumptions and empirical grounding on which they stand. Dimensions include: the cognitive-intellectual (most exclusively concerned with the representations of objects and epistemologies); the emotional (concerned with affective connection to individuals, ideas and academic self); and the social-interactional (concerned primarily with interaction, meaning-making, and group styles).

Beat Butz & Pasqualina Perrig-Chiello, NRP/DORE, SNSF
National Research Programmes (NRPs) and Practice-Oriented Research (DORE)

Science and society are in a continuous interdependence. Changes in society have generated new research topics, which require innovative scientific approaches. In turn, science and science policy have reacted to this, and are in a constant search for better methods and infrastructures. This is also the case for the Swiss National Science Foundation (SNSF). This presentation wants to highlight two SNSF instruments and their evaluation practice by discussing their strong points but also their limitations.

1) National Research Programmes: NRPs were established in 1975 as an answer to a strong political intent to urge the scientific community to contribute to the solution of problems of national interest, such as climate change or demographic ageing. No institution was considered more competent than the SNSF to keep up the high scientific criteria for the NRPs as well. Since 1975 nearly 70 programmes and thousands of projects have been carried out, undergoing a complex selection procedure where bottom-up and top-down elements alternate and where stakeholders join in and have a say in the choice of topics. The NRPs have thus become a sort of experimental model for inter-and transdisciplinary research in Switzerland.

2) DO-REsearch (DORE): The creation of Universities of Applied Sciences in Switzerland in 2000 with their mandate to conduct teaching and research was associated with new challenges for research funding: Like in most applied disciplines, research competence was lacking and adequate funding instruments were missing. For these reasons the SNSF created DORE, with the aim of promoting practice-oriented research in social work, education, applied psychology and linguistics, health, and the arts at Universities of Applied Sciences. DORE can be seen as a prototype of transdisciplinary research funding scheme, since one precondition for project promotion is the participation of a “practice-partner”. What are the procedures and criteria to evaluate this research?

A specter is haunting Europe — the specter of sustainability. All the powers of old Europe (America, China, etc.) have entered into a holy alliance to exorcise this specter: politicians and university presidents, the IMF and the Worldbank, academic radicals and neoconservative critics.

For the last 125 years science and disciplinization have been cut from the same cloth. Both presumed that the connections between things can be marginalized or ignored – in the one case via the controlled experiment, in the other via an internal economy of knowledge production. In this way we could end up with reliable (replicable, quantifiable) results. Moreover, scientific/disciplinary knowledge was thought to be automatically relevant — no extra step was needed for knowledge to be useful.

Today, however, accountability is the watchword. Disciplinary production was deemed insufficient. Terms like ‘interdisciplinarity’ and ‘transdisciplinarity’ were first offered as shibboleths marking the speaker’s research as relevant. But today further proof is demanded. Within the Assessment Regime academic knowledge must justify itself.

In principle this is a good thing. But it raises a dilemma: assessing matters within poorly bounded systems is inherently philosophic and political in nature. If it is countable, it may not count; if it counts, it may not be countable. And in any case, we are left in the space of interpretation and argument rather than proof – a problem within pluralistic cultures that doubt the ability of philosophy to resolve differences.

This calls for research into the epistemology of relevance. Signposts for assessing the relevance of ID and TD research include distinguishing between:

None of this, however, gets to the heart of the matter, when we frame accountability in terms of sustainability. Sustainability is the ne plus ultra of accountability; but it is a form of accountability that overturns all of our commitments to the infinity of production and consumption – of both consumer durables and knowledge itself. Sustainability thus remains a mere ghostly presence haunting all our conversation and research.

Dieter Imboden, President of Research Council, Swiss National Science Foundation

The general wisdom seems to be: inter- and transdisciplinary (inter/trans) research projects are more difficult to evaluate than ‘normal’ (disciplinary) projects and thus are handicapped in competitive research funding. Other contributors to this conference will deal with the empirical evidence, such as success rates of different kinds of research funding, as well as with the various methods which were developed by different evaluation bodies to take care of the special circumstances of inter/trans research. In contrast, my starting point will be the question whether our implicit point of reference, i.e. the evaluation of disciplinary research, is adequate. In other words, I will first address the question whether the common method to evaluate disciplinary research is meaningful to identify the quality of research. – It is true that for some kind of (disciplinary) projects the results of different reviewers are more in line with each other than for others. But homogeneity of opinion does not necessarily mean high quality. My personal experience with evaluation ‘cultures’ of the humanities, the social and the natural science makes me believe that the difficulty of the evaluation of inter/trans research has its origin in lopsided disciplinary cultures. Inter/trans research is like a magnifying glass; it shows more clearly the difficulties of normal project evaluation. Dealing with the challenge of inter/trans research evaluation will thus also improve our awareness of the problems and pitfalls of research evaluation in general.

Evaluating Interdisciplinary Research Proposals – The Experience of the Swiss National Science Foundation

In 2006, the Swiss National Science Foundation (SNSF) created a specialised body for the evaluation of interdisciplinary research proposals and the promotion of interdisciplinary research, the Specialised Committee Interdisciplinary Research (FA-ID). Statistical analysis had revealed considerably lower success rates of interdisciplinary proposals compared to disciplinary ones. The SNSF had attributed this situation to inadequate evaluation of interdisciplinary proposals by bodies which are organised along disciplinary lines.

Since the creation of the FA-ID, the success rates of interdisciplinary proposals have risen to an average of 45%; a level almost equivalent to that of disciplinary proposals. However, this quantitative improvement does not necessarily indicate that the quality of the evaluation process has increased and that the best proposals are selected. Having evaluated more than 250 interdisciplinary proposals in the course of the last five years, the FA-ID is now looking to assess the adequateness of its evaluation activity and to explore possibilities for further improvement.

In our presentation, we attempt to characterize the interdisciplinary proposals submitted to the SNSF as well as to identify key factors determining the success or failure of a proposal during evaluation and decision taking. In view of the relatively low numbers of proposals submitted at a given point, the wide variety of disciplines, the heterogeneity of academic careers as well as individual and disciplinary differences in evaluation cultures are major challenges with regard to interdisciplinary proposal evaluation.

Laura Raaska, Director of the Biosciences and Environment Research, The Academy of Finland

Experiences of Reviewing Inter- and Multidisciplinary Project Proposals in the Fields of Biosciences and the Environment

Inter- and multidisciplinary research is regularly associated with cutting-edge research with high innovation potential and renewal of research. It is also argued that the most favourable environment for innovations is at the interface of established research disciplines. The Academy of Finland is the main governmental funding body for basic research in Finland, covering all disciplines and having several different funding opportunities available for researchers. In its strategy, the Academy of Finland emphasises the clear goal to enhance and promote inter- and multidisciplinary research and to systematically develop and follow-up the evaluation process of research proposals. In 2010, the Academy’s Research Council for Biosciences and Environment granted a total of 65.5 million euros for research in biosciences and environmental research. This presentation highlights the evaluation process with a focus on inter- and multidisciplinary project proposals in the fields of biosciences and the environment.

The scientific evaluation and the decision-making are performed separately. The evaluation process starts by registration of applications to appropriate Research Councils. In this phase, special attention is paid to proposals with inter- and multidisciplinary nature. In the fields of biosciences and the environment, all proposals are reviewed by panels. Before the panel review, the proposals are rated (1-6) independently by at least two international experts who will attend the panel meeting. These ratings are used as a basis for panel discussions, and the final rating is made by consensus by the whole panel. The consensus ratings are used as a basis for the funding decisions by the Research Council. Besides the panel ratings, the Council also considers other important research policy issues when making funding decisions.

According to our experience, important aspects to enhance a fair evaluation of inter- and multidisciplinary research proposals are: (1) clear identification of inter- and multidisciplinary project proposals and awareness of their special characteristics; (2) clear indication of the added value of inter- and multidisciplinary research; (3) evaluation in international expert panels rather than by two individual reviewers; (4) having joint evaluation panels between Research Councils when appropriate; and (5) selection of experts who have broad experience and preferably a background in research with multi- and interdisciplinary nature. Furthermore, it is important to understand that the deepness and mode of inter- and multidisciplinarity in research varies significantly, and that for a fair and transparent evaluation the ability to identify the nature and added value of this kind of horizontal cooperation is crucial.

I will focus on the assessment of what arguably is the outcome of transdisciplinary research: socially robust knowledge. The term was coined in 2001 by Nowotny a.o. as the product of strongly contextualized research. Robustness, in their view, is produced when research has been infiltrated and improved by social knowledge.

The implications for assessment of this contextual approach of research are manifold; one of the main being that assessment no longer ought to focus on research alone but on the knowledge production process as a whole. This basically means a focus on the variegated interactions between all stakeholders in that process. It also means avoiding concepts that originate from more traditional unidirectional views on the relationship between research and its output in terms of social and intellectual impacts. Impact presupposes a sender and a receiver; however, in transdisciplinary research, the knowledge produced is the result of a team effort, and the team consists of members with backgrounds in different social spheres, be it academia, private labs, governmental departments, consumer organisations, the public at large. The ‘products’ of such collaboration are of a mixed nature, intermediary artefacts often, being part of longer term interaction in which research meets other interests. For all stakeholders to work together in a productive way, evaluation should serve as a learning process in stead of an accounting instrument or a ranking tool. This approach has wide implications for the review process.

In the FP7 project SIAMPI, we analysed three types of ‘productive interactions’ between stakeholders in transdisciplinary research production: direct or personal interactions; indirect interactions through texts or artefacts; and financial interactions through grants or ‘in kind’. By making the research process visible that way, we were able to enlighten the process for participants, and to suggest some robust measuring instruments.

Dan Stokols, University of California, Irvine (Department of Planning, Policy, and Design, Department of Psychology and Social Behavior, Department of Epidemiology)

This presentation will begin by reviewing earlier conceptualizations of uni-, multi-, inter-, and transdisciplinary forms of research, along with certain caveats about the complexities that arise when efforts are made to clearly distinguish among them. Following discussion of these issues, the talk will focus primarily on team-based transdisciplinarity (TD), as an approach for engaging in cross-disciplinary research that draws on the perspectives of multiple fields and research partners, and trans-epistemic research (TE), defined as a sub-type of transdisciplinarity in which at least one academic discipline and one non-academic epistemology or knowledge culture are integrated for purposes of creating novel approaches to analyzing and resolving complex community and societal problems.
The next portion of the talk builds on the above discussion to consider the question of what makes participation in TD or TE either strategic or non-strategic? As a starting point for this discussion, the concept of strategic TD collaboration is proposed, defined in terms of the ratio of intellectual and/or community improvement innovations achieved relative to the opportunity costs associated with participating in that collaboration. Particular emphasis is given to place-based TD research centers as a context for engaging in collaborative team science and practice, and the circumstances that either enhance or hinder their effectiveness. An important evaluation challenge raised by the notion of strategic TD collaboration is to establish measurable criteria for gauging the magnitude of TD research and policy innovations. Efforts to identify and assess the magnitude of TD innovations must be framed in relation to particular research and societal contexts. It is suggested that TD innovations are fueled by the tension between an existing research or policy landscape, and an imagined futurescape of research and practice possibilities that has not yet been fully charted and cultivated. As innovative research futurescapes linking multiple disciplines and epistemologies are gradually mapped and elaborated, they establish a new reference point or backdrop for identifying future TD departures toward newly emerging futurescapes and innovations.

The concluding portion of the presentation focuses on the concept of opportunity costs in team-based TD research. Key factors that influence the congruence or level of fit between team members’ attributes, the kinds of scientific and societal problems they are addressing, and alternative organizational infrastructures for supporting research partnerships are discussed as a basis for reducing opportunity costs and enhancing the strategic value of TD and TE research collaborations.

Christian Bertsch is a senior researcher at the University of Education in Vienna. He holds a PhD in science education focussing on inquiry based learning and scientific literacy. He worked as advisor for the Austrian Ministry for Transport, Innovation and Technology developing new ways of linking science and society. He took part in different FP6 and FP7 projects on the dissemination of inquiry based learning in Europe. In his current reserach he is focussing on Science-School-Cooperations and factors that facilitate or hinder the joint authentic research of scientist, teachers and students.

Veronica Boix Mansilla is the Principal Investigator for the Interdisciplinary Studies Project at Project Zero, Harvard Graduate School of Education where she has also taught and chairs the Future of Learning Institute. Her research examines the conditions that enable experts and young learners to produce quality interdisciplinary work addressing problems of contemporary significance. She brings together theories and methods in cognitive psychology, epistemology, pedagogy and sociology of knowledge to explore how experts, teachers and K-16 students advance interdisciplinary understanding of topics of global significance from globalization to climate change and migration. In addition, she studies the development of global consciousness among youth in America, Kenya and India and, most recently, has worked with the Council of Chief State School Officers and the Asia Society to advance a definition of “global competence” as an important aim of contemporary education. Furthermore, she is a Bernard Schwartz Fellow at the Asia Society. She is also an advisor for the Asia Society as well as for the Association of American Colleges and Universities, Council of Chief State School Officers, and International Baccalaureate. She has also taught at the University of Buenos Aires, and is the founder of L@titud, the Latin American Initiative for Understanding and Development. She is the author of Teaching for Interdisciplinary Understanding in the International Baccalaureate Middle Years Program (2010); the International Baccalaureate World Studies Extended Essay (2011); Educating for Global Competence: Preparing our youth to engage the world (2011) with Tony Jackson; and The Point of Integration: Pivotal reflections on quality contemporary interdisciplinarity (forthcoming).

Beat Butz graduated in Modern Languages and History at the University of Basel where he was an assistant (lecturer) at the Department of Romance Studies from 1974-1981. He holds a PhD in French Linguistics (Dialectology). For almost twenty years, from 1982-2011, he was Head of the Division IV: Research Programmes (National Research Programmes and National Centres of Competence in Research) of the Swiss National Science Foundation. He served as a member of various working groups and committees, including the td-net of the Swiss Academies of Arts and Sciences, and the Swiss Science and Technology Council.

Robert Frodeman is Professor of Philosophy and former Chair of the Department of Philosophy and Religious Studies at the University of North Texas (UNT), where he specializes in environmental philosophy, the philosophy of science policy, and the philosophy of interdisciplinarity. He served as a consultant for the US Geological Survey for eight years, was the 2001-2002 Hennebach Professor of the Humanities at the Colorado School of Mines, and was an ESRC Fellow at Lancaster University in England in the spring of 2005. In addition to more than 60 published articles and $1.7mil in grants, Frodeman is the author of Geo-Logic: Breaking Ground between Philosophy and the Earth Sciences (2003), co-editor of the Encyclopedia of Environmental Ethics and Philosophy (2008), and editor of the Oxford University Press Handbook of Interdisciplinarity (2010). Frodeman is the founding director of the Center for the Study of Interdisciplinarity at UNT (www.csid.unt.edu).

Sebastian Gölz studied psychology at the University of Heidelberg. Since 2002 he has been working at Fraunhofer ISE as an expert for non-technical issues in the application of photovoltaic and renewable energies in decentralised generation and smart grids. His focus is on social-scientific research and tests of innovative supply concepts specifically with regard to user behaviour and integration in smart grids. He is currently coordinating four research projects on Smart Metering and innovative tariffs and analyses the effects of energy feedback and tariffs on the consumption behavior in households. He is also coordinating the transdisciplinary project cluster „Intelliekon“ funded by the German Ministery of Education and Research (BMBF). Since October 2010 he is heading the research group „User Behavior and FieldTtrials“ in the Department for Intelligent Energy Systems at Fraunhofer ISE.

Gertrude Hirsch Hadorn has been Professor Titular at ETH Zurich since 2006. She received a PhD in educational sciences at the University of Zurich in 1989. In 1990 she joined the Department of Environmental Sciences (D-UWIS) and completed her habilitation on nature and ethics in debates on the environment at the Philosophy Department of the University of Konstanz in 1998 (published: Umwelt, Natur und Moral, 2000). Her research addresses the philosophy of environmental sciences and sustainability research (with a focus on land use and climate change) as well as environmental ethics and ethics of science. Since 2000 she is member of the Scientific Board of GAIA, she served as Vice President of the Swiss Academy of Sciences scnat (2001-2006) and President of the td-net (2003-2008). Among her major co-authered and co-edited books are: Handbook of Transdisciplinary Research (2008), Principles for Designing Transdisciplinary Research (2007), Unity of Knowledge in Transdisciplinary Research for Sustainability (2002, http://www.eolss.net).

Dieter Imboden has been full Professor of Environmental Physics in the Department of Environmental Sciences at the ETH Zurich since 1988. Since 2005 he is President of the Research Council of the Swiss National Science Foundation (SNSF), which is Switzerland’s main public funding agency for basic research. Since 2009 he is President of EUROHORCs, the association of the heads of European research organizations. He studied theoretical physics in Berlin and Basel and in 1971 received his doctorate at the ETH Zurich for his studies on theoretical solid-state physics. His interest for the environment, particularly water led him to the Swiss Federal Institute of Aquatic Science and Technology (EAWAG) and to the Scripps Institution of Oceanography, California. In 1987 he was one of the co-founders of the new curriculum in Environmental Sciences at the ETH Zurich. From 1992 to 1996 he served as head of the Department of Environmental Sciences.

Angelika Kalt was appointed Deputy Director of the Swiss National Science Foundation (SNSF) in 2008. She is responsible for quality assurance and development of the organisation’s evaluation principles and procedures. After several brief stays in Holland and France, Dr. Kalt became a full professor of petrology at the University of Neuchâtel, Switzerland, in 2000. She studied geology in Freiburg, Germany, and received her doctorate at the Universities of Freiburg and Münster. Besides her broad expertise in petrology, geochemistry and mineralogy, she can draw upon many years of experience in designing and carrying out research projects, fund-raising, journal and proposal reviewing and teaching on all academic levels.

Philippe Moreillon is a full professor and Head of the Department of Fundamental Microbiology as well as Vice-Rector of the University of Lausanne in Switzerland. With PhD degrees in philosophy as well as medicine, he previously held various positions at medical centers and university hospitals in Switzerland as well as in the USA where he is, since 2003, a member of the Adjunct Faculty of the Rockefeller University in New York. His research interests encompass the pathogenesis and ecology of Gram positive bacteria, microbial-induced inflammation, and antibiotic resistance. He also sits on the International Editorial Board of the journal Antibiotics for Clinicians.

Pasqualina Perrig-Chiello, professor at the University of Bern, Institute of Psychology, Department of Developmental Psychology. Studies at the University of Fribourg, post-doc researcher at the University of Colorado, Boulder, CO, USA, and at the University of the Saarland, Saarbrücken. Teaching appointments at the J. Wolfgang-Goethe University Frankfurt a.M. and University of Basel. Her research and teaching focus on topics of life-span developmental psychology, especially individual differences in well-being and health and family relations over the life span. She has been involved in various psychological and interdisciplinary research projects on these topics and chaired a NationalRresearch Programme on intergenerational relations; currently she is involved in a National Center of Competence in Research on vulnerability and in a multi-site interdisciplinary project on Ageing in Switzerland. She is President of the td-net as well as a member of the National Research Council (Swiss National Science Foundation); member of the Standing Committee for the Social Sciences of the European Science Foundation, Strasbourg; Consulting Editor of the European Journal of Ageing; Member of the Scientific Board of various scientific journals such as Swiss Journal of Psychology, Journal of Gerontopsychology & Psychiatry; Zeitschrift für Gerontologie und Geriatrie. Author of over 200 scientific publications.

Laura Raaska is Director of the Biosciences and Environment Research Unit at the Academy of Finland. She holds an Adjunct Professorship in the field of food microbiology at the Faculty of Veterinary Medicine, University of Helsinki. She has a strong background in research, especially in applied fields of microbiological safety and hygiene, food microbiology and microbial ecology in industrial process environments. She has more than 20 years’ experience as an expert and a manager in both national and international research projects that focus on interdisciplinary research questions. Most recently, she held the post of Technology Manager of Bioprocessing at VTT Technical Research Centre of Finland, where research is performed in close contact with industry and other stakeholders. At VTT, she gained significant experience of multidisciplinary and interdisciplinary activities. At present, she heads the Biosciences and Environment Research Unit of the Academy of Finland. She is also leading a development project studying the characteristics as well as evaluation processes and success rates of multidisciplinary and interdisciplinary research proposals. Furthermore, she is Chair of the Steering Committee for the Baltic Sea Research and Development Programme BONUS, which focuses on funding interdisciplinary projects in Baltic Sea research.

Jack Spaapen is working as a senior policy advisor for the Royal Netherlands Academy of Arts and Sciences. His specialty fields are research impact evaluation, higher education policy and humanities research policy. He co-developed the current national evaluation protocol SEP, a national program that regards the evaluation of socio-economic value of research (see for the latter www.eric-project.nl ), and he coordinates the FP7 research project SIAMPI, focusing on the development of instruments for the assessment of social impact of research (www.siampi.eu). He specializes also in broader questions about the relationship between academic researchers and industry or society (Public-private partnerships). He has a wide experience in evaluation, both as an evaluator, and as an organizer of evaluation procedures. He was trained as a sociologist and a cultural anthropologist at the University of Amsterdam with a Ph.D. in science and technology studies. His thesis focused on developing a method for the evaluation of research in the context of policy and societal demands. Previously he worked in two departments at the University of Amsterdam (Science and Technology Dynamics and the Institute for Development Research), and at the University of California San Diego in the Science Studies Program.

Rudolf Stichweh is Professor of Sociological Theory at the University of Lucerne where was rector from 2006–10. He was a researcher at Max Planck Institutes in Cologne and Frankfurt (1985-1994) and at the MSH, Paris (1987); Professor of Sociological Theory at the University of Bielefeld (1994–2003); visiting professor at the EHESS, Paris (2000), the University of Vienna (2001-2), the University of Klagenfurt (2007, 2009), Radboud University Nijmegen (2011), Princeton University (2011) and Fellow of the ‘Wissenschaftskolleg zu Berlin’ (2005-6). He is the author of: Zur Entstehung des modernen Systems wissenschaftlicher Disziplinen: Physik in Deutschland 1740-1890 (1984); Der frühmoderne Staat und die europäische Universität (1991); Wissenschaft, Universität, Professionen (1994); Die Weltgesellschaft (2000); Inklusion und Exklusion (2005); Der Fremde: Studien zu Soziologie und Sozialgeschichte (2011).

Daniel Stokols is Chancellor’s Professor of Planning, Policy, and Design, and Psychology and Social Behavior in the School of Social Ecology (where he served as Director and founding Dean during 1988-1998) at the University of California, Irvine. He is also Professor of Public Health and Epidemiology in the College of Health Sciences at UCI. He is past President of the Division of Environmental, Population, and Conservation Psychology of the American Psychological Association (APA) and is currently a Section Editor of the American Journal of Health Promotion and member of the Editorial Boards of the Journal of Environmental Psychology and the Journal of Architectural and Planning Research. Stokols received the Annual Career Award of the Environmental Design Research Association in 1991, the UC Irvine Lauds and Laurels Faculty Achievement Award and the Chancellor's Award for Excellence in Fostering Undergraduate Research in 2003, and UCI’s Outstanding Professor Award for the School of Social Ecology during 2009. His research examines contextual factors that influence the success of transdisciplinary research and training programs, as well as the health and behavioral impacts of environmental stressors such as traffic congestion, overcrowding, and information overload. Stokols is co-author of Behavior, Health, and Environmental Stress (1986) and co-editor of the Handbook of Environmental Psychology (1987) and Promoting Human Wellness (2002). Stokols presently serves as Scientific Consultant to the National Cancer Institute, Division of Cancer Control and Population Sciences where he is a member of NCI’s Science of Team Science (SciTS) team; and as a Team Science Evaluation Consultant for the National Academies of Sciences-Keck Futures Initiative (NAKFI).

Julie Thompson Klein is Professor of Humanities in the English Department at Wayne State University, USA. Klein is an internationally known scholar of interdisciplinarity, and a recipient of the Kenneth Boulding Award for outstanding scholarship and the Ramamoorthy & Yeh Transdiscipilnary Distinguished Achievement Award. She was past president of the Association for Integrative Studies, former editor of Issues in Integrative Studies, former Senior Fellow at the Association of American Colleges and Universities, and holder of invited positions in Canada, Japan, Nepal, and New Zealand. Klein has advised numerous task forces and agencies on interdisciplinarity, and has lectured throughout Europe, North America, Latin America, Australia, and Russia. Her authored and co/edited books include Interdisciplinarity (1990), Interdisciplinary Studies Today (1994), Crossing Boundaries (1996), Transdisciplinarity (2001), Interdisciplinary Education in K-12 and College (2002), Humanities, Culture, and Interdisciplinarity (2005), and Creating Interdisciplinary Campus Cultures (2010). She is also Associate Editor of the Oxford Handbook of Interdisciplinarity (2010) and co-editor of the University of Michigan Press series Digital Humanities@digitalculturebooks.

Sustainability Sciences: Evaluating Strategies of Setting and Shifting Boundaries

Organizers:
Willi Haas and Barbara Smetschka (Institute of Social Ecology Vienna, SEC, Alpen-Adria University Vienna), Alexander Bogner (Austrian Academy of Sciences), Austria

Part I: Shifting Paradigms: The impact of Theory of Differentiation and Mode2 Research in Transdisciplinarity Debates

In sustainability science transdisciplinarity research is often deemed tantamount to participation of non-scientific actors according to the theory of Mode 2. Competing approaches like Luhmann’s theory of differentiation, in contrast, are less popular for the reflection of transdisciplinarity. However, the theory of differentiation reminds us that increasing specialisation strongly influences research practices. Research still follows the requirements of scientific rationality set by disciplinary science. From the point of view of science studies, the first session deals with the question how the demands of ongoing internal differentiation of science (boundary-drawing) and calls for an “opening up” of science for its environment (transgressing boundaries) can be reconciled. We are interested in questions such as:

Markus Arnold (Institute for Science Communication & Higher Education Research, Faculty for Interdisciplinary Studies (IFF), Alpen-Adria University, Vienna

P1

1.1 Assessing the Societal Benefit of Transdisciplinary Research in Aquatic Science

Janet Hering and Sabine Hoffmann, Swiss Federal Institute of Aquatic Science and Technology (Eawag), Switzerland

Most research is supported by public funding. In the case of basic research, it is widely accepted that the benefits to society are often unanticipated and may develop long after the initial research was performed. In the case of transdisciplinary research, however, a more direct link between the research and its benefits is often expected. The justification for research funding commonly invokes the contribution of transdisciplinary research towards the solution (and prevention) of socially relevant problems. In either case, society requires some assurances that its investment (i.e., public funding) is used responsibly and that benefits to society are generated. Research institutions thus come under pressure to provide such assurances. For basic research, where short-term, direct benefits to society are not expected, the focus is on demonstrating the quality of the research, including its originality and novelty and the extent to which it stimulates scientific progress. Hence, a variety of indicators (e.g., impact factors, h-index, etc.) have been developed to assess basic research. These indicators are less useful for transdisciplinary research since they do not address the impact of research beyond academia. This broader impact may be evident only many years after the initial research has been conducted. This makes it increasingly difficult to determine its impact on society, i.e. to attribute the solution (or prevention) of a socially relevant problem to a single research project or program. Nevertheless, it is necessary to develop both quantitative and qualitative indicators that can be used by research institutions to determine the societal benefit of public expenditure on transdisciplinary research. The present paper does not provide a recipe for assessing the societal benefit of transdisciplinary research in aquatic science. Instead, it discusses the potentials and limitations of a set of indicators for assessing transdisciplinary research, using examples from a specific research institution, the Swiss Federal Institute for Aquatic Science and Technology (Eawag). Although it focuses on indicators that can be clearly defined (if not always quantified), the paper emphasizes that such indicators must always be complemented by other tools for assessing transdisciplinary research such as peer reviews and stakeholder interviews.

1.2 The Societal-Impact Perspective of Transdisciplinary Research: Conceptual Groundwork and Fields of Application

Sebastian Helgenberger, BOKU Center for Global Change and Sustainability, Vienna, Austria. Co-author: Roland W. Scholz, ETH Zurich, Switzerland

Keywords: societal impacts, impact model, transdisciplinarity index, ETH-UNS case study

The transdisciplinary research (TdR) approach – mutual learning between science and society – suggests that this research type holds positive effects on both scientific excellence and on involved societal stakeholders. We argue that successful evaluations of TdR need to be able to register the effects of mutual learning in both directions and need to go beyond scientific impact. However, the societal impacts of TdR have not yet found their way into evaluation frameworks in TdR. Based on our findings and experiences in studying the direct societal impacts of a major TdR project, the ETH-UNS Case Studies in the Swiss Canton Appenzell-Ausserrhoden (2000-2004), we elaborate the societal impact perspective of TdR evaluations. Two questions guided our work: „Which are these impacts?“ and „How to measure them?“.

The TdR under investigation was carried out in the field of sustainable development in land use and regional economy in the Swiss canton Appenzell-Ausserrhoden. In the course of the programme a total of about 200 local stakeholders participated in the TdR in terms of different intensities of involvement and modes of interactions. Based on in-depth interviews with key stakeholders we developed an initial model of societal impacts in TdR. The model was tested and further elaborated, using a quantitative approach, based on an extensive survey, sent out to all involved stakeholders (52% return rate), and a subsequent statistical factor analysis of the inquired data. We found that societal impacts appeared on a problem-related and an inter-personal level. Problem-related societal impacts refer to a changed approach to the problem under study – both in terms of problem perception (problem awareness, problem-solving competencies) and stakeholder-interactions (personal commitment for problem-solving, dissemination of learnings during the research collaboration). The collaborative efforts in the course of the transdisciplinary project had consequences for the inter-personal relations (personal identification and attachment with the stakeholder's local community, level of trust among the stakeholders) and networking effects (establishing new contacts, intensifying existing contacts). The findings underpin the significance of the societal impact perspective in evaluations of TdR and equally indicate the distinctness and added value of this research mode. We consider our societal impacts model as basis to extend the SCI as predominant evaluation framework of scientific effectiveness by the societal impact dimension in terms of a Transdisciplinarity Index.

In relation to our empirical and conceptual work on the societal impact perspective in TdR we discuss application examples where this perspective has influenced university research and international research programming.

Birge Wolf, Manfred Szerencsits, Jürgen Heß, University of Kassel, Germany, and Thomas Lindenthal, University of Natural Resources and Life Sciences, Vienna, Austria

In the established system of research evaluation (e.g. SCI), the effects of science on practice and society have not been sufficiently taken into consideration. Manifold concepts, however, are being developed to evaluate applied, interdisciplinary and transdisciplinary research adequately. These concepts differ considerably in purpose, methods and extent of criteria set. However, they show a large intersection regarding relevant achievements for such an evaluation: Cooperation with other scientists and actors from practice and society, as well as publications and products, relevance and real impact for practice and society. We have also identified the same findings via interviews with agricultural scientists for an adequate evaluation of practice-oriented organic agriculture research.

Fundamentally, the question arises how a supplementation of the scientific impact by an impact on practice and society can be established in research evaluation. Besides a wide discussion within the Scientific Community, data especially should be collected with less effort, and permanent data availability for various evaluation objects (scientists, projects, institutions, programmes) and context should be guaranteed.

Evaluations of inter- and transdisciplinary research have mainly been carried out as “stand-alone-procedures” with much effort for data collection via interviews, evaluation of project reports or facilitation. As connecting factor for improved data collection, the public research funding was identified, both in guidelines which focus on scientific quality and the societal value of the research, and data requirements. Considerable overlapping was found between the requirements for proposals and reports of many public research sponsors (e.g. BMBF, BMWi, BMELV, BMU) and the data requirement for an evaluation of impact on practice and society. To be able to utilise the information in proposals and reports for an evaluation, it is necessary to a) structure the data in such a way which makes it possible to use both for research funding and for various evaluations and b) to explicitly integrate necessary information for the evaluation which as yet has not been recorded, or only implicitly claimed. A database system seems here to be target-aimed for the technical implementation as data entered onetime can be filtered contextually, and analyses variably in detail or more aggregated. Structured entries by scientists into a database system would therefore substitute considerable parts of proposals and reports. How this can be carried out practically and how far the intended benefit for scientists, research promotion and evaluation is to be gained, must be shown by testing of case studies.

P2

2.1 European Reearch Area and the Evaluation of Inter- and Transdisciplinary Research Projects

Signe Martišūne-Schwagrowski-Buyse, former Science Attaché to the EU (Latvia, now Tunesia)

Keywords: European Research Area (ERA); European Cooperation in Science and Technology (COST); innovation; policies

Over the last five years the European Research Area has had an increasing part in European Union policies, covering a very broad if not the broadest number of fields ranging from medicine, health, environment, agriculture and to energy research, including atomic questions and also space research. New developments of the EU’s Research, Technology & Development (RTD) policies such as the Innovation Union Communication published in 2010 have given a new strength to the ERA 2020 Vision. The European Commission has stated that one of the goals of this Communication is to create a common framework of principles and objectives and that by 2014 the Union seeks to avoid fragmentation in its research systems. The evaluation of interdisciplinary and transdisciplinary research projects is one of the fields were fragmentation needs to be addressed and solutions to improve general evaluation methods have to be analyzed, discussed and systematically reviewed in order to establish new guidelines for the evaluation of such projects.

Most of the research projects funded by the EU are of an interdisciplinary (ID) or transdisciplinary (TD) nature, which is closely related to the overall developments in the fields of RTD and Innovation where interdisciplinarity and transdisciplinarity are becoming the core elements for success and excellence. Obviously these developments have had their impact on the evaluation process of research projects not only at the EU level, but also at the level of various national funding agencies and institutions.
This paper will focus on the analysis of EU policies and methodologies of TD and ID research evaluation. The following questions will be addressed: how is the European Commission (EC) implementing the evaluation of transdisciplinary project proposals (example of 6th and 7th Framework Programs)? Is there any impact of the choice of expert panels on ID/TD project evaluation results? Is there any impact of the EC organizational and administrative structures on the quality of evaluation. In addition, the case of COST programs will be examined as a good-practice example for the successful management of TD and ID research projects and networking support system not only at the European level but also worldwide. One of the main issues that will be analyzed is: what are the key factors that determine the success of COST program regarding ID and TD projects?
In order for ID and TD project evaluation to improve there is a need to have a solid overall EU level of experts and guidelines taking into account the strong connections that exist between national research projects and projects that are funded by the EU. The main objective of this analysis will therefore be to establish the link between EU positions as described in the policy documents on ID and TD research and the approach to the project evaluation with a view on how to improve the present situation and adopt common guidelines for ID and TD research evaluation.

2.2 Doing Transdisciplinarity in Research Cooperation for Sustainable Science: Joint Learning and Experiences from the German‐Moroccan Research Cooperation on Urban Agriculture in Casablanca

Keywords: urban agriculture, sustainable urban development, action research, research and development cooperation, Morocco

Global Change is a collective term that refers to all changes in the global environment that may alter the capacity of the earth to sustain life. One of the most powerful drivers of global change is urbanisation, including socio-economic transformations and the complex interactions of urban areas with their physical environment. With this in mind, the programme “Sustainable Development of Future Megacities” of the German Federal Ministry of Education and Research (BMBF), focuses on energy‐ and climate‐efficient structures in urban growth centres. Based on the results of the preliminary phase (7/2005 – 3/2008) and under consideration of the relevant funding frame, ten bilateral projects have been working on their specific research topics since April 2008 (until March 2013). One of the research projects investigates to what extent urban agriculture can make a relevant contribution to building a resilient city, and does this in Casablanca, Morocco. This inter‐ and transdisciplinary project titled “UAC – Urban Agriculture as an integrative factor of climate‐optimised urban development, Casablanca, Morocco” is executed jointly by researchers and practitioners from Morocco and Germany and is managed by TU Berlin.

The paper is dealing with transdisciplinarity and new alliances in research cooperation for sustainable science based on the working experiences over five years. The author wants to contribute to the current discussion on how to generate and implement new knowledge for adapted solutions in problem‐oriented research of global change issues, respectively in the context of future megacities development. Typical problems of an inter‐ and transdisciplinary project setting occurred. In addition, (inter)cultural issues influence the concept and setup of the whole project, such as: understanding about the research topic, research method (open process, action research), taking over responsibilities, decision making process among the various local stakeholders and influence of hierarchy and power, German‐Moroccan cooperation and influencing factors. Research and development cooperation can become innovative and crucial in tackling complex global problems, like urbanisation and climate change. With its complex partnership structure, the project requires all participants to contribute to transferring interdisciplinary, transdisciplinary and intercultural knowledge and approaches. Therefore, the task for project management is to ensure optimal design of the interlinked procedures as well as processes of communication within the project. Key results from internal project evaluation workshop (spring 2009) and external interim evaluation (autumn 2010) will be presented and discussed and how experiences and reflections for good transdisciplinary research practices. The issues are briefly described and examples will be given as well as procedures / ways how the project members currently deal with it.

Catherine Lyall, ESRC Innogen Centre, University of Edinburgh. Co-authors: Ann Bruce, Wendy Marsden, ESRC Innogen Centre, University of Edinburgh, and Laura Meagher, Technology Development Group, Fife, Scotland
Keywords: evaluation criteria; evaluation process; leadership; organisational learning; capacity

Evaluation plays a critical role in blocking or facilitating interdisciplinary research. At the same time, interdisciplinary research often flies in the face of academic conventions, structures and norms. We offer some practical suggestions for judging interdisciplinary work fairly (particularly when it is in competition with single discipline research); improving interdisciplinary evaluation processes; and a new vision for interdisciplinarity. These practical suggestions are based on the findings of a research project funded by the UK’s Natural Environment Research Council (NERC). Key objectives were to: develop multiple case studies to capture learning around the management and development of large-scale interdisciplinary investments; and promote organizational learning by providing transferable lessons of relevance to future interdisciplinary programs along with practical guidance to funders and leaders of such initiatives.
This paper reports on empirical research structured around four case studies of interdisciplinary environmental initiatives each representing long term, multi-million pound, multi-discipline and multi-centre investments by the UK Research Councils, and complemented by a set of international examples. Arising from this study, we suggest that the evaluation of interdisciplinary research would benefit from:
1. Improving evaluation criteria and processes - these are key to achieving a more stable and consistent role for interdisciplinary research and for improving its intellectual status in academia.
2. Making the quality criteria by which work will be judged clear from the outset to those being evaluated - thus encouraging explanation of why proposed research needs to be interdisciplinary; what disciplines are involved and why; how they will be integrated, and how the quality of the interdisciplinary outcomes might be assessed.
3. Managing the process of evaluation when employing review panels - which needs an informed chair for the panel giving clear guidance on how to evaluate interdisciplinary initiatives, with a good understanding of what is required, and a strong enough control over the process to ensure that the guidance is followed.
4. Funders providing strong leadership - research funders play an important role in shaping investments and on their longer-term impacts.
The UK Research Councils have recognized the challenges as they exhort institutional structures to change so that interdisciplinary research capacity can be developed. Many other countries’ research funding organizations are beginning to pilot funding schemes that require interdisciplinary collaboration. The effective and appropriate evaluation of interdisciplinary research is essential if it is to build capacity and achieve its potential.

3.1 Framing Quality: Constructions of Medical Quality in Swiss Family Medicine

Keywords: Industrial quality rationality; travelling concept; quality dispositive; quality rhetoric; medical epistemologies

Starting in the 1940s, the modern quality rationality developed into a global mission and market. What happens when the quality rationality travels from industry to fields such as family medicine? How does it move about, and what does it transport and transform? How is it appropriated, negotiated or resisted? With the revision of the Swiss health insurance law in 1996 external quality claims became a public issue in Swiss family medicine. Health providers and health insurance companies were obliged to co-create and implement quality concepts until 2012. I consider this legal claim to be a discursive event that initiated the quality discourse I am analysing from the perspective of family medicine.

The legal intervention of 1996 transformed medical quality from an intra-professional domain to a field of multiple expertise and negotiation. It represents the culmination of the change in status of the medical profession because professional autonomy is replaced by the shared control and administration of medicine with other actors and functional systems. That is why I consider 1996 as a caesura in family medicine: It forced family medicine to institutionalize and externalize a certain kind of quality thinking and to participate in quality discussions with medical as well as non-medical actors. So, 1996 initiated an ongoing re-mapping of family medicine that blurred the boundaries between intra- and extra-professional, and between State regulation and professional autonomy. It inevitably led to the re-framing of professional space and to the re-modelling of the “good physician”. Quality related developments since 1996 created a network of institutions, discourses, publications, practices, rhetorics, regulations, evaluations, and certifications that can be conceptualised as a quality dispositive. Within this dispositive the different actors use the same quality terminology and therefore use quality as a collective symbol. Yet, they developed competing readings of it, due to the diverging perspectives and paradigms they relate to. Thus, quality does not exist per se or a priori but is constructed and negotiated about in social and political processes. But what is the use of the quality term if it is linked to diverging meanings and thus transforms quality talk into an empty talk? In this respect I will show how the term quality serves as a rhetorical device and argumentation strategy for the framing of negotiations, legitimisations and realisations of proper interests. Negotiations are about medical knowledge and its production, about expertise, truth, power, status, autonomy, tradition, identity, future scenarios, and capital.

The currently emerging culture of transdisciplinary research aims at overcoming established gaps between disciplines and societal domains when it comes to face pressing societal challenges. Given this ultimate objective, transdisciplinarity wants to transform relationships and adjust boundaries between historically grown entities, which have heterogeneous identities according to their societal role, rules and objectives. ‘Boundary-work’ is therefore a main challenge of transdisciplinary research. To improve this task, i.e. the development of appropriate research designs, integration methods and mutual understanding, it is crucial to be aware of one`s own understanding of apparent entities, in particular concepts of ‘self’ and of the perception by others. Boundaries only exist due to identity, which allows for distinguishing between different entities. At the same time, they are linking entities to each other.

When addressing boundary shifts or changes of the quality of boundaries, the exploration of concepts of ‘self’/’other’ is an important basis for solid boundary-work. It will allow for making key coordinates of the transdisciplinary research process visible. These range from very general ones, e.g. reflecting worldviews and ideas of the human, to concrete ones, like unfolding pillars of theories that underly specific research tasks, or presumptions of roles and responsibilities.

Exploring identities and presumptions is not data collection and difficult to realize in the context of research processes. It is, however, itself a learning process, which has to be practiced in education, in particular in academic training programs. In the context of evaluation of inter- and transdisciplinary resarch, my contribution aims at discussing whether the exploration of these basics for boundary-work in transdiciplinary research can be fostered through linking theses categories to formative evaluation procedures.

Learning Procedures: Best practices of Evaluating Inter- and Transdisciplinary Projects
Organizers:

Matthias Bergmann (Institute for Social-Ecological Research, ISOE)
Antonietta Di Giulio and Rico Defila (Interfakultäre Koordinationsstelle für Allgemeine Ökologie, IKAÖ, University of Bern)

Inputs:
Sebastian Gölz (Fraunhofer Institute for Solar Energy Systems, ISE)
Christian Bertsch (Institute for Research, Innovation and School Development Vienna, IFIS)

Description
Societal problems often demand solution-finding processes that transcend the order of scientific disciplines and go beyond purely academic approaches of research. The heart of transdisciplinary research lies in these boundary crossings between disciplines and between academia and society, in the integration of knowledge from different cores of knowledge and in the methods supporting these processes. At the same time, this raises the question about quality standards and the practice of assessing this kind of research. There is no general, scientifically recognized "set" of criteria for good transdisciplinary research, no evaluation method shared and recognized by the scientific community.

Nonetheless, evaluation practices grew over the years, on one hand based on results of research projects investigating the issue of evaluation by, for example, screening transdisciplinary research projects/programs, their successes and failings and the resulting quality criteria, on the other hand based on evaluation strategies developed by researchers themselves or by reviewers being thrown into evaluation procedures.

In this workshop we will
1) give a short overview on the actual discourse on evaluation of transdisciplinary research;
2) present some principles for (self-)evaluation in transdisciplinary research; and
3) have two reports on evaluation procedures in two different fields and settings of research (sustainable consumption; research-education cooperation).
Questions that will be discussed are e.g. the integration of actors from outside academia in the evaluation of research, the impact of self-evaluation on the collaboration within a research team, and the question who shall participate in the internal evaluation of a transdisciplinary project in order to ensure the quality of the evaluation process itself.

Peer Review Processes for Transdisciplinary Articles: A Reflection on Journals’ Practices
Organizers:
Manuela Rossini (td-net and Institute of Advanced Study in the Humanities and the Social Sciences IASH, University of Bern), Susanne Wymann and Anne Zimmermann, CDE, University of Bern)
With prsentations by editors and reviewers of the following journals and book series:
GAIA: Almut Jödicke and Susanna Bucher
JAR, Journal for Artistic Research: Henk Borgdorff
MRD, Montain Research and Development: Susanne Wymann and Anne Zimmermann
NSS, Natures-Sciences-Sociétés: Claude Millier

Publishing is an essential means of validation and communication of research. This is no different in transdisciplinary research, where publishing also aims at contributing to the development of society through sharing of knowledge. In the scientific world, authors need to disseminate and validate results, reflect on issues, and participate in debates. On the other hand, institutions and individuals are assessed according to their publication record – as probably the most influential of all current evaluation criteria. the space between article production and counting impact factors, journal editors and reviewers play an important role in defining Occupyingand using rules to assess and improve the work submitted to them. Publishing transdisciplinary research poses specific challenges, in particular with regard to peer-review processes, as it addresses different knowledge communities with different value systems and purposes.

The workshop will contribute to a reflection on the peer-review process, which we understand as an essential evaluative process in transdisciplinary research practice. More specifically, the workshop will reflect on how the peer-review process can be shaped to serve the specific needs and the promotion of transdisciplinarity.

No established set of rules exists as yet for reviewing transdisciplinary work. Exchange on reviewing and publishing experience can help us fill this lack. We have invited a few editors of journals who have been publishing work considered to be transdisciplinary to share their practices and ideas. In the first part of the Workshop, these guests will present their perspectives and review practices. Among the questions they will answer are the following:

The second part of the workshop will be open for a general exchange between all participants: we will discuss common challenges and critically review how they are addressed by each journal’s specifically designed practices and rules.

Keywords: Regional Centres of Expertise (RCEs); education for sustainable development; multicultural; process assessment; storytelling

RCEs aim to build regional networks of actors, NGOs, GOs, research, educational or business institutions to exchange experience and contribute together on regional sustainable development. At the same time, RCEs are embedded in an international network of today more than 80 RCEs that cooperate in research and development programs. United Nations University – Institute for Advanced Studies (UNU-IAS), based in Yokohama, Japan serves as the Global Service Centre and supports RCEs in networking, knowledge exchange and development. The challenge for UNU-IAS as well as for individual RCEs is to work and cooperate in this multicultural environment of sustainability approaches, diverse regional challenges and individual organizational structures of RCEs. RCE projects differentiate from poverty alleviation programs to Mangroves protection initiatives or social entrepreneurship and business initiatives to educational research and development actions and many more types of actions and programs. At the same time it is the aim of the Service Centre as well as by each RCE and the whole community to assess the local and global impact of the network.

RCE Graz-Styria has become one of the main drivers in the development of RCE assessment methodologies during the past years and has collected data by RCEs from across the globe to find ways for objective, qualitative and quantitative assessment. Methods used range from indicator-oriented assessment to qualitative process assessment as well as storytelling methodologies. They are oriented to be transdisciplinary and combine numerous characteristics:

The global RCE community and their projects provide a manifold playfield for development and testing of transdisciplinary assessment methodologies. In the course of the presentation, experiences, outcomes and methods for sustainability impact assessment with RCEs will be presented and challenges discussed.

Frédéric Darbellay and Theres Paulsen, Institut Universitaire Kurt Bösch (IUKB), Switzerland

The design of an evaluation process should be closely linked to the particular definition of the inter- and transdisciplinary approach of a submitted project at the epistemological, theoretical as well as conceptual level. We consider inter- and transdisciplinary research (ITDR) as a macro-process which involves the interaction of theories, methods and practices among three complementary sub-processes: (1) the integration and synthesis between, across and beyond scientific disciplines and paradigms (epistemologies and research methodologies) in order to describe, analyze and understand the complexity of theoretical and practical questions in various fields; (2) the integration on the inter-institutional, managerial and structural level aimed at developing new organizational strategies and modes of governance that will be adapted to inter- and transdisciplinary research practices; (3) the integration of academic and non-academic networks and actors to identify complex problems, develop research questions and elaborate solutions in life-world contexts. This complex and multidimensional conception of ITDR is a new mode of knowledge production and requires therefore an evaluation process that is adapted to this complexity. Evaluation can no longer be exclusively based on disciplinary or multidisciplinary criteria. That is why ITD researchers demand an evaluation model for proposals, processes and outcomes that integrates: a) the reorganization of disciplines and scientific paradigms; b) the specificities of the institutional and structural dimensions of ITD in different cultural contexts and epistemic communities; and (c) the new ‘deal’ between science and society, with the implication of external networks and stakeholders.

We will explore these multiple dimensions through a critical self-analysis of some of our experiences with evaluation. We recently submitted – successfully – projects in the inter- and transdisciplinary field of Children’s Rights and Tourism Studies to the Swiss National Science Foundation. In our presentation we question the submission process in general and have a closer look at the specific questions to the researchers on their interdisciplinary approach. Finally, we like to discuss the tensions between disciplinary excellence on the one hand and the need to create a dialogue between multiple disciplines on the other. In conclusion, we will formulate a nuanced answer to our original question: is the evaluation process currently practiced outdated or can it be changed and reformulated? And how could it integrate the multiplr dimensions of ITDR?

Heidrun Moschitz and Robert Home, Research Institute of Organic Agriculture FiBL, Switzerland
Keywords: learning; monitoring; methods; knowledge systems; agriculture

The Agricultural Knowledge System (AKS) is a collective term for the public and private organisations dedicated to research, education and extension, and their interaction with farmers and other knowledge users. However, AKSs lack the flexibility and responsiveness needed to fully support innovative initiatives for sustainable rural development, so networks have developed that function outside the main AKS. These “Learning and Innovation Networks for Sustainable Agriculture” (LINSA) include farmers, consumers, NGOs, experts and local administrations, and operate on the principle of knowledge sharing and learning between farmers and other stakeholders. The EU project called SOLINSA (Agricultural Knowledge Systems in Transition: Towards a more effective and efficient support of Learning and Innovation Networks for Sustainable Agriculture) has been established to gain an understanding of how such networks develop and operate in practice.

SOLINSA is a transdisciplinary research project that will be characterised by policymakers, practitioners and researchers sharing and reflecting their experiences in learning and innovation in agriculture. In particular, the researchers will engage in a process of co-learning with practitioners in different LINSAs and countries. The project partners and a group of expert advisors will together reflect on the learning processes to adapt and refine the collaborative learning and research methods. To this end, an ongoing evaluation methodology is needed that is itself capable of evolving. The aim of this presentation is to describe how the evaluation problem has been addressed in SOLINSA.

The first steps were to establish, in a workshop setting, what the project partners expected from the project. Expectations with regard to the aims of the project were expressed in the forms of personal goals such as professional networking as well as project goals such as gaining a better understanding of the processes in networks. Expectations with regard to (learning and evaluation) methods were expressed in terms of internal needs such as an interactive information system, a focus on local and regional learning, and method selection criteria. The expectations with regard to outputs were expressed in terms of inter-project collaboration, academic output, and output that could have a practical or policy influence.

Evaluating the Broader Impact of Inter- and Transdisciplinary Research Proposals

Organizers: Robert Frodeman and Britt Holbrook (CSID, University of North-Texas) and Christian Pohl (td-net and ETH Zurich)
Chair: Christian Pohl
Introductions: Robert Frodeman and Britt Holbrook

Inputs from representatives from the following foundations:
Mercator Foundation Switzerland: Beno Baumberger (Communications) and Regula von Büren (Project Manager)

the cogito foundation: Simon Aegerter (President)
Swiss National Science Foundation: Angelika Kalt (Deputy Director)
Volkswagen Foundation: Thomas Brunotte (Communications, Knowledge Transfer)

Description

This workshop addresses the needs of public and private funding agencies concerned with questions surrounding the broader societal impacts of research, ex ante, ex nunc, and ex poste. The workshop will begin with a brief account of the CAPR (Comparative Assessment of Peer Review; http://csid-capr.unt.edu/) project. Funding agencies will then be invited to briefly present their approach to evaluating broader impacts of research proposals. The workshop will then turn to a conversation about different approaches that can be taken to the evaluation of broader impacts.

Inter- and transdisciplinary research is often driven by the desire to achieve an impact beyond the science system and the world of ISI-citations. The US National Science Foundation subsumes these impacts under the term „broader impacts“. Since the introduction of the broader impacts criterion in 1997 the US NSF has struggled to define and evaluate the term and balance it with the intellectual merits of research (Frodeman and Holbrook).

The Centre for the Study of Interdisciplinarity (CSID, University of North Texas) is completing a four-year project that has been examining these questions. Called the Comparative Assessment of Peer Review (CAPR, 2008-2012), the project examines how six US and foreign public science agencies conduct the peer review process, with a focus on how broader societal impacts issues are taken into account in the review of proposals.

The aim of the workshop is to reflect on current practices of evaluating the broader impact of inter- and transdisciplinary research proposals. Public and private funding agencies are invited to briefly present their approach, answering questions such as:

What is the agencies understanding of broader impacts?
How is a proposal’s potential for broader impact assessed?
What other questions are agencies facing in this respect?

The agencies’ approaches will be discussed in the light of the actual results of the CAPR project, as well as in exchange with the other funder experiences.

_top page

Sustainability Sciences: Evaluating Strategies of Setting and Shifting Boundaries
Part II: Changing methods: Transdisciplinarity in an era of scientification - its benefits and costs

Organizers:
Willi Haas and Barbara Smetschka (Institute of Social Ecology Vienna, SEC, Alpen-Adria University), Alexander Bogner (Austrian Academy of Sciences), Austria

Scientific excellence calls for scientification and, consequently, for clear-cut boundaries based on disciplinary identities. The demand for being relevant to real-world problems and, if possible, being able to deliver applicable solutions calls for a strong orientation towards non-scientific stakeholders. While each of these requirements for transdisciplinarity research is a challenge in itself, achieving both at the same time may overstress the scientists’ capacities. Against this background, the development of a transdisciplinary methodology can be seen as an attempt at coming to terms with these tensions. Regarding methodology we are interested in the following questions:

Which transdisciplinary methods have developed in sustainability science? Are they an adequate response to the challenges transdisciplinary practitioners face?
How do methods applied set or shift boundaries explicitly or implicitly when considering boundaries between both disciplines and science/non-science?
Are methods in sustainability research being reflected on more deeply now, e.g. have epistemological and ontological underpinnings become an explicit part of transdisciplinary knowledge production?
Is the scientification of transdisciplinarity an one-way option, or do we have also the possibility to come back from there with a next-level move towards the creation of new paradigms of knowledge production, which could be the result of what we learn when we go for a ‚transdisciplinarization‘ of sciences?
With regard to methodological practice: is there an equal attention for scientification and stakeholder orientation or is this seen to be mutually exclusive? What are the costs of different approaches in terms of societal relevance and scientific excellence, respectively?

Presenters:

Sami Mussbach (a paper written by Jens Nenwig, Daniel Lang, Sami Mussbach), Institute for Environmental and Sustainability Communication, Leuphana University Lüneburg

Stephan Rist, National Centre of Competence in Research (NCCR) North-South, Centre for Development and Environment (CDE), University of Bern

_top page

Evaluation Tools and Integrative Designs
Chair: Gabriele Bammer

5.1 How Disciplinary is Interdisciplinary Research Cooperation to Be? Action Strategy Mapping as a tool for formative evaluation and for improving research cooperation

Dirk Scheffler, e-fect (dialog evaluation consulting) Trier, Germany

Keywords: formative evaluation; cognitive mapping; action strategy; cooperation

Which goals do researchers pursue in interdisciplinary cooperation with which actions and under which conditions? Action strategies as knowledge representations can be used for planning, regulation and evaluation of actions. For the theory-based measurement of action strategies the Action Strategy Mapping (ASM) method proves to be very practicable, easy to learn, as well as qualitatively reliable and valid for the representation of action-leading knowledge in the context of a semi-structured interview.
The analyses of 57 action strategies of interdisciplinary research cooperation within three German Collaborative Research Centres show empirically very heterogeneous task-related, cooperation-oriented and personal goals. Communicative exchange is confirmed to be a central and a particularly frequent action of interdisciplinary cooperation. The interdependence of the action strategies correlates with the project interdependence of the researchers. A comparison of the action strategies of researcher with high, middle and low project interdependence shows several correlations: in case of middle interdependence contentwise and structural characteristics of the action strategies correlate positively with self-reported success, but with high and low project interdependence the action strategies shows negative correlations with success.
This paper contributes to the to the conference methodically regarding the application of the ASM method for the purpose of formative evaluation and as a learning tool as well as contentwise regarding the implications for the management of interdisciplinary research cooperation in Collaborative Research Centres, in particular with respect to the appropriate communication and interdependence of the scientists.

5.2 Evaluating Transdisciplinarity in Sustainable Land Management Projects – Design and Perspectives

Sebastian Rogga, Thomas Weith, Nadin Gaasch, Kristin Schulz and Jana Zscheischler, Leibniz Centre for Agriculture Landscape Research e.V. (ZALF) Müncheberg, Germany

Keywords: sustainable land management; evaluation design; impact assessment

The global demand for land accelerates at high pace due to a wide array of driving forces like climate and market changes as well as technological and institutional innovations. In consequence, rising conflicts about land-use are to be expected. On a global scale as well as on regional scale management solutions are much needed. To cope with this challenge the funding measure “Sustainable land management” funded by the German Federal Ministry of Education and Research aims at the development of innovative approaches of spatial governance towards sustainability by focusing on real-world problems. The funding measure officially started in October 2008 with a call for joint research projects and scientific coordination projects. Due to the number of involved scientific disciplines, project stakeholders and target groups, sustainable land management appears to be highly complex. In consequence interdisciplinary and transdisciplinary research seem to be an adequate approach for reflecting needs and challenges.

The coordination project of Module B (in responsibility of the Leibniz Centre for Agriculture Landscape Research – ZALF) set up an evaluation design to measure the impact of the joint projects and their ways of dealing with transdisciplinary research. Hereby the project coordination pursues a mix between ongoing process evaluation and ex-post impact analysis. In a first step transdisciplinary activities within the joint projects will be analysed and typified. After a comparison, which includes existing theoretical approaches, first results will be reflected in interaction with(-in) the joint projects. In the form of face-to-face meetings, which are planned frequently, stakeholder groups shall be integrated into a discursive reflection process. First results will also be reprocessed into presentations and discussion papers which will be documented in internal internet-based fora as well as presented at national and international conferences. As designed, the project coordination will derive further recommendations with special focus on innovative processes, transfer activities and support of value creation chains based on transdisciplinary research approaches. These recommendations will not serve for Sustainable land management joint projects only but shall be utilized and developed by the scientific community at large.

5.3 The Vexed Question of Evidence in Integrative Research Evaluation
Alice Roughley, Australian National University, Australia
Keywords: Integration measures; evidence, performance story reporting

Solutions to complex global issues such as climate change, pandemics or security management generally require a process of collaboration acrss disciplines, policy and practice. Decision-makers need good evidence on which to act. Yet, we still don’t have a good understanding of the factors that facilitate knowledge integration. While the need for integrative research has escalated in recent years, relatively little rigorous empirical evidence exists on the elements of research that enable synthesis of the knowledge of disciplines and stakeholders, the indicators of integration success. There is more discussion of barriers to integration than analysis of the factors that foster it. It appears that barriers to and enablers of integrative research is a topic that authors write about after being involved in an integrative research process; not one that many seriously pursue as a study focus in itself.

In this paper I will discuss evaluation of integration at two levels. Firstly I will outline a recent project I collaborated in; a literature review of barriers to and enablers of integration in the fields of health and environmental and natural resource management. I will explain the project methodology and the findings. The indicators (or types of measures) of integration will be discussed. In the second part of the paper I will reflect on the challenges of evaluating integration in this project, particularly the issue of defining evidence. This paper reveals that context is as central in evaluating integration as it is in doing integration.

_

top page

Parallel Workshop and/or Paper Session 4:

Sustainability Sciences: Evaluating Strategies of Setting and Shifting Boundaries
Part III: Boundary work in research practices: Cases analyzed and lessons learnt from empirical case studies

Organizers:
Willi Haas and Barbara Smetschka (Institute of Social Ecology Vienna, SEC, Alpen-Adria University), Alexander Bogner (Austrian Academy of Sciences), Austria

The third session in this workshop focuses on research practices. Transdisciplinary researchers are boundary workers in a double sense. Establishing (and setting) boundaries – by differentiating between science and non-science across the different stages of td research from framing problems to creating knowledge and using research results – helps to structure a multilayered open research process. Managing (and shifting) these boundaries helps to guide this process towards mutual understanding and a higher probability of an innovative outcome and its implementation. However, a precondition for success is td researchers’ willingness and ability to reflect on their different roles as scientist in td processes, their relation to the science system in general and disciplinary practices specifically as well as group-dynamics, interventions and communication architectures and to consequently facilitate team learning. Relevant questions to be addressed are:

Which kinds of tensions arise in research practice? What kind of boundary-work by td researchers is required?
How do td researchers reconcile in practice the tensions between sound science and practical relevance?
What role play scientific research methods and an analytical-systematic approach to manage the science-society interface?
Which (creative) ways of dealing with these challenges can be observed?
Which lessons can be learnt by analyzing case studies a) for forthcoming research projects and b) for the future of sustainability science at large?

Presenters:

Martina Schäfer, Center for Technology and Society, Technical University Berlin

Michael Stauffacher, Institute for Environmental Decisions (IED) ETH Zürich

Process Assessment II: From Project Design to Implementation

Chair: Lotten Westberg

6.1 Transdisciplinarity in Practice: Experiences in the FarmPath Project

Sharon Flanigan, The James Hutton Institute, Aberdeen, Scotland. Co-author: Lee-Ann Sutherland (project director)

Keywords: sustainability; farming; stakeholders; problem framing; implementation

The overall goal of the European Commission funded FarmPath project, which commenced in March 2011,is to “identify and assess future transition pathways to regional sustainability of agriculture in Europe, and the social and technological innovation needs required to initiate and progress along these pathways”.

By adopting a fundamentally constructivist or ‘mode2-science’ approach to conceptualising regional sustainability of agriculture from the outset, stakeholder involvement towards co-construction of knowledge is a defining feature of this project. This ‘transdisciplinary’ approach to research is not new, but is increasingly being applied in the context of sustainability science. In this poster, we reflect on experiences and identify issues raised in the early stages of designing and implementing this three-year project, including: the balance of power and obligation between academics and stakeholders in formally funded research projects; the practical realities of integrating lay, professional and academic knowledges and discourses; and questions of documenting, assessing and publishing transdisciplinary research.

_top page

6.2 Evaluating Knowledge Co-Production: Climate Proofing Urban Development in the Deepest Polder of the Netherlands

Dries Hegger and Carel Dieperink, Utrecht University, and Annemarie Van Zeijl-Rozema, Maastricht University, The Netherlands

Keywords: Co-production; science-policy interface; climate change; success factors

Hotspot Zuidplaspolder (ZPP) is a well-known Dutch showcase for climate-proofing ongoing spatial planning processes. The project (2007-2009) aimed to arrive at "climate proof urban development" in one of the lowest lying areas in The Netherlands (6.7 m below sea level). The area had been assigned as one of the country’s main areas for urban development, but these plans – combined with projected sea level rise due to climate change – were cause for much media attention and political concern. The project had been designed out of a wish to create scientifically and socially robust knowledge, making the case exemplary for present-day experiments with new forms of knowledge production on the interface between science, policy and society.

Integration of scientific knowledge and ongoing planning processes can be deemed necessary for identifying local consequences of climate change and opportunities for climate- proofing the plans. But such integration can be difficult due to conflicting political or knowledge interests; strategic use of knowledge; or inherent differences between the timeframes, epistemologies and criteria for judging the quality of knowledge in science and policy domains. “Hotspot Zuidplaspolder”, however, is termed ‘successful’ by many, making the project interesting from the perspective of evaluating transdisciplinary research.

This paper aims to shed more light on the opportunities and pitfalls of joint knowledge production for climate proofing spatial planning processes, focusing on the identification of action perspectives for successful knowledge co-production, addressing the following research questions:

How can the success of knowledge co-production be determined?
To what extent can “Hotspot Zuidplaspolder” be termed successful according to these success criteria?
Which factors have contributed to these outcomes and to what extent can we translate them in action perspectives for successful knowledge co-production?

We will argue that ‘evaluating success’ can only be done by developing a working definition which is applied by the authors in a self-critical way, realizing that evaluation of the success of knowledge co-production is always to some extent ‘in the eye of the beholder’: different actors have different perspectives and may have diverging and irreconcilable knowledge interests. In our paper we will take the process-related concepts of salience, credibility and legitimacy by Cash et al. as a starting point. In our ongoing research project we will use the lessons from the Zuidplaspolder case to evaluate eight other cases. In a later stage we intend to use the framework developed to help ongoing projects improving their co-production processes.

6.3 Creating Joint Arenas for Transdisciplinary Knowledge Production at Mistra Urban Futures

Merritt Polk, University of Gothenburg, Sweden

Key words: sustainable urban development; joint knowledge production

The urban problem arena is characterized by a specific type of complex problems. This complexity often defies the spatial, temporal and organizational expertise and mandates of politicians, practitioners, researchers and civil society. The combination of multi-scale and multilevel substantive issues, with multilevel governance structures, business and civil society interests also transcends many traditional forms of knowledge production and problem solving within both research and ‘real-life’ urban contexts. This paper will address this complexity by presenting a framework for transdisciplinary knowledge production and problem solving that is currently being tested and evaluated at Mistra Urban Futures in Göteborg, Sweden. This framework addresses the specific nature of urban complexity regarding multiple values, knowledge diversity and issues of power and legitimacy and is divided into two parts. The first identifies three main components in transdisciplinary knowledge production: 1. joint problem formulation and project design; 2. co-generation of information, materials, conclusions and solutions; and 3. on-going evaluation. The second part consists of five main guidelines that are used to design, support and evaluate transdisciplinary knowledge production. These are: inclusion, collaboration, integration, usability and co-reflection. During 2010 and 2011, this framework has been tested in five Pilot Projects within different parts of the urban knowledge arena. This paper will present the preliminary evaluation of the application of this framework in these pilot projects.

Evaluating Teaching and Education

Chair: Catherine Lyall

7.1 Do We Teach What We Preach? Evaluating Transdisciplinary, Problem-based Learning Projects in Academic Sustainability Programs

Katja Brundiers and Arnim Wiek, Assistant Professor, School of Sustainability, Arizona State University

Keywords: Sustainability science; sustainability education; participatory evaluation; perception

This paper presents a framework and initial applications to evaluate transdisciplinary, problem-based learning projects in academic sustainability programs around the world. These projects have different learning and teaching formats (from classroom exercises to internships and thesis projects); yet, they all are being conducted in transdisciplinary settings that allow students, faculty, and external partners to collaborate on real-world sustainability problems and solutions.
The evaluative framework has been developed from the literature and encompasses the following components:

Faculty’s motivations to engage in transdisciplinary research and to strive for excellence in these projects: What is the individual motivation for faculty engagement and what does the university contribute by means of incentives, rewards, recruitment, support, and organizational learning?
Students’ learning outcomes: Do these projects align with and foster competencies in sustainability science?
Community-University partnerships: Do partnership characteristics align with partnership outcomes to ensure broader impacts?
Contribution to the field of sustainability science: Are the projects framed along key determinants of sustainability and do they lead to innovation and impact?
Process and management: Does the process and project management reflect the participatory approach required for co-production of knowledge and for linking knowledge to action (quality of integration)?

On the process-level, the evaluative framework accounts for the collaborative, reflective and integrative aspirations of transdisciplinary research. It proposes a participatory approach that allows project participants to jointly design and implement the evaluation as well as discuss its results and their implications for the evaluative design and project outcomes. The framework incorporates a perception and cross-perception analysis in order to differentiate evaluation results and further build capacity for transdisciplinary collaboration. Obviously, each of the collaborating groups (students, faculty, external partners) holds distinct motivations and expectations, and contributes different knowledge and skills to the projects. The novelty of this evaluative framework consists in the integration of evaluative research streams and the development of a comprehensive evaluative perspective on sustainability education projects that promise to foster student learning, faculty development, and community engagement. _top page

7.2 Understanding and Evaluating Interdisciplinary Collaboration in Science-Based Technologies Through Practice-Based Learning

Dorothy Sutherland Olsen, Nordic Institute for Studies in Innovation, Research and Education (NIFU), Oslo, Norway
Keywords: Interdisciplinary learning proceses; nanotechnology; socio-cultural perspective

“Science-based technologies” such as micro-electronics, biotechnology and nanotechnologies are playing an increasingly important role in the development of modern economies. These technologies are described as having close links to the developments in scientific knowledge as well as the practical requirements of users and it has been suggested that development in science-based technologies is dependent on and perhaps even driven by, interdisciplinary collaboration. This paper takes a socio-cultural perspective (originating in Russian educational psychology and practice-based learning) and uses concepts of collaborative learning to study Research and Development within nanotechnologies and nanosciences. The paper draws upon several longitudinal studies of interdisciplinary research to identify the diverse learning processes taking place in multidisciplinary Research and Deveopment projects. By studying learning as it happens the focus is moved from the outcome itself to the way in which the outcome is achieved. These various learning processes suggest differing degrees of learning between the disciplines within the same project. Furthermore the study indicates that the presence of people, theories and instruments from different disciplines influences learning, in ways which might not be identified by measuring output alone. The paper argues that by taking a collaborative learning approach and identifying the various learning processes taking place, it becomes possible to develop some of the nuances necessary for understanding knowledge creation in heterogeneous environments. The paper diverges from earlier assumptions that knowledge produced must be interdisciplinary; instead, the paper highlights the interdisciplinary ways of creating any kind of knowledge. Thus the emphasis is on the process rather than the outcome. This way of understanding different learning processes has implications for how we evaluate interdisciplinary research. The paper addresses some of the issues and challenges confronting evaluators of interdisciplinary research and by using learning theory makes some important distinctions between different types of learning. These distinctions form the basis of a potential tool-kit for future evaluation purposes.

7.3 Evaluating Co-creativity: Assessing Diverse Creative and Experimental Outcomes of a Transdisciplinary, Studio-based, Project-organised Program

Charles Walker, Auckland University of Technology, New Zealand
Keywords: Project-organized curricula; design thinking; co-creation; methodologies of evaluation

This paper will present outcomes of a reflexive case study of the evaluation procedures developed for a new transdisciplinary, research-led, studio-based program in ‘Creative Technologies’.

In 2005, Auckland University of Technology drew together four existing Schools (Art & Design, Computing & Mathematical Sciences, Communications & Media Studies, and Engineering) into one new Faculty of Design & Creative Technologies. In 2007, the Faculty formed the Interdisciplinary Centre for Creative Technologies, an experimental research unit and hybrid 5th school, with a broad remit to conceive, develop and implement forward-thinking, inter- and trans-disciplinary research and teaching initiatives across these diverse communities of practice. The explicit linking of design with creativity, technology and, implicitly, with ‘innovation’, seeks to address the limitations of specialized knowledge production within current disciplinary boundaries. However, more pertinently, it also holds out the possibility of developing research and expertise that takes advantage of what has been called ‘design thinking’ to address complex thematic issues or connect previously incommensurable domains. We will specifically examine processes and outcomes of the new Bachelor of Creative Technologies (BCT) degree established in 2008 as a key component of this larger Faculty project.

The BCT is conceived as “a post-graduate program for undergraduates” – focused on the identification of new hypotheses, innovative research-based learning methodologies, and the ‘co-creation’ of original content by academics, practitioners and students. The project-organized curriculum is designed to create collaborative, research-intensive ‘ecologies of learning’, that have the potential to generate new and diverse outcomes in the form of designs, texts, artifacts, installations or performances. At another level, the BCT represents part of this author’s ongoing search for creative methodologies for evaluation that might address Ernst Boyer and Lee Mitgang’s idealistic and much cited, but less heeded, call for “a new educational language … driven by the conviction that the standards used to evaluate performance should be organized not so much around blocks of knowledge … as around modes of thinking: the discovery, integration, application and sharing of knowledge.” However, the notable weakness of Boyer and Mitgang’s thesis is that it proposes a structure without agency, and leaves personal experience and individual meaning-making of situated actors within real institutional contexts relatively unexamined. This paper will address the issue of agency in light of what sociologist Andrew Abbott (2005) calls ‘ecologies of practice’; characterized by “sets of agents, sets of locations, and the fluctuating, continually forming and reforming relationships between … multiple elements that are neither fully constrained nor fully independent”. Within this new learning context, institutional power relationships, disciplinary identities and individual agencies of teacher, learner, practitioner or peer may be altered, fluid or provocative enough to challenge traditional disciplinary expectations. The paper will present early outcomes of the program – the ‘results’ of attempts to reconcile, accommodate, challenge or transcend diverse agencies, epistemologies and research paradigms within an intellectually entrepreneurial institutional environment – and show how the way in which they are evaluated simultaneously reflects, validates and/or problematizes the program’s own institutional and/or symbolic raison d’etre.

_top page

P8:
Assessment Hermeneutics and Concepts
Chair: John van Breda

8.1 How Do We Evaluate the Process of Transdisciplinary Research for Achieving the Twin-Goal of Producing
New Theoretical/Scientific Knowledge as Well as Useful/Practical Knowledge?

John van Breda, Stellenbosch University, South Africa

Keywords: research process; shared problem definition

Transdisciplinary research processes dealing with complex real-world problems, associated with the challenges of sustainability transitioning, are motivated/driven by achieving the twin-goal of producing practical solutions as well as new scientific knowledge of these problems. Achieving this twin-goal when working on hybrid/coupled social-ecological systems problems, which are normally ill-defined and highly contested, poses some serious questions for the evaluation/assessment of each stage of the transdisciplinary research process, as well as the process in its entirety:

How do we evaluate the scientific ‘validity’ as well as practical ‘usefulness’ of the transformation of the real-world problem into the research problem when, at the start of the process, the real-world problem is still in a state of contestation and ill-definition? In other words, when there are no clear-cut disciplinary definitions of the problem at hand and when practical definitions of the problem are highly ‘charged’ by stakeholder interests and values, how do we assess whether the (translated) research problem is sufficiently ‘narrow’ to be useful for producing practical solutions, whilst, at the same time, being ‘general’ enough for producing transferrable knowledge and new theoretical insights?
How do we evaluate the knowledge that is being produced by the separation and analysis of the joint/shared research problem by the different/relevant disciplines involved in the research process? In other words, how do we assess whether the outcomes of disciplinary analysis, the object of study and integration during the next phase of interdisciplinary integration, has the potential of contributing to the twin-goal of practical solutions and new knowledge generation?
How do we evaluate the knowledge that is being produced during the phase of interdisciplinary integration? In other words, how do we assess whether the (re)integration of the (separated/analysed) research problem has produced relevant/appropriate outcomes for collaboration and transdisciplinary integration during the next phase?
How do we evaluate the knowledge and practical solutions being produced during the final phase of transdisciplinary integration? In other words, how do we assess whether the twin-goal of practical solutions and new theoretical knowledge has indeed been achieved? How do we establish whether the translated real-world problem into the joint/shared research problem, separated and analysed by the different disciplines, and (re)integrated twice during the phases of inter- and transdisciplinary integration respectively, will, or may, contribute to the actual resolution of the real-world problem, on the one hand, as well as the generation of new scientific knowledge, on the other hand?

8.2 Environmental Health and Transboundary Water Quality Monitoring in South Africa’s Vaal River: Towards Assessment Hermeneutics

Johann Tempelhoff, North-West University (NWU), South Africa. Co-authors: Andre van Zyl, Deputy manager, Environmental Services, Fezile Dabi District Municipality and BJJ (Kobus) Lombard, NWU, South Africa.

Keywords: Water Quality Monitoring (WQM), assessment, hermeneutics, environmental health, municipal health services

The aim of a project currently underway at North-West University’s Research Niche for the Cultural Dynamics of Water (CuDyWat) is to get environmental health authorities, who oversee water quality monitoring in the Vaal River system of South Africa, to collaborate in transboundary contexts across district and provincial boundaries. The plan is to actively promote the sharing of water quality monitoring (WQM) information on one of the country’s hardest working rivers (Van Zyl, 2011).

In view of the anticipated impact of climate change, the growing threat of toxic mine water pollution, and the porous nature of international boundaries in southern Africa as a result of migration trends, environmental health practitioners are concerned that water-related diseases could have a significant impact on environmental health conditions in South Africa. Although there are a number of policy guidelines, health

officials operating on district and metropolitan municipal levels, do not interact effectively in terms of WQM data sharing (Knight 2009).

For the purposes of the proposed project value-checks will be in terms of assessment instead of evaluation theory. Assessment lends itself more to the developmental circumstances and the heterogeneity of a variety of unique cultural traits amongst the stakeholders participating in the research. Assessment includes the processes of collecting, analysing, interpreting, recording and reporting of required data. In addition, these processes could be extended to indicate how the end results obtained from assessment should be utilized; meaning that results can either be informative in nature and to this effect could be used for decision making, or that it could be employed for improvement purposes. Whether assessment is done for decision-making purposes or for improvement purposes, monitoring is always implied and thus forms a

fundamental component of assessment.

Hermeneutics will feature in the assessment process. In particular, attention will be given to working towards a collaborative platform between stakeholders where natural science methodologies and the ideographic understanding of real life circumstances negotiate strategies to implement transboundary collaboration.

_top page

8.3 Shifting Conceptual Boundaries between Environmental Health and Sustainable Development

Julien Forbat, University of Geneva, Switzerland
Keywords: Environmental health; sustainable development; concepts

This presentation describes the interrelations between environmental health and sustainable development and aims at proposing a new way of conceptualizing environmental health, notably vis-à-vis sustainable development.

Since the end of the 1980’s, an attempt has been made by some of the most prominent international institutions (including the United Nations Commission on Sustainable Development) to integrate the concepts of environmental health and sustainable development. This trend appears problematic in the way that it dramatically reduces the intrinsic complexity of environmental health and induces a growing gap between those institutions and the public health institutions offering a more elaborate view of the concept (notably the World Health Organization). This presentation will analyze the specificity and the common characteristics of environmental health and sustainable development, in order to formulate a more autonomous definition of the former. It will also explain why an integration of the concepts of sustainable development and environmental health might still be relevant by arguing that a careful distinction between “instrumental interdisciplinarity” and “conceptual interdisciplinarity” should be made. The proposition to reconsider common conceptualizations is largely based on the study of the schematic representations of environmental health and sustainable development, because they provide a particularly relevant insight into the core principles defining these concepts. The presentation will then suggest a modified representation of environmental health, synthesising the findings related both to past and recent schematic representations and interpretations of interdisciplinarity.

Panel: Inter- and Transdisciplinary Research: Requirements for Evaluation Procedures

Input and Chair:
Gertrude Hirsch Hadorn (Environmental Philosophy, ETH Zurich)
Panelists: Veronica Boix-Mansilla, Julie Klein Thompson, Philippe Moreillon, Rudolf Stichweh, Jakob Zinsstag.

Inter- and transdisciplinary research play an important role for fundamental understanding, for problem solving, and for reflection-in-action. Do researchers guided by such different purposes of inter- and transdisciplinary research nevertheless face the same challenges? How can purposes and respective challenges be accounted for in evaluation procedures for research? Panelists with a background in inter- and transdisciplinary research will discuss the contributions of the conference to answering these and further questions regarding evaluation procedures for inter- and transdisciplinary research.

td-conference 2011

EVALUATION OF INTER- AND TRANSDISCIPLINARY RESEARCH
Experiences and reflections on best practice

Theme

Programme

Partners

artnersartnerartner

Keynote abstracts

Speakers (Keynotes, Panel, Workshops)

Parallel Workshop and/or Paper Session 1:

Parallel Workshop and/or Paper Session 2:

Parallel Workshop and/or Paper Session 3:

Parallel Workshop and/or Paper Session 4:

Panel: Inter- and Transdisciplinary Research: Requirements for Evaluation Procedures

td-conference 2011

EVALUATION OF INTER- AND TRANSDISCIPLINARY RESEARCH Experiences and reflections on best practice

Theme

Programme

Partners

artnersartnerartner

Keynote abstracts

Speakers (Keynotes, Panel, Workshops)

Parallel Workshop and/or Paper Session 1:

Parallel Workshop and/or Paper Session 2:

Parallel Workshop and/or Paper Session 3:

Parallel Workshop and/or Paper Session 4:

Panel: Inter- and Transdisciplinary Research: Requirements for Evaluation Procedures

EVALUATION OF INTER- AND TRANSDISCIPLINARY RESEARCH
Experiences and reflections on best practice