ASEAS 13(2) | 243 Linking European and Southeast Asian Transdisciplinary Knowledge Production: Lessons Learned by Doing Evaluation Frank Seemann & Christoph Antweilera a University of Bonn, Germany ► Seemann, F., & Antweiler, C. (2020). Linking European and Southeast Asian transdisciplinary knowledge production: Lessons learned by doing evaluation. Austrian Journal of South-East Asian Studies, 13(2), 243-259. This article aims to explore knowledge co-production through a critical (and self-critical) reflection of experiences with doing evaluation within the Fostering Multi-Lateral Knowl- edge Networks of Transdisciplinary Studies to Tackle Global Challenges (KNOTS) project. KNOTS started as a collaborative project to explore the possibilities and increase the expertise of seven institutions from Europe and Southeast Asia in teaching a transdisci- plinary approach at their higher education institutions. Planned as a capacity-building tool for higher education, its main objectives were to create a teaching manual and to es- tablish sustainable networks and knowledge hubs in this field of knowledge production. This was to be achieved mainly by means of summer schools and fieldtrips in Southeast Asia, which would enable learning through practical application of the knowledge devel- oped. The realization of this ambitious conceptual formulation turned out to be pretty complex and this holds for the very process of evaluation itself as well. We discuss and illustrate the specific problems of a strict evaluation in such a complex transdisciplinary project. The notorious complexity of interdisciplinary and the more transdisciplinary projects was further increased by the intercultural, respective, transcultural dimension involved. Topics discussed include structurally immanent difficulties, unintended effects of financial and political constraints, complications caused by hierarchies and language, and effects of cultural differences, especially different university science cultures. In the form of lessons learned during the evaluation process, we give some hints for the devel- opment and implementation of the transdisciplinary approach as a new tool for reaching socially relevant knowledge, especially in cross-cultural settings. Keywords: Capacity Building; Cultures of Science; Evaluation; Knowledge Co-Production; Transdis- ciplinarity  KNOTS AS A HYBRID PROJECT The core idea of transdisciplinary research (TDR) is to conduct real-world research and teaching for people and, explicitly, together with these people. In this paper, we consider TDR as a fundamental approach to do research and not as a method in itself. Even as a project incorporating only social sciences and humanities, we regard KNOTS – in its practical implementation attempts – as transdisciplinary, since TDR as an approach does not automatically require a broad interdisciplinarity in the sense of integrating technical and natural Aktuelle Südostasienforschung  Current Research on Southeast Asia w w w .s ea s. at d o i 10 .1 47 64 /1 0. A SE A S- 00 45 244 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production sciences (“big interdisciplinarity”; Bath & Wedl, 2013, applying this concept to crit- ical gender studies). The project itself was funded by the European Parliament’s ERASMUS+ program for capacity building in higher education institutes (HEI) – a program line that is not specifically oriented to a region, country, or topic (see Dannecker, 2020, this issue). The focus on TDR and on the five universities in Thailand and Vietnam was part of the individual project design of the University of Vienna, which applied for funding. KNOTS aimed at transdisciplinary approaches in general, and specifically at co-operation between institutions of the higher edu- cation sector and non-academic actors. The practical goal (our ‘product’) was to jointly develop a teaching manual for transdisciplinary research by organizing sum- mer schools that included, among other things, application tests of transdisciplinary working methods. It was hybrid in the sense that it focused on capacity-building and teaching, but implemented this within the framework of a people-oriented, research-based approach. This article aims to explore knowledge co-production through a critical (and self-critical) reflection of experiences while doing evaluation within KNOTS. We will do that by referring to some essential aspects of transdisci- plinary and transcultural research in the relevant literature. In a descriptive part, we will present crucial challenges and obstacles in the course of the project, and then make them accessible in a structured, tabular overview. Finally, we will summarize our most important experiences. Coordinated by the University of Vienna, this project brought together partners from five countries (Austria, Germany, Czech Republic, Thailand, and Vietnam) to strengthen the academic capacity to meet “new challenges in a rapidly changing world” (EACEA, 2015). In terms of content, the focus was on three main topics, all of which relate to current problems: (a) social inequality, (b) climate change, and (c) migration. Thus, three global issues were to be approached by using the example of Southeast Asia in an explicitly problem-oriented manner. An important point that is relevant for the following assessment of evaluating activities is the fact that KNOTS was decidedly not a research project, but an attempt to learn collaboratively and to establish an exchange between very different stakeholders on how to create a teaching manual for transdisciplinary research that should be used in the training of trainers (Train the Trainer), and then in the training of students. Activities within the KNOTS project included joint teaching activities, several summer schools and short fieldtrips, as well an international conference. IN-BUILT TENSIONS: EVALUATION WITHIN A TRANSDISCIPLINARY AND TRANSCULTURAL STRUCTURE As a project in the ERASMUS+ Capacity Building scheme, KNOTS was implement- ed by a consortium of nine universities. The hierarchical structure of ERASMUS + projects bears some risks with regard to reproducing global North-South hierarchies, as the consortium needs a specific ratio of Program Countries (EU) and Partner Countries (non-EU)1. In addition, the scheme is set up in a way that directs project 1 Cf. Presentation at Grant Holders' Meeting, Brussels, 25-26 January 2017: Erasmus+ Capacity Building projects in the field of Higher Education Call 2016, Financial management of the Grant, p. 31. ASEAS 13(2) | 245 Frank Seemann & Christoph Antweiler management tasks to program countries, which then makes the implementation of project activities primarily the partner countries’ tasks. Apart from deep-rooted North-South inequities, there were power differences based on gender, seniority, or the epistemic background within and between participating higher education insti- tutions, which intersected with the latent bias in the ERASMUS+ scheme. At the same time, the discussion and shared elaboration of the idea of transdisciplinarity (as a capacity for implementation at partner universities) included the demand for a partnership and cooperative approach (see Bärnthaler, 2020, this issue). For this reason, the authors and the QM team expected that the combination of an intended partnership with hierarchical relations within a hybrid project construction would bring along very specific complications. However, a glance at the literature makes it clear that many of the experiences we made are “common dilemmas in participation and cross-cultural partnership” (Angeles & Gurstein, 2000, p. 31), even in capacity-building projects.2 Dependencies on external funding, North-South power asymmetries, pre-defined hierarchy pat- terns, communication problems, and time pressure, both in the academic routine and vis-à-vis the sponsor (Binder, Absenger-Helmi, & Schilling, 2015; Schmidt & Neuburger, 2017), seem to be more the rule rather than the exception. For exam- ple, Angeles & Gurstein (2000) problematize the overall concept of “partnership” and speak of “the dilemmas of partnership and participation” (p. 40). Thus, they ask: Can there be equal partnerships between unequals? Are partnerships almost al- ways a form of limited and negotiated relationship, and therefore rarely equal? Our language and use of this seemingly egalitarian word tends to mask ine- qualities in resources, capabilities, and accountability of governments, funding agencies and NGOs between and within the North and South, as it obscures who takes the initiative in setting agendas. (Angeles & Gurstein, 2000, p. 40). From our experience, this is true also if we add Higher Education Institutes (HEI) to the sentence. In our case, these kinds of dilemmas occurred, although especially the European project members were particularly aware of these problems and tried to avoid them as far as possible. Hierarchical structures seem to be so ‘normalized’ as well as multi-faceted that they are hard to erode. An outstanding headline in a related article reads: “Digging Deeper: Old Roles Reproduced” (Schmidt & Neuburger, 2017, p. 63). The same authors point out that: The concept of ITR [inter- and transdisciplinary research] itself, developed in the North . . . applied to “problems” in the South, demonstrates such hegemonies that likewise characterise the dichotomy between the praised diversity of voices in futures studies and the western control and domination in the scientific dis- courses when designing, publishing, and citing such futures (Sardar, 1993).3 2 In the context of this article, we will limit ourselves to a few selected articles on this topic, which argue from a broad database and thus provide a fairly comprehensive picture of the state of research. 3 In recent publications, the definition of transdisciplinarity is contested (there are conceptual devia- tions and semantic slippings). Depending on the field of application and discipline, similar concepts are termed and abbreviated differently (see e.g. Hirsch Hadorn et al., 2008; Padmanabhan, 2018). 246 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production This proved to be particularly relevant within the KNOTS project, as the Thai part- ners have a longstanding expertise with their own participatory research format, which was developed in Thailand: Thai Baan (“villagers' research”), aiming at a transformative research focused on local needs and local competences and using qualitative methods allowing participation (Chainarong, n.d.; Chayan, n.d.; Heis & Chayan, 2020, this issue). Despite many years of experience, Thai Baan tended to be subsumed in KNOTS as a form of transdisciplinary research approach. In contrast to these similar findings on social and structural aspects of trans- disciplinary projects, we had to realize that the existing, not very extensive and partly inconsistent literature on quality management in transdisciplinary research could only be used to a limited extent. This was due to the status of the project between capacity building on the one hand, and teaching, learning, and exercising of transdisciplinary approaches on the other. Thus, an external expert recom- mended the very detailed and practice-oriented criteria catalog of the Institute for Social-Ecological Research (ISOE) during the quality management workshop later in the project (Bergmann et al., 2005). Indeed, this could have been a good orien- tation if KNOTS had been an explicitly research-oriented TDR project. However, most of the required criteria turned out not to be really appropriate, applicable, or answerable, as the project was not primarily a research project, and practice of transdisciplinarity and relevant research methods were only briefly examined dur- ing the fieldtrips. EU-Quality Management (QM) guidelines and requirements also proved to be of little help for the reflective and formative part of evaluation, as most of them were very formalized and focused on descriptions of the activities carried out and on the specific and measurable performance indicators – mainly in boxes of predefined tables with a limited number of characters. Apparently, what Angeles & Gurstein (2000, p. 31) wrote in their report on three participatory, transcultural research pro- jects still applies to the transdisciplinary approach, namely the experience of: How little things change when new orientations (e.g. capacity development, participatory development) and operating principles (e.g. gender-sensitivity, participatory approaches) are introduced within bureaucracies-as-institutions that are historically and socially constructed frameworks for devising behavioral rules of conduct based on technical knowledge, rational planning, routine, standardization, regularity, and predictability (Goetz, 1997; Staudt, 1997). This quote confirms the experiences of the Vietnamese project partners described by Doi (2020, this issue). Several authors consistently follow an “output-outcome-im- pact” concept for evaluation – a concept coming from project management (Binder et al., 2015, p. 547; Schuck-Zöller, Jakob, & Cortekar, 2018, p. 31). Other authors point out that “quality standards in transdisciplinary research are . . . not as clear-cut as it might be in case in other academic fields” (Lang et al., 2011, p. 38). One reason might be that, following Pettibone et al. (2018, p. 224): More than other forms of research, TDR itself needs to be understood as a nor- mative instrument, that means as part of an explicitly transformative political ASEAS 13(2) | 247 Frank Seemann & Christoph Antweiler agenda. Normativity therefore extends beyond epistemological issues of good scientific practice into the moral and political arena. Especially for our hybrid format (capacity building in higher education in and through the use of TDR), the differentiation between product-related outputs and process-related outputs (Binder et al., 2015, p. 547) seemed to be helpful: Process-related outputs are intangible and largely experiential, including (1) methodological, (2) organizational, and (3) social experiences. Methodological experience captures how actors from different backgrounds become familiar with each other’s way of working, including problem definition, language, methods, and working culture . . . . Organizational experience relates to the practical experience gained by planning, managing, structuring, and executing the project . . . and involves analyzing during or after the project whether or not the project plan matched the actual process. Social experience is defined as the interaction with other actors, entities, or institutions. Positive interactions build trust (as an impact of the social experience) while negative ones reduce it. Taking into account the central aims of (1) developing a teaching manual for TDR, (2) introducing TDR at the partner HEIs, and (3) forming TDR-oriented sustaina- ble knowledge networks, the project objective of KNOTS was more education- than research-focused. “Choosing appropriate criteria may thus depend on the project’s objectives and its normative orientations (e.g., policy-, education- , or science-fo- cused)” (Pettibone et al., 2018, p. 224). For this reason, it seemed to be most crucial for the QM team to focus on the evaluation of the above-mentioned intangible out- puts. We also found confirmation for this approach in Angeles & Gurstein (2000, p. 51), who wrote that: [It] is the need in these projects to design and manage clearer capacity-build- ing indicators developed by project participants that “focus more on pro- cess and behavioral change” (Morgan 1997, p. iv) than on the conventional “inputs-outputs-outcomes-impact” schema used in results-based management (RBM). Such indicators have greater diagnostic value in providing project par- ticipants better information and motivation in their work. Also Schuck-Zöller et al. (2018, p. 34) point out that, in order to do justice to the complexity in transdisciplinary processes, qualitative procedures are often required. It is precisely this qualitative approach that leads many of the mentioned authors to the conclusion that transdisciplinary approaches and participatory methods are viewed quite critically from a conventional perspective. These authors point out that “transdisciplinary research and similar collaborative approaches are not uncontested outside transdisciplinary research communities” (Lang et al., 2012, p. 27) and claim that “experience-based guidelines that build upon demonstrated success (and fail- ures) and satisfy all parties involved in transdisciplinary research are needed” (Lang et al., 2012, p. 27). With the following description of our experiences, we aim to con- tribute to that need, explicitly focusing on challenges, obstacles, failures, and gains of the KNOTS project. 248 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production AN EVALUATION OF EVALUATION WITHIN A TRANSDISCIPLINARY PROJECT Experiences in Doing Evaluation During KNOTS Measures for quality management (QM) provided for in the project description included quantitative and scheduling tasks as well as qualitative reviews of the pro- ject progress and its main end products. The more quantitative data (participants, dates, deadlines, financing, etc.) were reviewed by the project management of the Department of Development Studies, University of Vienna (from now on UNVIE), while the qualitative evaluation mainly of the summer schools and fieldtrips – which is the focus of this article – was carried out by the Department of Southeast Asian Studies, Institute of Oriental and Asian Studies, University of Bonn (from now on UBO), and the authors of the present paper. Additionally, two external peer reviews, a detailed mid-term, and a final report were added as internal QM measures. While preparing for the task of quality management, we had been dealing with the necessity of an evaluation through the lens of an explicitly transdisciplinary, planned project. But KNOTS was also an intercultural (or transcultural) project, since universities of different nations, and students with different language backgrounds were involved. In accordance with our literature review, this led to the insight that conventional methods of disciplinary evaluation would not be appropriate. They cannot simply be transferred and applied directly to a transcultural TDR project, due to the multiple forms of co-operation, scientific, cultural, and political backgrounds, methods, and theories involved. This applies all the more to the very special mix of transdisciplinary research and the focus on capacity-building of KNOTS. For that reason, we decided for a more or less discursive and formative evaluation. Rather than purely summarizing an inventory of project results and outputs, this initiated learn- ing processes via questionnaires and regular feedback loops during project meetings. For the following part, we mainly use our experiences resulting from (a) our func- tion as responsible for quality management, and (b) our role as participant observers of three summer schools and accompanying fieldtrips. After participation, we reflected on these observations during project meetings with our KNOTS partners. Furthermore, we used formal questionnaire sheets (closed questions) and evaluation via qualitative questionnaires (open questions). In addition, we draw on informal conversations with students and participating staff, and discussions among staff dur- ing the organizational meetings. Kick-Off Meeting in Vienna – Intercultural Experiences The official start of the project was in October 2016, but a kick-off meeting and the joint project work could only begin in March 2017, as contracts with the EU were only available at that time. The fact is worth mentioning, as the project was not extend- ed by these missing five months. This led to considerable time pressure already at the beginning of the project phase, and also affected the establishment of the qual- ity assurance activities within the project. As the review of literature showed, time pressure is considered by several authors as a major cause for asymmetries and com- munication problems in such transdisciplinary and transcultural projects (Binder ASEAS 13(2) | 249 Frank Seemann & Christoph Antweiler et al., 2015; Schmidt & Neuburger, 2017), affecting team- and trust-building activities and the agreement on a common understanding of key concepts and vocabulary and common language (Angeles & Gurstein, 2000; Lang et al., 2011; see also Bärnthaler, 2020, this issue). In order to meet the project schedule, preparations for the first summer school and fieldtrip had to start immediately, while, at the same time, the necessary organ- izational structures for the overall project had to be created. Another time-related decision was to organize a summer school and a fieldtrip in one block, one right after the other, and to combine them into two weeks instead of calculating two weeks for each at different times. None of the university staff could imagine to leaving work for four consecutive weeks due to their academic duties. For ecological reasons, air travel could also be reduced this way. For evaluation, however, this decision had an unintended effect, since both events had to be evaluated immediately, one after the other, without the possibility of transferring the experiences from one onto the other. Furthermore, none of the consortium members was an expert in QM. More time for content preparation and arrangements on the meaning, extent, and form of QM would have led to a better common understanding of evaluation measures and subsequent adjustments. For that reason, for instance, it proved necessary to organize a QM workshop at mid-time of the project – with reasonable results, but definitely too late for major changes in attitude and implementation. Another revealing experience during the kick-off meeting was the moderators’ explicitly non-hierarchical working method. The intention was not only to rectify the hierarchical structures implied by the ERASMUS+ scheme, to avoid any neo-colonial structures and top down governance within the project, but also to ensure an open, brain-storming and inclusive atmosphere to all participants. The actual effect, how- ever, was that existing power relations and cultural differences concerning hierarchy and working style between Asian and European partners remained inadvertently unadressed. While the ERASMUS + Capacity Building in Higher Education framework that envisages European colleagues ‘teaching’ partner countries from the Global South was openly addressed and circumvented, other unchangeable, structural hierarchies (e.g., financial and workflow control by UNIVIE, as well as seniority hier- archies, age, gender, and epistemic background) remained tacitly in place. Similar situations arose several times during the project, for example, with regards to terms of reference, conceptions, and project tasks such as dissemination or the role of quality management. This use of less hierarchical working methods was well-in- tended, but in effect problematical. For example, although it was obvious that UNVIE had the lead, was therefore organizing the project in general and had the lead of the kick-off event, several simple cooperation tasks (like brainstorming on possible non-academic stakeholders) were carried out in laborious, time-consuming participa- tory small group work instead of giving participants binding tasks for the meeting in advance. In the end, this led to time pressure and a lack of reflective space for reaching a common understanding of important issues, for instance, to discuss what and how evaluation should be conducted. Furthermore, this attempt at a hard, anti-hierarchi- cal approach led to some irritations not only amongst the Asian colleagues. If one were to compare the KNOTS project with a typical transdisciplinary pro- ject, these restrictions in time hampered the ideal-typical Phase A (framing the topic 250 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production and building a collaborative team), as described in many models (Lang et al., 2012, p. 28). This initial time pressure was transferred to the following phases; other typical QM strengthening formats, such as reflexive meetings and discussion forums to pre- vent conflicts and build up common understanding also became affected. First Round Trip – Cultures of Science Interacting The first round trip aimed to visit all the partner universities in Thailand and Vietnam, identify non-academic stakeholders, and organize a stakeholder conference in Bangkok. This brought to light new constraints in the preparation and implemen- tation of QM measures. The team of UBO had prepared questionnaires for evaluation of these events on a discursive basis with a focus on open questions. The idea was to learn from strengths and structural weaknesses, as well as from mistakes and failures made, and from unintended effects of some measures. It turned out that the survey worked relatively well among the present partners at the same academic level – even though academic colleagues from Vietnam, in particular, showed that they were less familiar with a cooperative working environment. Beyond that, feedback from stake- holders at the conference was hard to assess due to language problems and problems of understanding. In retrospect, one could have expected that similar problems will occur with students at the first summer school in Hanoi. Unfortunately, due to a lack of experience at Vietnamese universities, the QM-team stuck with the chosen approach and instruments. During our trips and meetings in Thailand and Vietnam not only cultural dif- ferences in terms of participation and hierarchy became apparent. Different science cultures sometimes made it hard to find common views on conceptual issues (such as transdisciplinarity) and concrete processes (e.g., knowledge transfer and its eval- uation). While the Thai colleagues, for instance, came from a department that was leading in qualitative social science research, working at the margins between aca- demia and activism (Heis & Chayan, 2020, this issue), the approach of our Vietnamese partners proved to be strongly quantitative and statistically oriented (see Doi, 2020, this issue). Accordingly – and strengthened by the requirements of the EU – ideas about QM measures varied from pure counting methods (participants, stakeholders, events, page), on the one hand, and the description of processes and the creation of feedback loops on the other. Different ways of communication made it even more complicated: Far from the assumption that there is a clear definition of transdisciplinarity, at least the authors and several European partners believed that a common understanding had been reached at the kick-off event. However, during the trip the discussions on transdis- ciplinarity blazed up again and the topic was discussed controversially again. Some of us had missed the simple fact that open arguments, which are commonplace in West-European countries are simply not customary in Southeast Asian universi- ties. To raise objection directly is possible in informal settings, but less common in formal meetings or open discussion rounds. In effect, a “yes or “ok” does not nec- essarily mean agreement or consensus. Angeles & Gurstein (2000, pp. 52-53) report similar experiences with participatory approaches branded as “new” or “foreign imports” from a Vietnamese project. Due to this fact, problems already thought to ASEAS 13(2) | 251 Frank Seemann & Christoph Antweiler be overcome reappeared in practical implementation and had to be worked on again theoretically (Bärnthaler, 2020, this issue). This also applied to answers in evaluation questionnaires. For example, a positive answer regarding the degree of progress of the common understanding of transdisciplinarity did not mean that later – during practical implementation – theoretical rejections did not reappear, with the effect that some participants felt ‘thrown back’ to the beginning of the project. However, our joint progress consisted in the fact that, by the end of a project, the participants became increasingly aware of these hurdles and learned to overcome them more pro- ductively – something that cannot be overestimated, even though it can hardly be measured by conventional QM means. Even at this early state, we became aware of the enormous bureaucratic work- load caused by EU-requirements, which were quite opaque to the Asian partners with their different bureaucratic background. Often, these restrictions dominated the project management meetings as they were perceived as ‘hard’ tasks compared to the ‘soft’ ones, such as QM. This influenced not least the ability and willingness to return questionnaires on time and thoroughly filled out. Corresponding meas- ures required frequent reminders and inquiries during the entire project and, thus, reinforcing management-related hierarchies of the North/South nexus, which the consortium struggled to minimize in academic interaction. This corresponds to experiences Schmidt & Neuburger (2017, pp. 61-63) made with the key position of German project members and their irritation and helplessness about this “interme- diary position between the funder and his hardly negotiable demands and on the other hand the hard(ly) approachable . . . partners. . . .The historically loaded pow- er-relations within the team thereby seemed to silence open debates on situations of conflict” (2017, p. 63). Summer School in Hanoi – Socio-Political Contexts Matter The first summer school and fieldtrips near Hanoi brought together university teach- ers from Asia and Europe as well as students from the Thai and Vietnamese partner universities. The first parts of the teaching manual for TDR had to be presented and tested during the summer school, while fieldtrips would enable practical testing. In addition to the difficulties mentioned above, some students were overloaded with participatory approaches and language problems, both due to quite different scien- tific and institutional contexts. There were significant differences in English skills, depending on the program in which they were enrolled. The Vietnam Academy of Social Sciences in Hanoi (VASS) and its Southern Institute in Ho Cho Minh City (SISS)4 run programs in Vietnamese only, while the Chulalongkorn University and Chiang Mai University have international study programs in English language, which attract also students from Vietnam. As important as the language difficulties, were the different educational backgrounds of participating students and teachers. For these reasons, many of the ideas, concepts and methodological approaches could only be discussed in a very rudimentary way, under the time pressure the project faced. 4 At this point there was a major restructuring at the operative level of the Open University (OU) in HCMC, which is why students from OUHCMC did not participate in the Summer School and field trip in Hanoi. 252 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production The same applies to answers in the evaluation forms. In addition to the difficulties of open criticism, some answers showed that the questions were not really under- stood. More time – instead of distributing of the questionnaires at the very end of the summer school – and an explicit explanation of the questions would have prevented this. In addition, the agreed continuity of attendance for all participants in the overall event was unfortunately not guaranteed. Due to other commitments, some teachers and students could not participate in the fieldtrip following the summer school and were replaced by other people who had no idea about the summer school. In such cases, a reasonable evaluation was not possible, as both events could not be assessed in their interaction. At the same time, student feedback, particularly of those students from Vietnam, who were not so often exposed to international cooperation, showed a big interest in the unfamiliar, ‘foreign’ ways of teaching and researching and in the underlying theoretical constructs. The authors observed that by working in a diverse, interna- tional team with open discussions and theoretical arguments, the participants of the event in general benefited from a new perspective on academic cultures and practices elsewhere. The more democratic, respective, liberal forces in the Vietnamese science community seemed to be enabled to discuss topics usually not openly discussed or to try methods off the official scientific agenda. From our perspective, such impacts as the acquirement of some kind of ‘global citizenship’ are of great (also political) interest and could be valued more within the evaluation criteria of the ERASMUS+ funding scheme, even if these learning expe- riences are difficult to document and are not explicitly mentioned in the project application. Taking this into account, further questions for an evaluation of the pro- ject success arise: (1) How to evaluate positive project results that were originally not intended? (2) How to evaluate results that can only be communicated subliminally or that should not be explicitly mentioned at all? In the end, the official evaluation feedback lacks information on these important results. The final evaluation and discussion of the QM had made it clear that further adjustments and changes in the structure and composition of the elements of teach- ing and practice of research methodologies at the next event seem necessary. One of the consequences was to examine the possibility of including students from European partner institutes during the next summer school in Thailand; a second was to assign UBO as co-organizer for that event. In order to further increase the participative teaching approach and the mutual learning effect between participating staff and students form different regions, concrete tasks for co-design, documentation, and evaluation (also part of a research seminar in their home university at UNVIE) were already assigned to all students before the summer school. Summer School in Chiang Mai – Challenges of a Hybrid Project These adjustments proved fruitful, as observations of the intensity and degree of participation in the discussion groups and during exercises made clear. In infor- mal discussions and separate evaluation rounds at their home universities, students later mentioned the benefit of direct contact and exchange between colleagues from different universities, particularly the exchange between Asian and European ASEAS 13(2) | 253 Frank Seemann & Christoph Antweiler colleagues. However, two new issues came up in Chiang Mai: Hierarchies between Asian and European students and differences in sensitivity on gender topics, such as the amount of active participation of female versus male students, sometimes mixed with the question of unequal participation of Asian, respectively, European students. Especially students from the European partner universities showed a high degree of sensitivity and willingness to discuss both topics openly. Since we had decided to collect the QM questionnaires from the students at the end of the summer school, and to consult the KNOTS staff via email later on, it became possible to gather reactions on these surprisingly-openly addressed hierar- chy and gender topics in their forms. Remarkably – subliminally also present in the consortium – this was the first time that the topics of hierarchy (related to structural power and alleged expertise) and gender were openly mentioned and questioned among the project partners. Although there was considerable feedback at that point and several discussions followed in later joint meetings, the topics were more or less limited to the students’ interactions, rather than the situation among the project members. Even though it might have influenced the subliminal awareness of the whole group, there was no room for an explicit discussion of such an unreflected reproduction of power asymmetries, as requested in the following feedback from a lecturer: All of these hierarchies and separations appear to be rather “natural”: No one of us can escape problematic identities forged in dominative social relations. Therefore, I believe they are unavoidable – there is nothing that could be done about it except to deal with them explicitly, for example, to address and discuss them (but that’s already a cultural bias), because they are and remain problem- atic as such. (respondent anonymized) Closely related to the issue of hierarchy were further difficulties in evaluating the results. The answers of Asian and European students differed in scope, detail, and understanding, and were often difficult to relate to each other. That was especially true for open questions. Here, the reasons already discussed (language, conversational culture, and discontinuity of presence) certainly played a significant role. However, the student questionnaires also revealed unexpected side-effects per- taining to understandings of international cooperation, or aid: The experiences of both, Asian and European students in Southeast Asia and with TDR had a positive influence on their own academic attitudes and expertise at home. This also applies to the participating departments where transdisciplinary ideas were increasingly discussed and implemented within teaching. Such reverse capacity building in the European institutions contradicted the underlying ERASMUS+ logic that European project partners guide Asian institutions in capacity building. In our case, this did not correspond to reality, as some of the Asian academic partners (e.g., at Chulalongkorn or Chiang Mai university) had a deeper expertise in transdisciplinary work at the level of research. “Whose capacities are we building” is a question that Angeles & Gurstein (2000, p. 57) define as a starting point of capacity building projects. For us, too, this question became increasingly important during the course of the project and was also given greater consideration in later evaluation runs. 254 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production The essentially hybrid character of the project always remained an obstacle during evaluation processes. Because of the overlapping roles and personal union of scien- tists, project managers, quality managers, and teachers, again and again the point had to be stressed that this is not a research project. The main point was to create and develop a teaching manual in traineeship and practice rather than to apply the transdisciplinary approach to real research situations, which the project design could not accomplish either in terms of time or structure. Nevertheless, it was precisely this lack of practical implementation that was repeatedly criticized in the feedbacks of students (and also of staff sometimes), which distorted the overall view of the pro- ject’s success. As a consequence of the greater involvement of students from different partner universities through special assignments – sometimes overlapping with quality man- agement tasks (e.g., interviews, documentation, etc.) – the diversity of results led to an abundance of information that was difficult to summarize and often not com- patible5. This information overflow, and the difficulty to evaluate answers to open/ qualitative questions of the forms used, was intensively discussed during the 2nd round trip, and a project and quality management meeting in Europe. As a conse- quence, UBO brought in the Center for Evaluation and Methods (ZEM) – a central facility for quality assurance at the University of Bonn to co-design and statistically and graphically evaluate the questionnaires. This made them more self-explanatory for participants and reviewers. Another consequence, after the good experiences with a dissemination workshop for all consortium members, was to organize a qual- ity management workshop with external experts during the following meeting in Europe. Although this was actually scheduled too late in the course of the project, the workshop confirmed the discursive evaluation methods that had been chosen for a transdisciplinary project, especially with regard to its additional transcultural character. LEARNING CO-PRODUCTION VIA TRIAL-AND-ERROR Summer School in Ho Chi Minh City and Final Conference in Bangkok After the third summer school, the new questionnaires, which could be evaluated statis- tically and graphically, made it easier to communicate the evaluation results, although it must be noted self-critically that no suitable format was found for many observations in quality management, for example, for the completely different atmosphere during the summer school and fieldtrips in Ho Chi Minh City (HCMC) compared to Hanoi. It could not be discussed – mainly for political reasons – whether the greater theoretical openness and freedom in the practical implementation during the fieldtrip were effects of (1) the increased trust and shared experience during the project, (2) the different atti- tude to the exchange of ideas in the southern part of Vietnam, or (3) the fact that this time the event was organized by a private university with strong ties to the government instead of a research institution directly under the government of Vietnam. 5 For a structured analysis of knowledge production and transfer during the summer school in Chiang Mai see Braunhuber, Goisauf, and Reinisch (2019). ASEAS 13(2) | 255 Frank Seemann & Christoph Antweiler Before our last meeting in Bangkok, the final feedback and evaluation process of the draft of the teaching manual began. This revealed another unresolved problem: the unspecified role of the evaluators and their institutions. For some consortium members, the QM task was limited to the preparation of feedback materials and the collection and processing of the corresponding responses, while, for others, critical reflection on the materials resulting from the course of the project was also part of the QM. This, in turn, was seen by the former as a transgression of competence, even though critique was meant as constructive feedback. Through intensive discussions and an increased involvement in the Train the Trainer sessions at the final confer- ence in Bangkok, this conflict was finally resolved for everyone. Overall, the consortium members painted a much more positive picture of the success of the project in the evaluation questionnaires of the final conference than in previous evaluations. This may be related to greater confidence in the project, but also to the improved questionnaires. However, it also fits in very well with the pos- itive assessments we received from the external evaluators. Some of the structural deficiencies were still criticized, but the overall evaluation was very positive, not least because of the many unintended positive spin-off effects and the perspective of sus- tainable cooperation between the partner universities, which builds on the mutual trust that had grown. Summary of Main Conditions and Contexts On the one hand, we consider the project to be successful, as it has achieved most of its formal objectives (teaching manual, implementation of TDR at the participating HEIs, establishment of knowledge hubs on TDR and of sustainable cooperation). At the same time, it has not been less successful on the ‘soft skills’ side, precisely because we learned so much from our mistakes, limitations, obstacles, and differences. The following list is therefore not intended to focus on the difficulties or to question the success of the project. Rather, we would like to recommend it as a list of circumstanc- es that future transdisciplinary and transcultural projects – possibly with EU funding again – should pay attention to in order to avoid some of the detours and failures we have experienced. From a self-reflective perspective, we conclude that we could have achieved even better results within and through quality management if we had been aware of the complexity and the stumbling blocks of this particular transdisciplinary and transcultural project from the very beginning. CODA: EVALUATING EVALUATION IN CO-PRODUCTIVE PROJECTS It would go beyond the scope of this paper to discuss all the above-mentioned circumstances in detail once again. See table 1 for a summative overview of main obstacles. We would like to take a closer look at just a few of the facts that seem par- ticularly important to us. First, there is the late start of the project due to contractual problems: If we imagine these five missing months would have been at our disposal, how much preparation (not only of the quality management process), confidence-, and expertise-building could have been done during this time? Our experience shows that the time ‘saved’ comes as a Pyrrhic victory and a complication for the actual 256 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production Figure 1. Main Conditions and Contexts of the KNOTS Project (Authors' Compilation) Constraints through the EU- format Logic of an EU-ERASMUS+ project, especially the pretended knowledge/ capacity gap, which did not meet the real situation Factual power hierarchies between the partner universities via control of finances and schedules Bureaucratic overload due to EU-requirements, often not fitting the project de- sign Delayed start of the project with no compensation, causing stress and shortage of time and insufficient preparation of, e.g., quality management Financial arrangements and restrictions, like lack of funding for essential parts of a transdisciplinary project (financed participation of non-academic stakeholders) Gap of financing between Asian universities and European partners (equipment vs. travelling) Internal constraints Content Structural Different scientific backgrounds: development studies vs. area studies, here South- east Asian Studies vs. Vietnamese Language Studies Different science cultures: more natural science-oriented vs. more social science- oriented and vs. humanities-oriented Hybrid nature of the project (between educational capacity building and research) All members came from social sciences or humanities, which is unrealistic for a real TDR-project and limited the benefits for the non-academic stakeholders Combination of summer school and fieldtrip due to time and ecological impact Deficiencies in continuity of the participants Unexplained role of quality management Real and underlying hierarchies within the project that influence evaluation: lead university vs. other partner universities, European vs. Southeast Asian uni- versities, Thai vs. Vietnamese universities, big vs. small departments, state lead institutions vs. private institutions; Europe vs. Asia (post-/neo-colonial gap) on staff and student level; Students vs. university teachers; (gender ratio; age (even more important in Southeast Asia) Unexpected and surprising changes and necessary adjustments Dominance of hard topics (financing, scheduling, planning the next event) vs. soft topics (evaluation, reflection) during the rare meetings Time restriction of all participating members as part of a university body Cultural Differences concerning participation and hierarchy Different ways of communication (lo. Hierarchy) Miscommunication on basic terms and tasks (due to cultural differences) Language: English, Thai, Vietnamese, (Karen, Lao,….) Between staff, students, stakeholders/people concerned Political (within countries) Freedom of research (Vietnam), hierarchies within the political landscape (Vietnam), restricted expression of opinion (Vietnam and Thailand) Positive results and progress that can only be subliminally communicated Other limitations Difficulties in interpreting open questions (due to cultural, hierarchical and language reasons) Sufficient documentation of observations made Unintended positive effects Capacity building also in European universities (staff and students) Better understanding of political, social, educational and scientific situation at the participating universities and in the countries TDR activities at European universities Support of democratic, respective liberal groups, bringing unorthodox ideas and methods into the discussion, strengthening forces in civil society ASEAS 13(2) | 257 Frank Seemann & Christoph Antweiler project work. Sufficient time for preparation and common agreements before the project starts seem absolutely necessary to us. This would allow for brainstorming, sighting, and open discussion of possible problems and restrictions as well as a dis- cussion of the project design itself, instead of a mere presentation of the framework conditions and schedule of the project, as we had to do due to time pressure. This lead time could also have included important workshops to build up common exper- tise (e.g., in dissemination or quality management). A jointly discussed and thus shared attitude on the role of (not only) quality man- agement in general and the position of responsible persons would have fostered the attitude that evaluation (like dissemination) is not only something we had to do for the EU-officers – as it sometimes seemed – but something that is essential for our own learning process through the project and for achieving the objectives of our project in general. This would have meant, for example, giving (or, self-critically, demanding) much more space and relevance to the exchange of evaluation results during the project meetings. In reality, hard facts such as finances and planning of further activities often came to the fore, while feedback results communicated via email or cloud folders often received apparently little attention. The same applies to the role of the quality managers. In our opinion, reducing their activities to the mere collection, documentation, and dissemination of feedback from participants limits the possibilities of quality management as a collegial questioner and regulator. Like project management, quality management should also be explicitly assigned a reflec- tive role in such projects. Presumably, more lead time would have included also a more comprehensive investigation of the transcultural aspects. With the Southeast Asia Department in Bonn and the Department for Vietnamese Language in Prague, experienced experts to identify cultural constraints in advance were on board, which would have enabled at least European partners a better understanding of many situations. Beyond the transcultural aspects, the partners and stakeholders involved had quite different backgrounds, experiences, and expectations. In terms of area-knowledge, less experienced partners have been involved in the case of (a) docents and students from Europe not experienced in Southeast Asia, and (b) students from Vietnam and Thailand having neither experience in the respective other Southeast Asian country nor with European science culture. This is also true for most of the stakeholders at the political and administrative levels as well as for those involved at the local level during our fieldtrips, including peasants and local workers. The language barrier contributed to a less intense involvement of these partners than projected. Beside a financing problem, there was also the lack of time on our side as well as on the side of the mostly busy local interlocutors. Hierarchical relations and shyness also played a role. The overall rare feedback they gave was difficult to interpret, since most of them – contrary to the transdisciplinary approach – had no insight into the project and mainly expected concrete help in difficult situations – something that the proj- ect, which was neither real research nor involved natural-science colleagues, could not achieve. On the other hand, by implementing this project and especially quality manage- ment measures, we experienced several unexpected outcomes and positive spin-off effects hard to document and to evaluate. One was the quite dynamic interaction of 258 | ASEAS 13(2) Linking European and Southeast Asian Transdisciplinary Knowledge Production students via social media, also expressing commendation and criticism of the KNOTS- project. For example, after a hefty critical remark during the Chiang Mai summer school posted by a European university staff member to a limited social media group, stating that some fieldtrips might develop into a mere tourism activity, several par- ticipants were frustrated and provoked comments on why this critique had not been made offline to the whole group, which would have allowed an open debate. Another revealing positive spin-off effect was the comparative view of similar top- ics and learning processes in higher education, but in different countries and different university institutions and scientific cultures. This also included seeing different stu- dent audiences during the fieldtrips. All this gave us opportunities to learn about processes of knowledge creation beyond the specific project objectives in particular. Much of this was based on participant observation and, for us, one of the major gaps in the quality management process was that we did not find an adequate evaluation format for such observations. It would be worthwhile to work out solutions before further projects. We would like to conclude with one of the most positive academic effects for us: the sheer experience of transdisciplinary research and transcultural teams work- ing together in real-time/real-space contexts. Especially from the evaluators’ point of view, we succeeded in going beyond purely programmatic statements that can be found in most US- or European literature on TDR. Seen in this light, the multiple limits of transdisciplinary research, especially in transcultural contexts, can them- selves serve as an empirical window to transcultural reality.  REFERENCES Angeles, L., & Gurstein, P. (2000). Planning for participatory capacity development: The challenges of par- ticipation in North-South partnership in capacity building projects. Canadian Journal of Development Studies, 21 (special issue), 31-62. Bath, C., & Wedl, J. (2013). (Große) Interdisziplinarität. Braunschweig: Braunschweiger Zentrum für Gender Studies. Retrieved from https://www.fg-gender.de/wp-content/uploads/2010/05/FG2013_Forum- Interdisziplinaritat_Paper-Protokoll.pdf Bärnthaler, R. (2020). Conflict, controversy, compromise, and compression: The pragmatics of transdisciplinary (development) projects. Austrian Journal of South-East Asian Studies, 13(2), 193-210. Bergmann, M., Brohmann, B., Hoffmann, E., Loibl, M. C., Rehaag, R., Schramm, E., & Voss, J.-P. (2005). Quality Criteria of Transdisciplinary Research. A Guide for the Formative Evaluation of Research Projects. ISOE-Studientexte, No. 13 (English Version), Institute for Social-Ecological Research (ISOE), Frankfurt am Main, Germany. Binder, C. R., Absenger-Helmi, I., & Schilling, T. (2015). The reality of transdisciplinarity: A framework- based self-reflection from science and practice leaders. Sustainability Science, 10(4), 545-562. Braunhuber, B., Goisauf, T., & Rinisch, J. (2019). The (re)production and transfer of knowledge in the con- text of the ‘KNOTS Summer School’ in Chiang Mai 2018 - a critical approach. Working Paper No. 8. University of Vienna, Vienna, Austria. Retrieved from https://ie.univie.ac.at/fileadmin/user_upload/p_ ie/INSTITUT/Publikationen/IE_Publications/ieWorkingPaper/ie.WP8_Braunhuber_2019-12.pdf Chainarong, S. (n.d.). Case study for empowerment and democratisation high level panel. Thai Baan research (villagers’ research): Local wisdom for resources management. Living River Siam Association. Retrieved from http://www.livingriversiam.org/2work/tb/tb_a7.html ASEAS 13(2) | 259 Frank Seemann & Christoph Antweiler Chayan, V. (n.d.). Thai Baan research: An overview. Retrieved from https://www.iucn.org/downloads/ thai_baan_research_an_overview_1.pdf Dannecker, P. (2020). Transdisciplinarity ‘meets’ power structures: Challenges and experiences of a capa- city building project on transdisciplinarity. Austrian Journal of South-East Asian Studies, 13(2), 175-192. Doi, N. M. (2020). Institutional prospects and challenges to transdisciplinary approach in the knowledge production system of Vietnam: Reflections on a north-south partnership project. Austrian Journal of South-East Asian Studies, 13(2), 229-242. EACEA (European Commission). (2015). Call for Proposal EAC/A04/2015, Development & Global Studies: Fostering multi-lateral knowledge networks of transdisciplinary studies to tackle global challenges (KNOTS), DETAILED DESCRIPTION OF THE PROJECT (unpublished project document). Heis, A., & Chayan, V. (2020). Thai Baan methodology and transdisciplinarity as collaborative research practices. Common ground and divergent directions. Austrian Journal of South-East Asian Studies, 13(2), 211-228. Hirsch Hadorn, G., Biber-Klemm, S., Grossenbacher-Mansuy, W., Hoffmann-Riem, H., Joye, D., Pohl, C., ... & Zemp, E. (2008). The Emergence of Transdisciplinarity as a Form of Research. In G. Hirsch Hadorn, H. Hoffmann-Riem, S. Biber-Klemm, W. Grossenbacher-Mansuy, D. Joye, C. Pohl, ... & E. Zemp (Eds.), Handbook of Transdisciplinary Research (pp. 19-39). Dordrecht: Springer Netherlands. Lang, D. J., Wiek, A., Bergmann, M., Staufacher, M., Martens, P., Moll, P., & Thomas, C. J. (2012). Transdis- ciplinary research in sustainability science: Practice, principles, and challenges. Sustainability Science, 7(supplement 1), 25-43. Padmanabhan, M. (2018). Introduction: Transdisciplinarity for Sustainability. In M. Padmanabhan (Ed.), Transdisciplinary research and sustainability. Collaboration, innovation and transformation (pp. 1-32). Abingdon: Routhledge Pettibone, L., Blättel-Mink, B., Balázs, B., Di Giulio, A., Göbel, C., Heubach, K., . . . & Wyborn, C. (2018). Transdisciplinary sustainability research and citizen science: Options for mutual learning. GAIA, 27(2), 222-225. Schmidt, L., & Neuburger, M. (2017). Trapped between privileges and precariousness: Tracing transdisci- plinary research in a postcolonial setting. Futures, 93, 54-67. Schuck-Zöller, S., Cortekar, J., & Jakob, D. (2018). Evaluation transdisziplinärer Forschung und deren Rahmenbedingungen - Vorüberlegungen zur Nutzung im Bereich von Klimaservice. Journal for Research and Technology Policy Evaluation, 45, 28-37. ABOUT THE AUTHORS Frank Seemann (M.A. Comparative Religion) is a lecturer and scientific assistant at the Depart- ment for Southeast Asian Studies at the University of Bonn. He was the organizational director of the KNOTS project at the department in Bonn and was responsible for quality assurance and evaluation within the project. His main areas of interest are Buddhism in mainland South- east Asia, ethnic minorities in Southeast Asia, and ecological issues related to the region. ► Contact: frank.seemann@uni-bonn.de Christoph Antweiler (Dr. phil. Cultural Anthropology) is a professor of Southeast Asian Stud- ies at the University of Bonn. He was a member of the KNOTS project and worked within the quality assurance and evaluation within the project. His main areas of interest are develop- ment, ethnicity, and political ecology in insular Southeast Asia, and urban culture, provincial identity, and cosmopolitanism in urban Indonesia. ► Contact: christoph.antweiler@uni-bonn.de