PBBL course evaluation – preliminary thoughts

Thursday evening my colleague Heilyn and I had the final session in the PBBL course and we had asked the Learning Teams to prepare evaluations of the course. We have actually designed several evaluation activities, mainly because reflection-on-action is an important component is our pedagogical philosophy, but also because the course is part of the EU funded research project COMmunity of integrated Blended Learning in Europe (COMBLE) and as such we need to document our findings. Friday Heilyn and I did our weekly meta-reflection on the course as a whole, and I think Heilyn was a bit surprised, since I was not willing to say that the course has been a complete success. Judging from the various types of feedback we have received from the participants, there is no doubt that the course has been a success. Both the participants and Heilyn and I as designers/facilitators/researchers have learned so much and all in all it has been a very engaging, rewarding and fun experience. As part of the evaluation process we asked the participants to look at their individual expectations they all stated in the application form before entering the course and comment on whether the course had meet their initial expectations or not. Here are some quotes from their responses:

It confirms what I maybe knew before: No one can be learning by a system alone. The systems are just supporters for all the good intentions and the heavy work done by the facilitators. And it is probably not the time alone, it is the timing too, and the whole spirit from the facilitators which inspires the learners to show up and participate every time.

Now I know: I was wrong in the assessment of my previous knowledge about BL – thank you very much for the new insights.

My expectations, on the other hand, were more than fulfilled. Also, I like fulfilling other people’s expectations. I did learn some theory which will be useful, and the collaboration in our virtual team was wholly enjoyable. I did get re-started nicely on teaching in second life and I am taking a bag full of ideas home with me. Especially on PBL and how to deepen my application of this theory, but also with respect to how much time this takes when you take it seriously as an educator.

The work in our team has pleased me very much. It was surprising to see that we as a small group already had so many different approaches and we had longer discussions than I thought before, but we have reached a very cooperative process of discussion of how we ultimately manage and present our mini project. Working in our team was really fun!

In short – all these expectations are met and are even exceeded. In a little longer – at work we used a bit “americanized” AL in some high potential training courses. That is basically all I knew about this kind of adult learning method and now I feel I have a good first insight what PBL means and what you can achieve with it. This will very much help me to influence what we do in the future with such programs at work. Also seeing what online learning opportunities beside just simple WBTs are out there was eye opening for me, however I am not so sure what I am going to do with that new knowledge in regards to me current work environment.

Also thank you very much for an interesting course – you have done a great job, and also shown us, how much time we have to be willing to invest, if we want to implement BL courses in our practice. A very important information for the institution before the boot. I wanted to get better assumptions to transform f2f courses into BL and get knowledge about PBL and technology to have a better foundation to choose the right method and technology for the target groups. I do think I got it all. It has been very interesting to learn about the methodology and explore the virtual environment. The team process has been excellent with lots of good discussions.

For me this course was more than I expected. I liked BL, when something was difficult in audio version, ppt gave the missing parts. I learned lot from Comble members in SL. Very useful was reading forums. The most boring for me was my computer, movements in SL were slow, arrows buttons were slow, there was kind of break or time between me to press the button and me to move in SL. Then I pressed the button again and again and once, when I started to move in SL I went and went … and got lost. This course gave me a new dimension of teaching. From flat Internet to 3 D in SL, from one tool lecturing to BL. Thank you all, it was fun.

I learned very much – but unfortunately also something I didn´t care to know about myself: I am not any longer resilient enough for such a giant course. I didn´t visit each optional meeting and I’ve only been able to read about a third of the requested literature efficiently. After some days full of daily life I couldn’t manage to stay fully concentrated trough the evening sessions. On the other hand it has been great to see the advantages of SL, to gain a lot of experiences and to develop my aims to handle PBL and AL. Thank you for this very intensive but great experience.

I really enjoyed learning in SL which was a complete novelty to me. I regret that I wasn’t able to turn up during all SL optional meetings as I really liked them. For someone like me, to whom getting around in SL (broadly speaking) was really completely new, the tours gave many hints how to let yourself being educated in the virtual world and maybe, in the future, how to organize the learning process there. Before the course I had just theoretical basis of PBL/AL (quite superficial, I believe). Now, I know much more, have a lot of materials and some experience to use.

When entering the course I was hoping to experience SL as e-learning platform on my own skin and it happened on the skin of my avatar ;) But seriously I really had a chance to see how education process may look like in SL. I think it is too early to state that now I will also use it in my courses but I do really have something to think about. If in the future I had opportunity to provide similar course I think I would be ready from instructional point of view but there is still a lot of things I should learn about PBL… In fact during that course I realized how complex and difficult it is and how hard it is to implement it into the teaching/learning process.

Like everybody else, I want to thank you for the fabulous work, you have been doing during this course. I will forever be inspired and have your doing in my mind working with BL. It is hard to find words to express my experience during this course. I am amazed, and I really feel sad, that it’s now over, even though it has been hard work  :-) I feel connected of course mostly to the danish team but also to the rest of the participants in a way, that I never thought possible after 6 weeks working together only online and never meeting in RL. I have really learned a lot too.

Now looking at these comments and the rest of the feedback we have received, it may come as a surprise why I’m somewhat reluctant to label the course a complete success, so I’ll try to elaborate a bit on that as a) course designer/facilitator and as b) researcher.

As course designer and facilitator my responsibility together with Heilyn has been to design for optimal conditions for learning to happen in order to meet the promised outcome. Even though taking responsibility for one’s own learning is very much part of the PBL pedagogy, the facilitator still has a huge responsibility to try to ensure that learning actually can happen (through the design), and when evaluating a course it thus makes sense to look at the expected outcome and review the final learning result. In this PBBL course the general course objective and the outcome was the following:

The main objective of the course is to provide the participants with a combination of conceptual, theoretical and practical strategies with regards to designing, implementing and teaching/training courses of different duration in blended modality using an overall PBL approach.

The participants will learn how to design blended learning courses/learning units based on a PBL approach.

There is absolutely no doubt that the participants learned a great deal about Blended Learning (BL) and PBL and that they got a lot of inspiration and new knowledge with regards to designing, implementing and teaching/training. But I would not feel comfortable in saying that they actually learned how to design, implement and teach/train – and how to are the operative words here. Heilyn and I discussed this and agreed that the course outcome actually should have been differently stated with an emphasis on learn about instead of learn how to, since we both agree that learning how to is way too ambitious for a short 6 week part time course, where the majority of the participants had little or no prior experience with BL, PBL and the 2 chosen course technologies, Moodle and Second Life (SL). We cannot propose to prolong the course, since our 3 European COMBLE partners find longer courses unrealistic to implement in their local settings. Consequently we will suggest changing the course outcome, when we do our final report, but even so I find it necessary to review the design and see if anything could or should have been done differently.

The most important course component aimed at learning how to design, implement and teach courses based on a PBL approach was the Mini-projects we asked each Learning Team to produce. As mentioned in a previous post only half of the 6 Learning Teams managed to hand-in their projects in due time, 2 teams handed in later and 1 team resigned. And in my book that is not satisfactory. Insufficient time to do the projects has been stated as the main explanation, and so we need to look at the general workload and consider carefully if the project description, incl. assessment criteria can be done differently. Another issue pointing at a revision is the quality of the projects. When looking at the quality of the 5 project reports, 3 of them were quite good content wise, whereas the 2 final reports did not in a convincing way show that the teams were able to identify characteristic and attributes of PBBL, discuss the advantages and challenges, align goals with local standards and design a course.  The course has been offered by Aalborg University at master level, and as such we have been assessing the projects according to our local standards for academic reports at this level.  The course has shown that there are cultural differences in what is to be considered a good academic report, and if we had more time in the COMBLE project, this would be something I would want to investigate further. For the final report we will recommend assessment criteria based on local standards and this would also influence the content of the projects.

Other ways of ensuring a higher quality would be to;

  • Form the Learning Teams in the first week and get them started earlier
  • Ask the Learning Teams to submit their problem formulations earlier
  • Ask the Learning Teams to submit an example of their writing before handing in the final reports
  • Have a synchronous supervision session with each Learning Team

Given that project work is a core element in the way we do PBL at Aalborg University I would hesitate to reduce the Mini-projects further, but would prefer to reduce some of the other course activities. We have had 2 lectures and 1 optional meeting pr. week, and in our final report we will suggest only 1 lecture and 1 optional meeting pr. week. In this course we also included Action Learning (AL), and even though there are similarities between PBL and AL, Heilyn and I agree that in a future course we would not include AL, since it seemed to create some confusion among the participants and bringing in AL in the projects did not improve them quality wise.

Heilyn and I have also discussed the use of especially SL. It is evident that the steep learning curve in SL has been very time consuming for the majority of the participants. Depending on course objectives one could chose a different environment for synchronous communication and probably gain some time, but it would be at the expense of some other important factors, which leads us to the research perspective.

As mentioned above this course has been part of a research project and Heilyn and I had the task of designing, implementing and evaluating a Blended Learning course based on innovative technology and Problem Based pedagogy. Besides this we decided to formulate an investigation question more targeted at our respective PhD interests and we came up with this question:

How can deliberate pedagogical design promote community creation and motivation among participants?

In order to investigate this question we decided to apply an action research approach meaning that we worked in (small) cycles of planning, acting, observing and reflecting throughout the course. We also decided to share our findings with the participants inviting them to comment on our meta-reflections, and this really turned out to be a great learning experience for all of us. Based on the feedback we got from the participants, we were also able to act proactively and in this way I think our research perspective supported our functions as designers/facilitators a lot. We planned many different activities targeted at promoting the community creation and motivation, and we both agree that the course has been a huge success in this sense.  Many of the activities we planned were carried out in SL, and I’m convinced that the immersive and engaging character of SL contributed greatly to the community creation and the motivation.  I was happily surprised to see so many participants show up for especially the lectures in-world, and I’m not convinced they all would have been as motivated if the synchronous lectures had been in a different platform.  This part of the project is something Heilyn and I will investigate further in 2 papers for the PAN PBL2010 Conference and the 7th Networked Learning Conference 2010. For now I’ll finish with a quote from one of the participants, who had no prior experience with SL:

In the beginning I though SL is only some funny tool/place and more gaming than really learning something seriously. However after getting over the voice issue I had in the beginning I really feel that his is somehow better than just telecons with sharing PowerPoint presentations. Somehow it’s more fun and more engaging to join in for sessions and being able to write within different groups in parallel to speaking is very interactive and useful. …. Someone that normally favors f2f against any sort of technology meeting.

Summing up the course has been a success, but there still is room for improvement, and this is actually what I appreciate most about being a course designer, facilitator and researcher – it is a lifelong learning experience :-)


More observations from the PBBL course

So we’re approaching the final week of the PBBL course and it is time to reflect a bit. My colleague Heilyn and I have managed to reflect on the course activities for every week and we have posted these reflections in Moodle, so that the participants could get an idea of why we have designed course elements in certain ways and what we have thought of the outcome. These meta-reflections  (on which the participants have commented) have proved to be very valuable in both our and the participants’ learning experience and it is definitely a course component I want to further develop in my next course. In this post I’ll highlight some of our reflections.

In the first SLecture the text and voice check took almost 45 min. and Heilyn and I agreed that this was unacceptable, so in the second SLecture we reduced the scheduled time to 15 min. and we put out further information on voice settings etc.  It worked for the majority of the participants, but there still seems to be 1-2 participants in every session struggling with technical problems. We actually asked everybody to check out if they could access especially SL before enrolling in this course, but in future courses this should be stressed, since we cannot help/ do anything about local problems e.g. firewall problems, inadequate graphics cards, poor internet connections and lack of headset etc. In designing this course we had planned a preparation week where participants were expected to solve a newbie assignment aimed at learning the basic skills in SL. These activities were optional and unfortunately very few participants decided to make use of this offer. For future courses we would recommend making newbie assignments/activities mandatory.

SLecture 2 in session …

In this course the participants are expected to work in Teams and Heilyn and I decided to form the teams based on nationality. The argument for this type of division was quite pragmatic. We knew that all participants would have difficulties in finding time to participate, so we figured it would ease their process if they were able to work in their own languages  for some of the activities (mainly the Mini-project work), and in the second SLecture these Learning Teams met for the first time.  Whether this has been a good decision we’ll know more about in the final course evaluation …

Learning Team Germany decided to meet on one of the Pink Elephants …

The third SLecture was Heilyn’s first real teaching experience in SL, and I think she did a great job. Her presentation was about Action Learning and by the end of the SLecture the Learning Teams were asked to go shopping for Team mascots.  I asked Heilyn about her experience and she pointed out two things. First of all she did not consider doing a presentation a creative way of teaching. I agree, but I do believe presentations are necessary when the participants have little or no prior experience with the subject matter, come from very different pedagogical cultures, speak different languages and in general have very different prerequisites. Lecturing then serves the purpose of trying to establish common ground, common language, common terminology and as a means to clarify questions. I also believe that the participants’ general SL skills should be taken into consideration before planning more difficult activities. Besides this, I do believe that lectures in SL are quite different from conventional f2f lectures due to the possibility of using simultaneous voice and text.

SLecture 3 in session …

Another point Heilyn made was that she felt somewhat alone during the SLecture, in the beginning she felt worried because she couldn’t see the participants’ reactions. It felt like talking to an empty space, but then the comments started to appear in the chats. By the end she was more tired than usual after a 2 hr lesson. Before the SLecture, we discussed if she should ask the participants to wait with their questions till after the presentation, but I recommended her to allow for questions during exactly because I thought she might otherwise feel quite alone – and it is also more interesting for the participants if they are allowed to contribute to the discussion. But lecturing this way and in a foreign language certainly is something you need to learn and especially when there are many questions/comments it can be difficult to stay focused. I’ve logged both local and group chat from all the lectures and as an example there were a total of 33 pages in the second SLecture – that is a lot of information you need to attend to while lecturing …

Discussion on the MI theory …

Recognizing that participation in lectures is not the most interesting way to get to know SL, we have also included Optional visits, and in the first visit I showed the participants Zotarah Shepherd’s MI design and Thursday Xu’s designs of Bloom’s taxonomy and Wenger’s CoP.  Touring, exploring and trying out things together in an informal way combined with discussion on topics that are somewhat peripheral to the subject matter is in my opinion a very nice activity, and judging from the feedback the participants also appreciate this kind of activity.

Discussion on the CoP theory …

The fifth SLecture was on PBL, networked learning and web 2.0 technologies conducted by my other colleague, Thomas. I was not able to attend, but judging by the comments in Moodle the participants enjoyed the lecture. At this point in the course Heilyn and I decided to make part of the lectures more interactive, since the participants now seemed more at ease in SL. For the 6th SLecture we prepared 5 statements on PBL/AL and used them together with the Opinionater tool. This turned out to be a very interesting exercise. We had not anticipated that the participants would disagree to the extent that they actually did.

No consensus on problem types in PBL/AL …

The Opinionater is a quite simple tool, but highly effective. We managed to have some interesting discussions on different aspects of PBL/AL and in that sense the tool also served as a formative evaluation tool giving us educators some hints on the participants’ perceptions. After this exercise I took some of the participants to U21 Global Island where we tried out the Metaphor Tour.


The Metaphor Tour … a trip focusing on constructivist learning  …

In the second optional visit we met my friend, Inge Qunhua who is a Danish educator and SL designer. Inge has a lot of creative ideas and has made several small displays of her designs on her island.

Investigating one of Inge’s designs …

The Bank setting in Inge’s holodeck …

The Kindergarden …

In the seventh SLecture this week we experimented with video display, but it did not turn out quite as we had hoped it would. However, we have not yet reflected on this experience, so I’ll return to that in a future post. This week the participants were expected to hand in their Mini-projects on Friday. Only 3 out of 6 teams did so, and this in my opinion is not satisfactory.  For the last 10 years I’ve been using project work in my teaching and I’ve never before had the experience of only half of the students/participants handing in projects. One team has asked for an extension of the deadline, whereas the two other teams haven’t given us any indication of what’s going on.

Mascot display
Learning teams displaying their mascots in the sandbox …

This course has been designed as a test course with the aim of finding out whether this certain design is sustainable and even though we all have learned a lot, I’m not convinced that this particular design is suitable for the general learning objectives.  Learning about PBL – especially the way we see this at Aalborg University, where we combine PBL with project organization – is a process that needs a lot of time. I’ve been skeptical of the very short course period (6 weeks, incl. preparation) from the beginning. We actually wanted to design a 10 week course, but preliminary investigations among our project partners indicated that a 10 week period would be too long. I can think of several reasons as to why only half of the teams managed to hand in their projects on time, but I’ll wait with further conclusions till we have done both project and course evaluation next week….


Problem Based Blended Learning course next 6 weeks

Monday August 24th my colleague, Heilyn Camacho – also from e-Learning Lab at Aalborg University, and I kicked off a 6 week course on Problem Based Blended Learning (PBBL).  We have designed the course as part of an EU funded research project called COMmunity of integrated Blended Learning in Europe (COMBLE). In the COMBLE project my colleagues and I from Aalborg University work together with 3 other lead partners from Estonia (University of Tartu), Germany (The University of Applied Sciences Wildau)  and Poland (Marie Curie Sklodowska University).


We have all been offering the course to our affiliated partners which means that we have participants not only from the above mentioned countries, but also from Argentina and since Heilyn is from Costa Rica we really have an international cross-cultural setting. Most participants come from the academic sector, but we also have a few participants from the cooperate world. Besides this my colleagues, Thomas Ryberg and Jacob Davidsen will at some point bring in Danish on-campus 7th semester students from Human Centered Informatics.

The main objective of the course is to provide the participants with a combination of conceptual, theoretical and practical strategies with regards to designing, implementing and teaching/training courses of different duration in blended modality using an overall PBL approach. There is no consensus on how to define Blended Learning. We will, however use this definition from Sonja Trapp (2006:1) as a starting point:

Blended learning can be defined as the combination of multiple approaches to pedagogy or teaching, e .g. self-paced, collaborative, tutor-supported learning or traditional classroom teaching. Blended learning often refers specifically to the provision or use of resources which combine e-learning with other educational resources.

What makes this PBBL course particular interesting and challenging is the fact that there are no f2f activities for the adult participants – we only meet each other online in different platforms. For the on-campus students this of course will differ a bit, but as I understand it their main activities will also take place online. Heilyn and I have chosen Moodle as our main platform for the teaching and learning activities.


Flexibility not only regarding place, but also time is a keyword in this course, and in our point of view Moodle provides a good environment for especially text-based, asynchronous activities. Our participants are located in different time-zones and should be able to contribute at any convenient time, but a far more important argument for using an asynchronous platform for the main activities can be found in our pedagogical foundation. We have chosen to blend Problem Based Learning with elements from Action Learning and in both strategies reflection is a keyword.

By and large the course design is heavily inspired by another course I run at the Masterprogramme in ICT and Learning (MIL), and based on my experience from that course we have also chosen to use Second Life (SL) as main platform for synchronous activities in this PBBL course. The course outline can be seen in the picture below:


Since Aalborg University does not own land in SL – I only rent a small place on the Island Wonderful Denmark for my MIL course – our Polish partner has kindly let us use their Second UMCS Island for the course. Below is a picture of what the main teaching and learning area looked like, when we fist arrived there:

Second UMCS owner, UMCS Maximus, and I discussed the design of the area …

Later Heilyn and I met to discuss the design …

One of the first changes was replacing the existing presenter with The Clever Zebra presenter …

My dear friend, Coughran Mayo, recommended the CZ presenter and thanks to Fleep Tuque’s great video, I easily managed to set it up :-) Even though Heilyn, Thomas and I will use traditional presentations in our in-world lectures we still wanted the environment to better reflect our pedagogical foundations, so we also had the white chairs removed.

Yeah! .. lots of empty space to play with …

This week of the course is preparation week meaning that the participants are expected to familiarize themselves with both Moodle and SL. Only a few of the participants have prior experience with SL, so this week has been spent on learning the basic features of SL. As a preliminary setting I put up a few objects to enable the participants to practice very basic, but important skills such as learning how to walk and sit:


I rezzed 5 chairs close to each other – trying to walk in between without bumping into any of them is not an easy task for newbies ;-) I also rezzed “The Opinionater”, created by Entropy Hax, which can be bought for the tiny sum of 99 Lindens at this location. Besides being a great tool for decision making I like the fact that “The Opinionater” is interactive making walking practice a bit more fun.

This week I’ve also held optional in-world “Get-off-to-a-good-start” meetings, and here are some pictures from those meetings:

Heilyn and I getting ready to embark the Testis Tour with participants Dido and Ina

Um, “immersed” inside a testis …

and outside a womb in Doctor Asp‘s (my regular landowner :-) amazing holodeck

Participant, Hans and I trying to build Bears … LOL not easy, since the Bear is meant to wear!

The Bear was created by Danish resident, Kaj Bing, and can be found in Wonderful Denmark’s Freebie Factory, where you’ll also find my favorite SL object; The Pink Elephant created by Asp himself :-)

Last night participant, Marcus and I met in my place …

Marcus is one of the participants with a couple of years experience in SL. BTW Marcus directed my attention to an interesting post written on this first week’s impressions by participant, Susanne from Germany :-)

Both Heilyn and I are using this PBBL course to collect empirical data for our PhD-projects and we will meet later today to assess this first week as part of our own Action Learning cycles. We will post our reflections in Moodle, so that the participants are able to see how we actually try to apply the chosen research and teaching and learning strategy in our own projects encouraging the participants to comment on our thoughts. Furthermore Heilyn and I will rearrange the teaching and learning area to match the next weeks’ activities …

And I will for sure return with more posts on this as the course progresses …