Archive | Evaluation RSS for this section

High student satisfaction in SL

On June 16th, 22 students graduated from the Master’s Program on ICT & Learning (MIL) at Aalborg University, and this is where I’ve been running courses on SL for my PhD-project since 2007. As always, graduation day was an exciting day combining student anxiety and great relief and joy. After all the exams, there was a reception where the Masters received their diplomas, the daily manager of MIL, Ulla Konnerup and the Dean of Humanities, Lone Dirckinck-Holmfeld spoke about the students’ achievements and their new roles as “ambassadors of ICT & Learning”. Finally, we finished off the day with a wonderful dinner/dance at the Utzon Center, downtown Aalborg.


22 very happy Masters of ICT & Learning 2011.

As something new, the steering committee behind the MIL Program had decided to award the Program’s “Teacher of the Year”. All courses/modules are anonymously evaluated by the students, and based on these evaluations; I was fortunate to receive this award :-)


1 happy Teacher of the Year 2011 and 1 happy Dean (right).

Ironically, I’ve not (yet) seen these particular students evaluations, however, based on the evaluations the students and I did as part of the SL course, I do have a few ideas as to why the students find teaching and learning in SL so satisfying. To understand this a little background information is necessary. In my PhD-project, I’ve conducted 4 research cycles, spanning from 2007-2010. Each cycle consisted of designing, implementing, and evaluating a 6-8 week online course on ICT and instructional design based in SL and a conventional 2D VLE. From a theoretical point of view, I’ve been inspired especially by Wenger’s (1998) social theory on learning as participation in Communities of Practice (CoP), Schroeder’s (2011) ideas on presence and co-presence, and Bolter & Grusin’s (1999) concept of remediation. From a methodological point of view, I’ve been inspired by Insider Action Research (Coghlan, 2007), and ethnographical methods such as longitude participant-observation (Boellstorff, 2008). 53 adult MIL-students (majority are educators) in total have participated in my study. The table below provides a brief overview of the research cycles.


PhD-overview – July 2011.

Based on my data, I’ve been able to identify 3 analytical units that will inform the answering of my research question; namely what it means to learn via 1) a new, virtual environment, via 2) a new, virtual body, and finally via 3) new, virtual activities. The picture below shows the 3 units and the related topics that emerged in all four research cycles.


3 analytical units; virtual environment, virtual body, and virtual activities.

In this short post, I will not go into details with the units, but my findings show that being remediated as avatars in a new, virtual environment where it is possible to participate in a variety of new virtual activities greatly influenced the students’ perceptions of presence and co-presence, and from a Distance Education perspective this is one of the most valuable contributions SL has to offer. Conveying a sense of “being there together” as Schroeder puts it, is essential in Distance Education, not only in terms of student satisfaction, but also in terms of learning outcome. Further, SL also provides the participants with unique opportunities of “doing things together”, and as such it is possible to attribute some of the students’ satisfaction to SL’s affordances. I would, however, like to stress that relevant affordances do not necessarily guaranty satisfaction, and though this holds true for all technology, especially in a complex system like SL, the instructional design becomes pertinent. Basically, my PhD-work has been about designing for optimal learning via SL, and in this respect, I’ve found great inspiration in Wenger’s four dimensions of learning;

  • Learning as a process of experiencing – outcome: changed meaning
  • Learning as a process of becoming – outcome: changed identity
  • Learning as a process of belonging – outcome: changed community
  • Learning as a process of doing – outcome: changed practice
Even though, I’ve not designed exclusively for the creation of a community of practice in SL, e.g. by solely using Wenger’s proposed design principles*, the ideas of the theory are part of my, and the MIL Program’s general pedagogical foundation, and I do believe that SL is a medium that offers very good opportunities for creation of communities of practice, both in educational and other settings. Looking at my data, I’ve found a distinct connection between elements from CoP-theory and presence/co-presence as shown in the figure below.


Connected elements of presence and CoP-theory in 3D-remediated learning.

In short, the figure shows how the sense of presence facilitates the creation of meaning and identity, while the sense of co-presence facilitates the creation of community and practice. In practice, the elements overlap, and it is in fact the oscillation between the elements, which constitutes the dynamics of SL as teaching and learning environment as seen from a CoP-perspective. Based on the findings from my study, I believe that the combination of a social pedagogical strategy and the use of a medium that affords a strong sense of presence/co-presence and which is rich in terms of co-creative possibilities, actually can promote student satisfaction. Evidently, this is a very brief description of my work … more details will follow in my forthcoming dissertation that is due in September.

/Mariis

*) For an excellent example of integrating Wenger’s principles and ideas in design for teacher development in an online community, please have a look at my (now former) colleague Dr. Mayela Coto’s PhD-work.

Connecting theory and practice through SL designed case-based simulations

On Wednesday, January 19th Milano Bluewood, Savage Seaside, and Vibs Bluebird from the MIL-Health group did their presentation and analysis of SL as teaching and learning environment based on a Danish in-world project called CaseConnexion (Danish description). This is a project aimed at Nurse Education, and Milano is actively involved in the project. A pilot study was run in the fall of 2009, and it was the experiences from this study, the group had decided to present and discuss, and so naturally we started off on the island CaseConnexion.


Milano, Vibs, and Savage presenting in CaseConnexion’s sandbox

The main purpose of starting the in-world CaseConnexion project was to investigate its potentials in bridging/connecting some of the theoretical and clinical elements of Nurse Education. A survey, with answers from 85 students and 7 teachers, pointed to a variety of issues that should be addressed in future use of CaseConnexion (thoroughly described in this Danish evaluation report); steep learning curve, lack of identification with the avatar, fear of looking stupid in front of class-mates, technical difficulties and “waste” of valuable time spent on learning the system rather than the subject matter as a consequence of the former issues.

These were all issues that we also had encountered and discussed during this MIL-course, and so the group wanted our feedback on ways to solve these problems. As the group explained, the survey fortunately also pointed to issues on the plus side such as variation in teaching and learning methods, and the ability to practice skills without actually hurting/harming real humans through case-based role-play. As part of the set-up for their presentation, the group provided us with wheel chairs, which contributed to set the scene. It was interesting to experience and observe how we all felt the limitations of this type of handicap, and this led to discussions on how design can enhance/limit the sense of embodiment. We were also asked to watch this brief Danish video abut the project.

Next part of the presentation was to drive over to the hospital …


Racing towards the hospital … except that racing while taking pictures is a no-go in SL!


First quiz at the entrance; pick the right uniform or you’ll not be allowed enter!


Reception area with free uniforms and bots to add to the RL-feel of this respectful remediation of a hospital environment.


Wash your hands upon entrance into patients’ rooms – otherwise simulations don’t work.


Staff’s conference room.


Exploring patient cases.

After exploring the hospital we went back to the sandbox, where Milano, Savage, and Vibs presented other, international examples of in-world Nurse Education – one of them being a project led by University of Auckland in collaboration with several other universities. After watching a few minutes of the video below, we were asked to go to the island Land of the Long White Cloud.

In the hospital on the Land of the Long White Cloud, we had the opportunity to investigate a different patient room also designed to be used in case-based teaching and learning.


An avatar  (as opposed to a bot as we saw on CaseConnexion) can climb into the bed and actively participate in the case.


The patient room from one side of the curtain …


… and the same patient room from the other side. A nice feature enabling the instructors and other audience to overlook the simulation. Just as in real life  – except for the ability to change POV.


Debriefing/feedback session

Having tried out different features of the patient room, the MIL-Health group invited us to the debriefing table, where we continued discussions on especially design for embodiment, sense of presence and collaboration. Milano, Savage, and Vibs had prepared a presentation, which in itself illustrated the connection between theory and practice, and it was refreshing to discuss some of the more difficult parts of designing for teaching and learning in 3D-avatar based worlds like SL.

The CaseConnexion project showed that designing for teaching and learning in SL is challenging and complex, and this is also my personal experience from having run 5 in-world courses. Milano, Savage, and Vibs’ session reminded me of László Moholy-Nagy by whom I’ve previously been inspired, and one of his famous quotes on design from Vision in Motion:

Design has many connotations. It is the organization of materials and processes in the most productive, economic way, in a harmonious balance of all elements necessary for a certain function. It is not a matter of façade, of mere external appearance; rather it is the essence of products and institutions, penetrating and comprehensive. Designing is a complex and intricate task. It is integration of technological, social and economic requirements, biological necessities, and the psychophysical effects of materials, shape, color, volume, and space: thinking in relationships.”

/Mariis

Evaluation, Literacy and Transliteracy

As previously described I’ll be using my Connective Model for ICT-remediated Didactic Design for the general analysis of my PhD data. A first step in this process is working with the 9 basic elements in the model and in this post I’ll focus on some preliminary work I’ve done on the element of Evaluation as depicted in the model below.

Given that definitions shape the way we think about and practice particular phenomena the very act of defining something should not be done without prudence. I’m quite confident that working with the different elements in the model will refine the way I’ll end up describing them, so for now I settle for working definitions and as such I’ve found inspiration in UNESCO’s definition of Evaluation and after modifying it so that it fits better into my study field of Didactic Design it would read as follows:

Evaluation means arriving at a value judgment on the basis of measures (qualitative or quantitative) considered to be valid and reliable, which compare the actual results of a Didactic Design with its anticipated results.

The element of Evaluation is to some degree connected to all the other elements, but according to the working definition there is a particular strong connection to the element of Goals, since this is where we can derive the criteria for evaluating the results. When dealing with Didactic Design there will always be a least two major perspectives from which we can look upon certain elements, namely the teaching perspective and the learning perspective, and in what follows I’ll present some initial reflections on the particular part of evaluation that concerns the evaluation of learning outcome/results. To do so I want to dwell a bit on the concept of Literacy, which I consider to be vital when discussing the purpose and goals of especially formal education.

In its most narrow sense literacy refers to the ability to read and write, but used as a more general concept literacy refers to being knowledgeable or educated within a particular field. In an interesting UNESCO report on the plurality of literacy and the concept’s connection to the right to education as stated in article 26 of  The Universal Declaration of Human Rights of 1948, the evolving notion of literacy is discussed and defined: “Literacy is the ability to identify, understand, interpret, create, communicate and compute, using printed and written materials associated with varying contexts. Literacy involves a continuum of learning in enabling individuals to achieve their goals, to develop their knowledge and potential, and to participate fully in their community and wider society.” (UNESCO.2004:13). Above all the plurality of literacy refers to the many fields in which literacy can be employed – specific literacies denominated by prefixes such as information, computer or media to name a few.

In my research on literacy I recently came across a very interesting article in First Monday that explores the concept of Transliteracy, which the authors Thomas; Joseph; Laccetti; Manson; Mills; Perril & Pullinger. 2007 define like this:

Transliteracy is the ability to read, write and interact across a range of platforms, tools and media from signing and orality through handwriting, print, TV, radio and film, to digital social networks. (Thomas et al. 2007)

What interests me the most is actually not the definition, but rather the idea and purpose of developing some sort of meta-literacy concept. According to the authors Transliteracy can be characterized as:

  • a possible unifying perspective on what it means to be literate in the twenty-first century
  • an extension of transliteration that also includes the increasingly wide range of communication platforms and tools at our disposal
  • a concept that calls for a change of perspective away from the battles over print versus digital, and a move instead towards a unifying ecology not just of media, but of all literacies relevant to reading, writing, interaction and culture, both past and present
  • a concept that doesn’t replace but contains both media and digital literacy
  • a possible literacy for (media) convergence
  • a concept of not just computer–based materials, but about all communication types across time and culture
  • a concept that insists on a lateral approach to history, context and culture, an interest in lived experience and a focus on interpretation via practice and production
  • an inclusive concept which bridges and connects past, present and, hopefully, future modalities
  • a concept that pays attention to the whole range of modes and to the synergies between them to produce a sense of a ‘transliterate lifeworld’ in constant process
  • both a concept and a practice productively situated in a liminal space between being a new cognitive tool and the recovery of an old one
  • a concept that deliberately refuses to presuppose any kind of offline/online divide
  • the kind of literacy we require to be able to simultaneously attend to multiple media and modes of communication as well as the kind of literacy we use to apply the literacies of one mode or medium to another one

Based on these characteristics I would interpret Transliteracy as a meta-literacy and I do find the characteristics both relevant and much needed in trying to define some sort of unifying literacy. The authors describe their work with the concept as a work in progress and “a good example of open source thinking between diverse collaborators” and they encourage further discussion and development of the concept. First author, Sue Thomas and her co-writers are all involved in the Production and Research in Transliteracy (PART) Group at the Institute of Creative Technologies (IOCT) at De Montfort University, UK and they’ll be hosting a conference on Transliteracy on February 9th 2010, which can be followed via several social media. In the video lecture below Sue Thomas explains the concept of Transliteracy based on the above mentioned article:

Returning to my interest in Literacy and especially new forms of literacy, I believe that there is a strong need to consider and develop new ways/methods of evaluation of learning outcome – not least when the learning processes and products have been facilitated by ICT-remediated Didactic Design. In this process of developing new evaluation methods, I think concepts like Transliteracy can prove quite valuable in giving indications of what criteria to focus on. Especially in Academia we seem to be stuck in using evaluation criteria and methods based on traditional literacy giving primacy to old media and modalities. Quoting Yancey. 2004: 90 “we use the frameworks and processes of one medium to assign value and to interpret work in a different medium”, which obviously is not the most appropriate way of accommodating the use of multiple and/or new media. In my third research cycle in the MIL case, I experimented with both criteria and methods of evaluation, and the results from this experiment will form the basis of a forthcoming post on evaluation of new media productions/compositions…

/Mariis