Interpreting hybrid ecology in augmented reality

A draft of my major ideas.

Background theory

Concepts: embodiment, neural representations of sensory-motor actions, action potentialities, affordances, anticipated goals, action, mediation, coupling, tool, hybrid ecology

The basic ideas in Activity theory (Leontjev, 1975) relate ‘people who want to reach some goals’ with their ‘mediating tools for realising activities’ necessary to ‘reach the goal’. Mediating tools can be cognitive (eg. language, gestures, content of narrative artifacts or pictures etc.) or material artifacts (tools, objects etc.).

Discoveries in cognitive and neuroscience about the functioning of mirror-neuron systems (Gallese et al., 1996), claim, that cognition is embodied through grounding knowledge directly in sensory-motor experiences without the mediation of symbolic representations (Pecher & Zwaan, 2005).

From observation of others and the environment (Rizzolatti et al., 2001), from listening narratives (Rizzolatti & Arbib, 1998; Iaccoboni, 2005) or from reading narratives (Scorolli & Borghi, 2007) and looking everyday images of objects or works of art (Gallese & Freedberg, 2007) we perceptually activate certain multi-modal action-potentialites to mediate our purposeful and goal-directed actions (see Gallese & Lakoff, 2005). These embodied dimensions are activations of neural representations located in sensory-motor areas in brain.

Hommel (2003) assumes that action control to all behavioral acts is ecologically delegated to the environment – when planning actions in terms of anticipated goals, the sensory-motor assemblies needed to reach the goal are simultaneously selectively activated in the environment, and bind together into a coherent whole that serves as an action-plan, facilitating the execution of the goal-directed actions through the interaction between the environment and its embodied sensory-motor activations.

My understanding of affordances sees them as the emergent constraints in the activity system dependent of various system components and interactions. I interpret them as action potentialities. However, affordances can also be potentially embedded into mediating tools due to cultural use of language or due to culturally defined activity potentials objectified in artifacts and tools.

Our hybrid nature

Hybrid ecology binds together into one inseparable whole the subjects aiming to do something, the surrounding environment with its objects, and the mediating tools of their action. In this view all external entities of the person can be considered as an environment (eg. living and nonliving things and their causal and structural interrelations).

In this symbiosis we can see that in any activity there is no clear and distinct border between subject and the environment. The mediation between the personal (internal) and environment (external) is a dynamic two way process of embodiment of environment’s dimensions, and expressing the embodied in the environment.

Any action can be taken only on the basis of embodied dimensions of the environment.

Environment becomes a place for action if we accommodate it with our tools, if we embed part of our embodied sensory-motor experiences to the external environment and if we reuse it constantly matching our internal embodied sensory-motor action potentialities and action potentialities that couple with it around us.

Extending and accommodating ourselves to the environment

To be purposely navigating and acting in the environment, persons need to embody part of the environment in order to toolisize it, and start using it for their action. This is not necessarily a conscious and goal-driven process. Also the environment with its culturally defined action potentialities in relation to persons’ previously embodied action-potentials and emotions can evoke certain goals for action. In such case certain aspects of the environment may be extended above the others to the range of the persons’ perception.

People can also toolisize the environment by externalizing their internal imaginations, emotions and sensory-motor action patterns to the environment as objects for use for themselves and the others.In a way it can be seen as making the environment as part of their symbiotic being where they can find and use the common sensory-motor action potentialities over and over again.

Toolization is not necessarily merely a process of symbolizing meanings. If toolization of the environment is done by language, or making narrative or visual artifacts it is done by using symbols as carriers of sensory-motor action potentialities. But toolization is also direct activity carried out in the environment, when the objects of the environment and some perspectives of the environment are actualized to express and carry out goals-directed action plan, which has sensory-motor neural correlates.

Creating new tools

Our usual acting in the personally accommodated environment is based on the preferred perception of affordances that match with our sensory-motor action correlates of previously embodied emotions and actions. Basically we repeat ourselves to feel safe in the environment. We actualize the affordances that we define for ourselves as the members of certain culture, those that enable us to take active part in this culture.

However, some of the affordances, that are very distinct from the usual affordances, may also be perceived and may start intruding the activation of embodied sensory-motor paths. These affordances can be viewed as the noise to be ignored or noise to be considered.

Some of the noise we try to ignore, telling to ourselves that it is the similar affordance to some other usual affordance. Similar affordances are coupled with the previously anticipated sensory-motor action potentialities.

Some of the noise we perceive as analogical affordances. Analogy is accepting the difference in nature but making the relation on the basis of some features. In this case partial coupling of anticipated affordances with the actualized affordances happens, which may activate only part of the previously embodied sensory-motor neural correlates or even some new ones.

Looking for the noise to activate different sensory-motor experiences is an important aspect of our creativity. Looking for the noise may lead us toolizising the environment for us differently. We basically create new tools to accommodate ourselves into the environment with our usual goal-directed plans. Or, the creation of these new embodiments from the environment would lead us to the totally new set of goal-directed action plans. We shift from one set of anticipated affordances we are looking for in the environment to another set.

Tools we create from tools left to the environment

Lev Manovitch has pointed out to some trends of tool designs in modern environment.

Manovitch deals with ‘interfaces’ enclused to ‘objects’. There is a new trend in design: disappearance of technological objects as such, which become integrated into our spaces. In such cases the previous culturally accommodated tool or artifact and the previously culturally accommodated technological interface may together start triggering different potential affordances for action.

The action potentials technology offers through ‘interfaces’, and the action potentials that the objects had in another previous culture, can be contraversial and may cause certain “battles” between what action to carry out, evoking the necessity for internal grounding between the different sets of affordances. This would result in not using these new devices most effectively as planned by the designers and visioners of new technology, because we would be triggered simultaneously by the affordances evoked by two different cultures.

Basically what we need, is immersion of cultures, becoming these new ‘immersive technology generations’ and we will no more distinguish the potential affordances the objects evoke as old and new activity potentials. Rather we will shift out internal immersed and augmented intentions into the environment activating combined affordances.

Do we need new designs to obtain the new culture or would the new culture emerge in using currently not ideally integrated technologically enriched objects and augmented reality? If so we will just start evoking different affordances in this environment and the design and immersion of old and new should not necessarily be melted to each other organically, we would do it by ourselves, with our perception and imagination.

In this case we can act in the current environment we are used to, but we can see, if we want so, in this environment also this virtual, technologically added space – we will get hybrid, augmented or ecologically defined environment for totally new activities, where we can go or not go depending our intentions.

Manovitch assumed that another trend in design is that tools are vanishing and becoming into seamless interfaces. It seems that in interaction with new technological and hybrid tools affordances of the activity-side become stronger and stronger perceived, while the affordances of the previously recognized tool or artifact sides, where interface is embedded, are decreasing in the perception of the user.

In the PhD thesis of Elza Dunkels “Bridging the Distance: Children’s strategies on the Internet” (2007) she conducted online interviews in chatroom to ask about how children perceive the Internet environment. She wrote: From 8000 words in the interviews kids actually mentioned the word computed 19 times, and only at 2 occasions this was initiated by the child. Communication and fun themes, rather the use of technology per se prevailed.Kids totally forgot that technology is there as a mediation tool of their action goals!

To interpret these results, it seems that the actualized affordances of Internet seemed not to depend of the objects, computers, technology with its functionalities and limitations, but instead these ‘information age immersed kids’ had real life and very warm and alive activity goals and thus they perceived in the technology environment other affordances than the grownups from earlier generations. Kids actualized for themselves the ‘communication and interaction affordances’ and did not consciously actualize the affordances of the technology as something that mediates their actions.

Hybrid beings interact

Interesting perspective emerges if the person, who externalized the embodied sensory-motor action potentialites (basically their tools as mediation devices) for themselves to the environment, is viewed from aside by another person. For the other person two versions of the environment may appear.

One is where he can see the other subject together with its mediation devices. The studies of embodied cognition indicate that it is highly likely to embody and directly activate the sensory-motor patterns they view the others doing in the environment – thus what is happening is the culturally defined selective embodiment of the action-potentialities (tools they make from the environment to use it)of the others. Evolutionary, this can be viewed as some kind of reuse or optimization strategy, getting the same result without using the energy what was initially used on creating the initial interrelation between the person and the natural environment.

Vyas and Dix (2007) suggest that sets of affordances exits at the level of person, group/community and culture, and that the interactions between these affordances may influence each other.

Second is if only these man-made objects (various cultural tools eg. texts, patterns, images, artcrafts etc.) and other type of traces of their activity are left to the natural environment, but their initial originators have left the scene. Now, it is the question of culture to couple the previously experienced sensory-motor action potentialities with those that are recognized in the environment. It is highly likely that this kind of embodiment and toolization of the environment for purposeful action is always partial or even completely transforming the initial action-potentialities of the tools.

The notion of meanings in activity-centered view to hybrid ecology needs to be elaborated. Meaning is the result of embodiment of environmental entities partially as sensory-motor action potentialities. Meaning-making is a more or less conscious search of action-potentialities in the environment that can be coupled with previously embodied ones in the process of making them mediating tools. What is meaning making for an individual in these situations when he/she can see only the environment toolisized by someone else?

First is, when does the person notice that the environment is actually toolisized – perhaps this is a culturally defied process of noticing certain affordances, coupling it with embodied affordances and taking action if there is a match of affordances. Noticing affordances and actualizing something from the environment as tools similarly as other people depends of whether they are part of this community and culture.

It can be assumed that cultures leave traces of activity potentials as patterns to the environment which can be actualized if the pattern is strong enough, frequent enough and matches with some of the anticipated affordances for the person. It is clear that if these patterns are left to the environment without perceived interrelation with their creators, they are harder to be actualized similar way by other people.

Hybrid ecology in augmented environments

The previous aspects of hybrid ecology are quite general and deal with the nature of individuals acting in an environment.
What happens if the environment too becomes hybrid?

Recent trends in the Web development have caused the immersion of borders between the real and virtual spaces, giving rise into the new potential learning environment. New kind of social software eg. blogs, wikis, social bookmarking services, social artifact repositories enable user integration into democratic content-development and publishing. Mashup technologies allow publishers to syndicate their data into machine-readable RSS feeds to which readers can selectively subscribe with free social software. Geotagging systems make it possible to create locative content by mobile devices, situated both in real and virtual environment (Tuters & Varnelis, 2006). Locative content is digital media applied to real places, any kind of link to additional information set up in space together with the information that a specific place supplies, which is triggering real social interactions with a place and with technology (Tuters & Varnelis, 2006; Hanzl, 2007, Kaipainen & Pata, 2007).

This new learning environment – an augmented reality/virtuality – consists of distributed virtual spaces generated by social software tools, and of the real spaces and objects, in which locative content has been added with mobile devices. Augmented reality, the reality overlaid with virtual reality, and virtual reality, in which representations of the real world have been embedded and contextualised, is enabling interactions both in real and virtual spaces. Lonsing (2004) suggests that an augmented reality system generates a composite view in real time – a combination of a real scene viewed by a user and a virtual scene generated by a computer, where the real scene is submerged with additional information in order to enhance the perception of the user.

Rich layers of embodied knowledge and practice in the real spaces, and authentic context triggering activities and knowledge-building in virtual spaces, makes augmented space into a potential learning environment with new challenges for the learners. This new learning medium is a distributed activity space in which learners meet other learners, knowledge artifacts and practices.

New augmented learning environment provides novel tools and triggers new types of activity patterns in this distributed space. Possible learning patterns involve both actions in real and virtual spaces – thus we need to view this space as one whole.

Several community-activities with new technologies can be built upon the relationships between real spaces/objects, people and meanings:
Space with dynamically embedded meanings (eg. spoken narratives, movement) entails action potentials. The embodiment of these action potentials is a process of coupling affordances evoked by internal imagination and goals in one hand, and the perceived and culturally predefined external affordances from the environment, on the other hand. As a result, this space becomes into a place for new interpretors and starts triggering activities.

Question is of course when can persons in such places perceive the patterns of activities, when can they perceive affordances of the certain culture to interact with this culture?
(eg. How much geotagged content at places/virtual content in blogs, social repositories there needs to be, that it would reveal activity potentials for triggering certain community activities. What happens if the activity potentials coupled with different anticipated activity potentials, how do such creative threads and activity derivations emerge and influence the augmented reality activity potentials?)

In which areas of the augmented reality (real or virtual) can the activity potentials be actualized, will the proceeding activity take place in the same dimensionality or can it cross the imaginary border of these spacial dimensions of the augmented reality?
(eg. i can add geotagged content in real place, but later it starts my virtual activities in virtual space, that in turn can be traced as activity patterns from the real place)