Andrea Resmini and co-organizers of the upcoming workshop on Architectures of Meaning (part of the Pervasive Computing conference at Newcastle University in the UK) asked me to participate this year. I’m not able to be there in person, unfortunately, but plan to join remotely. What follows is the “paper” I’m presenting. It’s not a fully fledged academic piece of writing — more like a practitioner-theorist missive.

I’m sharing it here because others may be curious, and it’s also the best summary I’ve done to date of the ideas in the book I’m writing on IA and designing context.

This is a straight dump from MS Word (with a few tweaks). Caveat emptor.

Information Architecture and the Composition of Context

Andrew Hinton

Final Draft for Architectures of Meaning Workshop

June 18, 2012

Introduction

We lack fully articulated models for context, yet information architecture is especially significant in how context is created, changed or communicated in digital-based information environments. This thesis proposes some principles, models and foundational theories for the beginnings of a framework of context and proposes composition as a rubric for tying these ideas together into IA practice.

The thesis follows a line of reasoning thus:

Context is constructed.

There’s a deep and wide intellectual history around the topic of context. Suffice it to say that there are many layers and threads in the ongoing conversation among experts on the subject. Even though all those threads don’t agree on every point, they add up to some generally accepted ideas, such as:

• Context is both internal and external. Our minds and bodies determine and influence how we perceive reality, and that internal experience is affected by external objects and interactions. Both affect one another to the point where the distinction between “inner” and “outer” is almost entirely academic.
• Context has both stable and fluid characteristics. Certainly there are some elements of our lives that are stable enough to be considered “persistent.” But our interactions with (and understanding of) those elements still can make them mean something very different to us from moment to moment. Context exists along an undulating spectrum between those poles.
• Context is social. Our experience of context emerges from a cognitive history as social beings, with mental models, languages, customs — really pretty much everything — originating from our interactions with others of our kind.

Context is not so simple as “object A is in surrounding circumstance X” — the roles are interchangeable and interdependent. This is why context is so hard to get our hands around as a topic.

(In particular, I’m leaning on the work of Paul Dourish, Bonnie Nardi, Jean Lave, Marcia Bates and Lucy Suchman.)

Context is about understanding.

This phenomenological & post-modern frame for context necessarily complicates the topic — not to point out these complexities would keep us from getting at a real comprehension of how context works.

Still, it can be helpful to have a simple model to use as a compass in this Escher-like landscape.  Hence, the following:

Context is conventionally defined as the interplay between several elements:

• Situation: the circumstances that comprise the setting (place, time, surroundings, actions, etc.). The concept of “place” figures very heavily here.
• Subject (Event/Person/Statement/Idea): the thing that is in the situation, and that is the subject of the attempted understanding.
• Understanding: an apprehension of the true nature of the subject, through awareness and/or comprehension of the surrounding situation.
• Agent: the individual who is trying to understand the subject and situation (this element is implied in most definitions, rather than called out explicitly).

Context, then, is principally about understanding. There is no need for discussion of context unless someone (agent) is trying to understand a subject in a given situation. That is, context does not exist out in the world as a thing in itself. It emerges from the act of seeking to understand.

This also forms a useful, simple model for talking about context and parsing the elements in a given scenario. However, it gets more complicated due to the ideas, mentioned above, about how context is constructed. Just a few of the wrinkles that come to light:

• There can be multiple subjects, even if we understand them by focusing on (or foregrounding) one at a time.
• The subject is also always part of the situation, and any of the circumstances could easily be one or more subjects.
• In fact, in order to understand the situation, it has to be focused on as a subject in its own right.
• All of these elements affect one another.
• Importantly, the subject may be the agent. And there can be multiple agents, where another observer-agent may be able to understand the situation better than the subject-agent, because the subject-agent “can’t see the forest for the trees.” In design for a “user” this is an especially important point, because the user is both agent and subject — a person trying to understand and even control his or her own context.

As you can see, what looks like a simple grammar of what makes context can actually expose a lot of complexity. But this simple model of elements helps us at least start to have a framework for picking apart scenarios to figure out who is perceiving what, which elements are affecting others, and where understanding is and isn’t happening.

In order to unravel this massive tapestry, we have to grab a thread; a good one to grab is what we mean by “understanding.”

And that means we have to understand cognition, which is the engine we use for understanding much of anything.

Cognition is embodied and extended.

The embodied mind thesis is not fully settled science, and there are many varying threads and contentions in that body of work. However, the basic ideas I’ve with which I’ve aligned my thinking contend that the evolutionary history of all cognition is rooted in bodily, sensorimotor experience, and that current human cognition still relies on the body and the environment.

That means cognition is not “in the mind” and separate from “the body” and “the world” – cognition is the result of the interplay of all those dimensions. In the act of perception, cognition doesn’t make meaningful distinctions between these dimensions.

This does not mean that all cognition for the contemporary human is dependent on “on-line” sensorimotor activity. In the language of Andy Clark at University of Edinburgh, cognition is “loopy” – it loops within and without the body, in a hybrid spectrum of cognitive methods, depending on the needs in the moment.

For example: I may count with my fingers or count with sticks; or I may count with more recent abstractions like numbers and mathematical operators, whether written down or typed into a device, and I may simultaneously be doing parts of that work “in my head.” I hardly pay attention to which methods I’m using and in what combination; I just use them to get to the answer I need.

Cognition depends on Perception.

At the core of my understanding of cognition is a model of perception developed by American psychological researcher and theorist James J. Gibson. Gibson’s theory starts with vision, but Gibson meant that it would be applied to any mode of perception.

Gibson on Perception: To Gibson, perception is “ecological.” He starts with what the environment contains (objects, shapes, material properties of gas, solid, liquid, etc.) and then explores how light interacts with those physical properties to create information that can be perceived by visually-enabled organisms. Gibson’s thesis is that the brain does not have to process all the visual information real-time in order for the organism to behave, move, and survive in the environment, because the organism’s physical capabilities already respond appropriately to most of the environmental information.

Gibson on Information: To Gibson, information is the perceptual stimulus created by the interplay of energy in the environment and the physical properties (shape, material, etc.) of the environment. In the case of vision: the way light interacts with the physical environment creates structural information that shows difference between boundaries and connection points. Water reflects light differently from land, direct light sources create visual information differently from reflected light sources, and so on.  It is only “information” because it is being perceived by an organism that can make use of it (i.e. something has to be ‘informed’).

Gibson on Affordance: Probably, Gibson’s best known idea among user-centered-design circles is that of affordance. For Gibson, affordances are action-relevant properties of the environment. The way light interacts with air is different from its interaction with water, so an organism learns the physical properties of those different experiences of light and moves through them or around them accordingly.  The visual information from the interaction of light and land affords the actions of walking and standing. The information from air + light affords “moving through” and breathing. The light reflected from a fallen branch shows it to be roughly the right size and shape to be used as a tool for digging.

I dwell on Gibson’s ideas here because they form a rigorously reasoned foundation for understanding how cognition depends heavily on physical perception, which therefore is also a core concern for context.

Information: Three Modes

A small detour here to sketch out what I think of as three modes of information:

1. Ecological (The Gibson mode): information as explained above — the interplay of energy (light, sound, etc.) and environmental properties (shape, material, etc.) and the perceiving organism (in our case, a human “agent”). The “meaning” in this sort of information is rooted in affordance, for basic needs like locomotion and physical survival as well as higher-order activities like manipulating tools.
2. Linguistic: the perceived environment humans have collectively invented first through speech and much later through writing. The “meaning” in this mode is semantic rather than based in physical-properties-plus-energy (although written language makes use of that sort of information as well, as way to communicate the symbolic/semantic stuff, but getting into that is more than we have time for here). We will look at this more below, when considering language as a human-made environment.
3. Digital: information in the Claude Shannon sense, as essentially difference: one or zero, and anything that can be built up from that logical switch.  Shannon’s breakthrough was seeing that to get machines to work effectively and accurately with human-made information, we had to ignore “meaning” entirely and focus on encoding and transmitting it in such a way that semantics and human-perceived affordance are no longer involved, at least until some output needs to be human-readable, in which case it gets translated (decoded).

A big challenge with context in digital-based environments is that we don’t make an explicit, careful distinction between these modes, and the meaning-free mode ends up leaking into the others.

Language is a human-generated environment.

An environment, in the Gibson sense, is the collection of stuff in the world with which we interact, and which generates information through interplay with energy, which we perceive as affordance.

In that sense of environment, language can be seen as a human-generated environment. But instead of speaking words that make trees, water, land and horizon appear in front of us, the words instead create cognitive simulations – sort of like shared hallucinations – that we inhabit together simultaneously with the physical environment.

Language is the human species’ way of creating its own environment of information, with which we label the world around us, plan together about how to perform actions in that world, or anything else that we find valuable. Language both changes how we perceive the physical world and also creates new, non-physical worlds (concepts, ideas, abstractions) that we inhabit together in conversation, discussion and stories.

Language is embodied.

It’s no accident that the initial exploration of embodied cognition came from linguistics, in the work of Lakoff and Johnson (“Metaphors We Live By”). A large part of how cognition works involves the use of signs and symbols that represent or arise from abstractions of the body, and the body’s sensorimotor experience of and interactions with the world.

There is compelling evidence that language has been with our species longer than conventionally assumed, and spoken language “co-evolved” with us long enough to act as a powerful force in our natural selection. That is, language acted as a shaping constraint in the survivability and adaptation of homo sapiens. (Leaning heavily here on the work of anthropologist Terrence Deacon).

Regardless of whether language is millions or just many tens of thousands of years old, it is not just an abstracted layer of non-physical signification, tacked onto human life as a mere convenience. Language is a collectively emergent technology we’ve come to rely upon for survival over a long enough period that it’s essential to the nature of being human; it’s a metaphysical, shared organ of meaning, identity, and understanding. A human being with no language would be, compared to other humans, cognitively “broken.”

We’ve been coexisting in the language environment in just as profound a way as we coexist in the physical one.

Therefore …

Language is essential to contextual experience.

So, to understand context, we have to understand cognition; in particular we have to understand how cognition is “embodied.”

And in understanding cognition, we have to understand how language works as an organ of individual & communal cognition & understanding.

We then see that language doubles back into context as a core element: it nudges, informs, shapes and even creates contextual experience.

Since language first emerged among humans, it has been a sort of technology for constructing our shared reality. We inhabit a shared linguistic construct that literally changes what surroundings, objects and experience mean to us; and that construct also creates new surroundings — situations, conditions, contexts — that are just as immersive and meaningful to our lives as any mountain, stream, building or city square. In fact, it’s language that gives even those physical entities so much shared cultural significance.

Digital information brings new complications to Ecological & Linguistic information coherence and comprehension. 

Since human beings first used any sort of language, we’ve been inhabiting a shared “information dimension” of sorts. But the Internet has made what was tacit, analog and “meta” into something explicit, digital and “actual.”

This dimension unmoors context from many of the embodied, physical referents that our brains and bodies evolved to take for granted. This brings new complications to how context is experienced. As mentioned above, we tend to create things using digital information as raw material, without paying attention to the full needs of how we comprehend ecological & linguistic information.

Information architecture arose as a way to compose habitable structures in this new dimension.

Information architecture is a powerful framework for designing context; the challenge of designing context with information architecture can better be understood and performed if framed as an act of composition. Composition is a relevant frame in several ways:

• We compose context in the sense of two dimensional art (e.g. “a well-composed photograph”): composition in art is about the arrangement of elements within the frame of the work of art, foreground and background, juxtaposition and relevance, what is included and not included — all these affect the user contextually as they affect the viewer of an artwork.
• We compose context in the sense of architectural composition: the arrangement of spatial cues and physical boundaries to create specially recognized places, that afford certain actions over others. A hallway affords travel from one place to another; a foyer affords entry, meeting guests, getting settled; etc. The space is understood in an especially embodied way. Similarly for context, which is also about what activities are afforded over others, public vs private, etc.
• We compose context in the sense of creating meaning with language (e.g. “composing a sentence): composition of language involves syntax, semantics and semiotics to arouse messages and meanings in other people; similarly, context is highly dependent on language; for example, labels can completely change the experienced nature of two otherwise identical objects or places.

It turns out that for the information dimension, language has to do much of the work of all these forms of composition, not just the “writing sentences” variety; and an understanding of syntax, semantics and semiotics is similar to having an understanding of physics for architecture, or an understanding of perspective in visual art.

Composition of context, then, depends upon the careful and fully aware creation and manipulation of all three modes of information. As we continue to mature the practice of information architecture, we need to improve our understanding how the materials we use function as information for cognitive affordance, and how the structure of these affordances creates and changes how the agent (user) comprehends context: the understanding of the relationships between subjects an situations.

Note: This is based on work in progress for a book on the design of context. I am not a full-time academic researcher and writer, but a theoretically-minded practitioner looking to ground these ideas in responsible research. Any critique, suggested sources or other feedback is quite welcome.