Gallery Overview
Research Agenda
Our work can be roughly divided into two long-term research agendas.
The first is developing new interactive narrative platforms, models, genres, and works. Our interactive narrative research and development, more precisely called "multimedia interactive and generative imaginative discourse," practices centralize the following three concerns:
(1) Story Construction: Cognitive Science-Based Approaches,
(2) Story Analysis: Culturally Diverse Models, and
(3) Story Use: Community Deployment of Expressive Technical Practices.
The second, executed under the umbrella of the "Advanced Identity Representation Project" investigates the construction of social identity in computational media as it arises cognitively, narratively, and through infrastructures for classification and navigating diverse community memberships.
The following computing research threads enable and inform our approach to computational narrative and identity and the related technologies we develop. A brief account of several research threads we pursue follows:
A) Subjective Computing Subjective meaning and computation are usually seen as completely separate issues. We develop technologies and theoretical tools that allow authors, programmers, and artists to (1) enable digital media authors/artists to "technically embed subjective meaning to within media," i.e. construct ontologies (formal descriptions of knowledge structures) as metadata for their media elements (graphics, animation, text, etc.), (2) generate meaningful text and multimedia compositions dynamically, and (3) blend multimedia structures to generate new content dynamically for use in interactive narratives and related works. These tools use the models of imagination from cognitive semantics to represent these "meanings" as formalized concepts and their composition and generation as blending and mapping operations. In this area, joint work with Ph.D. students has also investigated issues such as how intentionality and animacy are conveyed via computational media forms such as AI systems or interactive animation.
B) Cultural Computing Our research approach advocates the grounding of computing practice in diverse cultural traditions. We are open to serving the needs of communities with values other than those that are privileged in computer science and engineering currently (which are often driven by "Western," logocentric, and productivity-oriented modes of thought). For example, I have explored the notion of orature (oral literature) as it has been theorized and practiced across the African diaspora as a grounding for new technical practices implementing computational discourse generation and interaction. As another example, I have developed a generative multimedia discourse project informed by an account of the interplay between iconicity (e.g. in the Japanese language) and conceptual metaphor by Masako Hiraga and C.S. Peirce's semiotics.
C) Social Identity Computing Our research results in theory and technology to enable social identity models for deployment in social networking, computer supported cooperative work (CSCW), digital media arts and games, and educational technologies. These new models are dynamic, integrated across applications, sensitive to social context, empowering for diverse and underrepresented groups, and ethically engaged. Such broad applicability of the models developed are enabled by (1) methods that invoke the shared cognitive mechanisms for categorization that undergird human construal of socio-cultural identity classifications, and (2) computer science approaches to multimedia semantics that address algorithmic and data-structural reciprocities between two dominant models of digital self-representation: avatars/player characters (mediating proxies for action in digital environments) and user profiles (informational surrogates in digital applications). The models also apply to AI identity representations such as software agents and characters.
The GRIOT Platform
The GRIOT System
The GRIOT system was invented by Fox Harrell and it comprises technical support for implementing narrative and other forms of computational discourse with the following characteristics: generative content, semantics-based interaction, reconfigurable discourse structure, and strong cognitive and socio-cultural grounding. Strong cognitive and socio-cultural grounding here implies that meaning is considered to be contextual, dynamic, and embodied. The formalizations used derive from cognitive linguistics theories with such notions of meaning. Furthermore, the notion of narrative here is not biased toward one particular cultural model. Using semantically based media elements as a foundation, an author can implement a range of culturally specific or experimental narrative structures.
Figure 1: The GRIOT System Architecture
The following describes the GRIOT architecture as used in initial text-based experiments with narratively structured poetry. User input, in the form of keywords, is used to select the conceptual space network from a set of ontologies, called "theme domains," that each contain sets of axioms about a particular theme. These axioms consist of binary relations between sorted constants. This conceptual space network, called an "input diagram," consists of a generic space, two input spaces, and mappings from the generic space to each of the input spaces. The input diagram is passed as input to the ALLOY conceptual blending algorithm. ALLOY is the core component of GRIOT that is responsible for generating new content. An "output diagram," consisting of a blended conceptual space and morphisms from the input spaces to the blended space, is output by ALLOY. Concepts are combined according to principles that produce "optimal" blends. Typically this optimality results in "common sense" blends, but for particular poetic effects different, "dis-optimal" criteria can be utilized. "Phrase templates," granular fragments of poetry organized by narrative clause type, are combined with the output of ALLOY (converted to natural language by mappings called "grammar morphisms") to result in poems that differ not only in how the phrases are selected and configured, but in the meaning being expressed by the blended concepts. The phrases are said to be "instantiated" when they are combined with the natural language representations of the blends by replacing "wildcards" in the text. These wildcards are tokens representing where generated output can be incorporated, they also contain variables that specify how they are to be replaced, e.g. constraining the choice of theme domains, or selecting the lexical form to be mapped to by the grammar morphism. These templates are selected according to an automaton called a "Narrative Structure Machine," which also structures the reading of user input.
The GRIOT system can also dynamically compose modular graphical elements. This functionality is split between a server (implemented in LISP) to handle semantics and graphical discourse structuring and a client (implemented in Processing) to handle graphics processing and user input as depicted in the Figure below.
Figure 2: The Generative GRIOT Multimedia Semantics System
The server consists of the following components:
- 1) Semantic annotation
- 2) Discourse structuring rules
- 3) Priority morphism (matching) algorithm
- 4) Image layout data structure
This extends elements of the GRIOT system with an algorithm to judge fitness (matching) of images to be composed and a layout structure for multimedia images in addition to the standard discourse structuring rules. The client consists of the following components:
- 5) Graphical assets
- 6) Graphical layout rules
- 7) Duplicate image layout data structure
- 8) GUI input and output protocols
Graphical assets are actual image data files. These assets are described on the server side using semantic annotation (the relationship between the metadata and image data files is indicated by a dotted line in the Multimedia Semantics Figure above). This annotation, using XML that is parsed and processed to produce LISP data structures, describes the visual, structural, and conceptual content of images.
Loss, Undersea
Project Overview
Loss, Undersea is an interactive narrative/multimedia semantics project by Fox Harrell in which a character moving through a standard workday encounters a world submerging into the depths -- a double-scope story of banal life blended with a fantastic Atlantean metaphor. As a user selects emotion-driven actions for the character to perform, the character transforms -- sea creature extensions protrude and calcify around him -- and poetic text narrating his loss of humanity and the human world undersea ensues.
Selected Relevant References
D. Fox Harrell. "Walking Blues Changes Undersea: Imaginative Narrative in Interactive Poetry Generation with the GRIOT System," in Proceeding of the AAAI 2006 Workshop in Computational Aesthetics: Artificial Intelligence Approaches to Happiness and Beauty, AAAI Press, 2006. [pdf]
Example Video (music: Sun Ra, "India")
Demo Poster
Generative Visual Renku
Project Overview
The Generative Visual Renku project presents a new form of concrete polymorphic poetry inspired by Japanese renku poetry, iconicity of Chinese character forms, and generative models from contemporary art. Calligraphic iconic illustrations are composed by the system with both visual and conceptual constrains in response to user actions into a fanciful topography articulating the nuanced interplay between organic (natural or hand-created) and modular (mass-produced or consumerist) artifacts that saturate our lives.
Artists' Statement
Since the industrial age, modularity has revolutionized our everyday lives. For the sake of efficiency and optimization, things and activities were shaped into mass-produced interchangeable units, including our furniture, our dwelling places, our commuting, our consumptions, our entertainments, and our identities. In consumerist societies, modularity always lies at the center, whereas the complements are just scattered peripheries. Life is a journey back and forth between clustered majorities and isolated minorities.
Selected Relevant References
Kenny K. N. Chow and D. Fox Harrell, “Generative Visual Renku: Linked Poetry Generation with the GRIOT System,” Visionary Landscapes: Electronic Literature Organization Conference 2008, Vancouver, WA, 2008.
Example Video
Demo Poster
Memory, Reverie Machine
Project Overview
Memory, Reverie Machine (MRM) is a text-based computational narrative system that is informed by stream of consciousness literature, cognitive linguistic theory of blending and analogy, artificial intelligence research and conventions of Interactive Fiction (IF). The system generates stories in which the main character shifts dynamically along a scale between a user-controlled avatar with low intentionality and an autonomous non-player character with high intentionality. Built on Harrell's GRIOT system that algorithmically controls the semantic hooks for interpreting system through blending, MRM adds a new degree of dynamic discourse structuring in order to create the appearance of dynamic intentionality and system agency.
Selected Relevant References
D. Fox Harrell and Jichen Zhu. "Agency Play: Dimensions of Agency for Interactive Narrative Design," in Proceeding of the AAAI 2009 Spring Symposium on Narrative Intelligence II, AAAI Press, 2009.
Jichen Zhu and D. Fox Harrell. "Narrating Artificial Daydreams, Memories, Reveries: Toward Scalable Intentionality in Expressive Artificial Intelligence Practice," Visionary Landscapes: Electronic Literature Organization Conference 2008, Vancouver, WA, 2008.
Jichen Zhu and D. Fox Harrell. "Daydreaming with Intention: Scalable Blending-Based Imagining and Agency in Generative Interactive Narrative," in Proceeding of the AAAI 2008 Spring Symposium on Creative Intelligent Systems, AAAI Press, 2008. [pdf]
The Griot Sings Haibun
Project Overview
A griot is a revered storyteller in many parts of the African Diaspora. "The Griot Sings Haibun,"is an improvised performance of music, poetry, image, and computation. Live musicians fuel collective improvisation with Harell's GRIOT, a cybernetic system on which a human "plays" an ever-changing polypoem, an interactive multimedia polymorphic narrative poem. The core of GRIOT is the novel Alloy algorithm to generate new concepts and metaphors by blending, based on recent research in cognitive linguistics, computer science, and semiotics. A polypoem is not the output of a single GRIOT execution, but the space of possible poems and/or the code that makes them. In this work, GRIOT generates (neo)haibun: combined narrative prose and haiku-like poetry of everyday experience, influenced by Basho, and the traditions of beat poetry and African call-and-response; tonight our collective griot sings qualia, the qualitative feel of this human life world.
Examples
Sample GRIOT Generated Text
Sample Griot Sings Haibun Code
Screenshots
Human generated neo-haibun
The Girl With Skin of Haints and Seraphs
Project Overview
The Girl with Skin of Haints and Seraphs is a polymorphic poem first implemented in a non-interactive form as the initial deployment of the Alloy algorithm for generative purposes within another system. It has been subsequently updated with each iteration of GRIOT and it provides a good example for tracing through the execution of an interactive polymorphic poem. As stated above, this polypoem is a commentary on racial politics, the limitations of simplistic binary views of social identity, and the need for more contingent, dynamic models of social identity. The dynamic nature of social identity is also reflected in the way the program produces different poems with different novel metaphors each time it is run. This LISP program draws on a set of theme domains such as skin, angels, demons, Europe, and Africa, given as sets of axioms.
The following example and discussion illustrate the interactive nature of this polypoem. It also recapitulates and extends the discussion of GRIOT architecture and formal description of text generation described above. After processing a user input keyword such as "Europe" entered at a ">" prompt, the first line could be:
her tale began when she was infected with white female-itis
or
she began her days looking in the mirror at her own pale-skinned death-figure face
or any of a number of alternate phrases (there are fourteen templates for such opening phrases). As an example of variation within a particular phrase due to wildcard replacement, among many other possibilities the first example above could have also been either:
her tale began when she was infected with tribal-warrior spectre-itis
or
her tale began when she was infected with black demon-itis
depending upon how the phrase template was instantiated.
In the example above, one set of phrase templates contains "(her tale began when she was infected with (* g-singular-noun)-itis)" in the LISP syntax of the implementation, where the inner parenthesis is a variable, here called a "wildcard,"that gets replaced with a noun cluster or a noun paired with a modifier. Exactly how the wildcard is replaced is determined by a combination of user input and the contents of the wildcard itself. A wildcard consists of two or more parts including a "*"marker that indicates it is a wildcard, and a variable that determines whether it is to be replaced by another phrase (denoted by the prefix "p-"attached to a clause type name) or by content generated using the Alloy algorithm (denoted by the prefix "g-"attached to a grammatical form name such as "singular-noun"). Optional variables can be used additionally to constrain domains or axioms selected as input to the Alloy blending algorithm (denoted by "d-"and "a-"prefixes respectively, though in practice we have not had to use axiom determining variables). Optional variables can also be used for structural effects such as forcing repeats of wildcard replacement text from earlier in the poem. User input plays a role in wildcard replacement as the user entered keywords determine one of the domains to be used in constructing blends that will be used in template instantiation. In most of the polypoems implemented so far, phrase templates have been most commonly instantiated by replacing the wildcards with English language mappings from conceptual blends produced by Harrell's Alloy conceptual blending algorithm.
The text grammar below gives a basis for rationally reconstructing and enhancing the poetry system (the current implementation was not conceived this way when built, and is less general). A simplified formalization of Labov's narrative of personal experience (due to Joseph Goguen) gives top level structure for poems. One poetic domain contains the template (her tale began when (* g-verb-clause)) in the LISP syntax of the implementation, where the inner parenthesis is a variable that gets instantiated with a verb phrase containing a past tense transitive verb, such as (was infected with (* g-singular-noun)) where again the inner parenthesized phrase is a variable that gets instantiated with a conceptual blend produced by the Allow algorithm. The axiomatic form of the first template is
[her tale began when X. :: , .9]
where we assign saliency .9 (although the current implementation does not have saliencies). Arguments of other templates are instantiated with elements from domains for persons (e.g., a protagonist), places, objects, etc.; it is a major issue for the artist to choose such material appropriately.
An interesting philosophical issue is raised by this program: human input might be considered cheating by traditional AI practitioners, since most AI text generation projects are oriented towards total automation and Turing test competence. But our quite different goal is to use the blending algorithm in a human designed system that generates poetry containing novel metaphors in real-time; just as with computer games, it is desirable and necessary for humans to provide rich content. For such projects, artistic freedom must take precedence over dogmatic Turing test reductionism.
Selected Relevant References
D. Fox Harrell. "Algebra of Identity: Skin of Wind, Skin of Streams, Skin of Shadows, Skin of Vapor," in Critical Digital Studies, Arthur Kroker and Marilouise Kroker, Eds., Toronto:University of Toronto Press, 2008. (originally in CTHEORY, An international peer-reviewed journal of theory, technology, and culture, October, 2005.) [html]
D. Fox Harrell. "GRIOT's Tales of Haints and Seraphs: A Computational Narrative Generation System," in Second Person: Role-Playing and Story in Games and Playable Media, N. Wardrip-Fruin and P. Harrigan, Eds. Cambridge:MIT Press, 2007. [html]
D. Fox Harrell. "Shades of Computational Evocation and Meaning: The GRIOT System and Improvisational Poetry Generation," in Proceedings of the 6th Digital Arts and Culture Conference, 2005, pp. 133-143. [pdf]
Examples
Sample output
Example of authoring polymorphic poetry with GRIOT
Define Me: Chimera
Project Overview
DefineMe: Chimera is a social networking (Facebook) application in which users define metaphorical profiles and avatars for each other, and several games and avatar creation systems where users' representations change dynamically based upon social context, user interaction, and artifact use. The DefineMe database is designed to be lightweight, dynamic, and extensible, while implementing categorical relationships between members. When comparing profiles, DefineMe is designed to match lexical items and logical relations directly, or it can compare the structures of profiles following insights from the analogical structure-mapping engine (SME) developed by Ken Forbus et. al. Influenced by Eleanor Rosch's influential prototype theory, the labeling system can also be used to define aspects of categories themselves. For instance, a 'robin' tag can be added to the category, 'birds,' to define the prototype of that category. In this way, members can belong to multiple groups, but individuals can represent the prototypical members of groups. This relatively lightweight structure avoids some of the pre-defined categorization built into many social networking infrastructures, and has the potential to more nuanced identity phenomena than many hierarchically organized social networking profiles allow.
Demo Poster
Chameleonia
Project Overview
Chameleonia: Days of Lost Selves is an online multi-player game of shifting identity where the construction of self is at stake. Players make gestures associated with traits such as aggression, commerce, ideology, and more. Player's avatars, and their opponents', then transform in response. One moment the player character is a bazooka-toting cowgirl/boy sipping a softdrink - at the next moment a gold chain and pocketwatch wearing tycoon with stock charts bursting from its top-hatted head.
Demo Poster
Avatar Breeder
Project Overview
Avatar Breeder is a genetic algorithm-based procedurally generative satirical artwork constructed by Daniel Upton and Jisun An in Dr. Harrell's graduate course LCC 6312: Design, Technology, and Representation. It is intended to undermine normative categories of identity encountered on bureaucratic forms and in everyday lived experience. In such discrete categorization systems, ethnic identities can be based on geography, nationality, ancestry, family, culture and sub-culture, religion, language, race or any combination of these. In this new ideological thread, how do people identified with mixed ethnicities inhabit the interstitial spaces and margins amidst these complex factors? How do essential categorizations of ethnicity and/or race properly reinforce or obliterate social illusions or personal histories.
Avatar Breeder allows a user to breed avatars together to create new ethnic categories, labeled by users. The user is provided with an initial pool of avatars, each with a labeled ethnicity. The initial pool was seeded with categories from a Georgia Tech admissions form. The user then selects two parents, which genetically combine to create four potential children. Subsequent generations can be created by selecting one of the children and selecting one of four avatars supplied from the initial pool to "breed." All of the different ethnicities begin blending together as the user combines avatars over generations. The user can continue to genetically combine avatars, leaving a family tree tree that ends at the current generation. Throughout the process of breeding a user can trace back through the entire family tree and see which avatars lead to the creation of the newest generation.
Demo
Launch Project
[Demo note: This demo is the prototype resulting from a 3-week graduate project asking students to construct an avatar creation system and backend data representation that challenges disempowering social norms.]
Abstract Avatar
Project Overview
Abstract Avatar allows for users to construct graphical self-representations akin to social networking profiles. These representations are composed of basic media elements such as uploaded images, text, and multimedia files, connected to each other in a self-determined graph structure. Bloggers can attach links to avatars, which readers can access upon receipt of permission. Abstract Avatar was created by the quartet of B.S. computational media students: Allen Morrison, Michael Groves, Kristi Champion, Charles Gunnell in Dr. Harrell's course LCC 3314: Technologies of Representation.
Demo Example
Launch project
[Demo note: This demo is the prototype resulting from a 3-week undergraduate project asking students to construct an avatar creation system and backend data structure that challenges disempowering social norms. The icon in the upper left is a "back" command. Each node is clickable, resulting in further zooming in until a user reaches the data entry level.]
