Synthetic Teammate

The Synthetic Teammate project (Ball et aI., 2010) is an early applied research project aimed at research and development of language and task enabled constructive entities that can function as synthetic teammates in Air Force relevant team training simulations. The research is primarily supported by the Air Force Research Laboratory, 711th Human Performance Wing, Human Effectiveness Directorate, Warfighter Readiness Research Division. Cognitive Engineering Research on Team Tasks (CERTT) is the Remotely Piloted Aircraft (RPA) simulation in which the Synthetic Teammate is being integrated. The Synthetic Teammate functions as the pilot, interacting with a human navigator and photographer to perform a series of 40-minute reconnaissance missions. Cognitive Engineering Research Institute (CERI) conducted an experiment involving all-human teams that provided the text chat corpus that supports research and development of the language analysis and generation capabilities of the synthetic teammate. Next year, CERL will conduct an empirical study of the synthetic teammate interacting with two human teammates.

The Synthetic Teammate project uses the ACT-R Cognitive Architecture (Anderson, 2007) to support development of a psychologically realistic teammate. ACT-R (Adaptive Components of Thought-Rational) is a theory of human cognition implemented as a computer program. ACT-R combines a declarative memory representing knowledge of facts with a procedural memory representing skills. ACT-R also provides perceptual-motor mechanisms for interacting with a computer monitor, keyboard, and mouse. ACT-R is designed to support the research and development of high-fidelity computational cognitive models of human behavior.

The Synthetic Teammate project uses Double-R (Ball, 2007, 2011) for language analysis. Double-R (i.e. referential and relational) is a linguistic theory of the grammatical encoding of referential and relational meaning implemented as a cognitive model in ACT-R. Double-R has been under research and development in-house over the past 10 years.

The other components of the Synthetic Teammate have been under research and development in-house for the past 3 to 5 years. To represent the synthetic teammate's situation and task awareness, we are developing a situation model (Rodgers et aI., 2012). The situation model interfaces with the language analysis and agent-environment interaction components (Myers, 2009). The agent-environment interaction component implements the task behavior of the synthetic teammate, interacting with CERTT to fly the simulated RPA. The language generation component generates messages, which are sent to human teammates. Basic dialog management capabilities have also been implemented. An overview of the synthetic teammate, highlighting the major components, is shown in Figure 1.

Figure 1: Synthetic Teammate Overview

The Synthetic Teammate project is unique in attempting to build a functional teammate that is also cognitively plausible, or human-like, in its behavior. The focus on building a large-scale, functional model differentiates the research from typical cognitive modeling research. The synthetic teammate is one of the largest cognitive models ever built. Overall, the synthetic teammate consists of more than 1500 productions to implement skilled behavior and 58,000 facts to represent knowledge.

The focus on cognitive plausibility, or human-like behavior, differentiates the research from typical Al research . We are attempting to demonstrate that adhering to well-established cognitive constraints may facilitate the development of functional systems of complex human behavior. Such constraints may prove functional in that they guide research in directions that will ultimately lead to human-like behavior. For training purposes, it is essential that the synthetic teammate act like a human, not an intelligent agent. A synthetic teammate which failed to model the capabilities and limitations of humans would reduce the training benefit to human teammates.

The Synthetic Teammate project is entering a crucial stage. Text chat messages can now be sent from a human navigator and photographer to the Synthetic Teammate. We are in a position to advance the language and situation modeling capabilities of the system by exposing it to the full range of variability that occurs in human to human communication. Handling such variability is crucial to development of a functional system. We previously relied on use of lightweight synthetic agents to send messages to the synthetic teammate. Now, a human navigator can send a message about the next waypoint to the Synthetic Teammate, and the Synthetic Teammate sets the waypoint and adjusts the course in response. In addition, if a human photographer sends a message stipulating altitude and airspeed restrictions, the synthetic teammate will comply with these restrictions.

Our current goal is to expand the capabilities of the Synthetic Teammate over the next several months in preparation for the empirical study of the Synthetic Teammate interacting with a human navigator and photographer. The performance of a group of synthetic pilot plus human navigator/photographer teams will be compared against the performance of a group of all-human teams. Once the synthetic teammate has been empirically tested in the RPA reconnaissance domain, we aim to generalize the capabilities of the synthetic teammate to support rapid transition to other Air Force relevant team training domains.

Figure 2: CERTT and Synthetic Teammate Displays

Figure 3: CERTT Synthetic Task Environment (Upper Left Display)

Left: Navigator (DEMPC) task consoles (warnings/status and waypoint map) Middle: Photographer (PLO) task consoles (warnings/status and camera settings) Right: Pilot (AVO) task consoles (warnings/status and flight instruments)

Figures 2 through 6 show the current state of the synthetic teammate. Figure 2 shows the physical layout, providing perspective for Figures 3 through 6. Figures 3 and 4 show the CERTI system. Figures 5 and 6 show the Synthetic Teammate.

Figure 4: CER7T Synthetic Task Environment (Lower Left Display)

Left: Chat server and experiment control station Right: Photographer and navigator chat clients

Figure 5: Synthetic Teammate (Pilot) - (Upper Right Display)

Left: ACT-R emulation of pilot warnings/status console Right: Double-R representations generated for: "The first waypoint is LVN. It is an entry. There are no airspeed or altitude restrictions .

Figure 6: Synthetic Teammate (pilot) - (Lower Right Display)

Upper left: ACT-R emulation of pilot flight instruments console Lower left: ACT -R emulation of pilot chat client with red ring showing current focus of attention on the word " altitude" Right: ACT -R production execution trace during the processing of "the altitude..."

For additional details about the Synthetic Teammate Project, see here.