AllegroGraph 8.3.0 LLM chatState Magic Predicate Details

• Documentation Index
  • Release Notes
  • Quick Start
  • SSL/TLS Quick Start
  • Introduction
  • Setup and Configuration
    • Download
    • Server Installation
    • Server Configuration and Control
    • WebView
    • Database Upgrading
    • Managing Users
  • Server Management
    • Performance Tuning
    • Deleting Duplicate Triples
    • Purging Deleted Triples
    • Auditing
    • Managing Users
    • Replication
    • Point-in-Time Recovery
    • Transaction Log Archiving
    • Security Overview
    • Security Implementation
  • Tools
    • agtool
    • Data Loading
    • Data Export
    • Backup and Restore
    • Querying
    • RDFS++
    • Materializer
    • Event Scheduler
    • SHACL
  • Details
    • Triple Indices
    • Freetext Indices
    • Encoded IDs
    • Connection Pooling
    • N-dimensional Geospatial Overview
    • 2-D Geospatial
    • Temporal
    • Social Network
    • JavaScript
    • Datatypes
    • Data Conversion (Rapper)
    • Stored Procedures
    • Triple Attributes
    • Lineparse Format
  • Querying
    • SPARQL documentation summary
    • SPARQL Reference
    • SPIN
    • SPARQL Magic Properties
    • Prolog
  • 3rd-party tools
    • Solr text Indices
    • MongoDB integration
    • TopBraid Composer Plugin
    • Cloudera
  • Client APIs
    • Gruff
    • REST/HTTP interface
    • HTTP reference
    • Javadocs (Sesame and Jena)
    • Python API
    • Lisp Reference
    • JavaScript
  • Tutorials
    • N-dimensional Geospatial
    • SPARQL Tutorial
    • Java Sesame
    • Java Jena
    • Python Sesame
    • Lisp
    • Prolog
    • More...
  • Other
    • Conferences and Seminars
    • Source Code on Github
    • Amazon EC2
    • Copyrights
    • Change History
  • Franz, Inc.

LLM chatState Magic Predicate Introduction

The chatState predicate implements a chatbot with a long-term memory of conversation history. The LLM process defined here is called 'Retrieval Augmented Generation with Feedback', or 'RAG with Feedback'. The term 'Feedback' in this case refers to storing short-term memories of the recent dialog in the long-term memory of a vector database.

When responding to a human input, the chatState predicate draws upon two sources of information:

• The long- and short-term memory of the conversation with that individual, called the History
• All other facts, memories and experiences the bot knows, called the Expertise.

The chatState predicate retrieves this content from two vector databases, referred to as the Expertise Vector Database and History Vector Database respectively. As the dialog progresses with each subsequent execution of chatState, RAG with Feedback embeds windows of conversation history into the History Vector Database, in real-time.

We query the vector database with a string called the match string. The match string may contain arbitrary text, optionally including the bot's name, the human's name or role, the most recent human input, and an ingredient called the feed or dialog feed.

The feed is simply a transcript of the last few lines of dialog. Conceptually it may be easy to think of the feed in the context of texting on a phone. The feed is analogous to what you can see on the screen, before it scrolls off the top. The number of dialog lines in the feed is the same as windowSize (see below).

With 8 inputs and 14 outputs, the general form of the chatState predicate looks complicated,

(?response  
?score  
?citationId  
?citedText  
?source  
?feed  
?story  
?expertiseMatchString  
?expertiseMatches  
?expertiseHints  
?historyMatchString  
?historyMatches  
?historyHints  
?prompt)  
  llm:chatState  
(?text  
?expertiseRepoSpec  
?expertiseMaxMatches  
?expertiseMinScore  
?historyRepoSpec  
?historyMaxMatches  
?historyMinScore  
?botId). 

This complexity diminishes when the predicate utilizes only the 3 required variables:

?response llm:chatState (?text ?expertiseRepoSpec). 

We'll come back to the meaning of that simple statement in a moment, but first let us review the meaning of each chatState variable.

On the input side, the required variable ?text binds the text of the human input. The required variable ?expertiseRepoSpec binds the name of the Expertise vector database.

We describe the remaining optional input values in this table:

For the optional variables in the input table above, the value of each depends on an order of precedence:

On the output side, the predicate binds the LLM response to the required parameter ?response.

The optional variables provide the SPARQL query author or repo admin with useful feedback on the construction of the LLM prompt. A typical query in production may include only ?response, ?score, ?citationIdId, ?citedText and ?source.

The complete set of output-side variables:

In order to achieve RAG with Feedback, the optional variables ?expertiseMaxMatches, ?expertiseMinScore and historyRepoSpec must be bound. The two required ?text and ?expertiseRepoSpec items must also be bound.

Let's return to the simpler expression using chatState, from above.

  ?response llm:chatState (?text ?expertiseRepoSpec). 

In fact, the query

(?response ?score ?citationId ?citedText) llm:askMyDocuments (?text ?expertiseRepoSpec ?topN ?minScore)  

is equivalent to

(?response ?score ?citationId ?citedText) llm:chatState (?text ?expertiseRepoSpec ?topN ?minScore) 

as long as the stateFmtString property value (see above Profile Configuration Properties table is the same as the askMyDocuments prompt:

Here is a list of citation IDs and content related to the query '\{QUERY\}': \{EXPERTISE\_MATCHES\}.  
Respond to the query '\{QUERY\}' as though you wrote the content.  Be brief.  You only have 20 seconds to reply.  
Place your response to the query in the 'response' field.  
Insert the list of citations whose content informed the response into the 'citation_ids' array. 

chatState Configuration

In any repo where you use the chatState predicate in a query, you may include a set of triples to configure the bot and design a template for the LLM prompt, overriding the default Profile properties.

These triples, called Profile triples, define properties and values to build what might be a very long prompt, and several of the Profile property values specify substrings of that prompt, so that the full prompt can be built in smaller steps.

The property values may also contain string replacement expressions to be populated with the the values of other previously instantiated properties. For example the expression {BOT} appearing in an property value is replaced with the value set by the bot property. Each property value may contain only the string replacements set previously. The table column labeled May Include in the table below, lists the replacement expressions permitted in the corresponding property value.

The profile triples are all utilize the namespace <http://franz.com/chatState/>. The subject of each is <http://franz.com/chatState/profile> or, if there is more than one profile in the repo, <http://franz.com/chatState/profile#bot-id>, were bot-id is the name of the bot. The predicate names consist of the the property name appended to the namespace.

Some examples from the table below, in ntriple format:

  <http://franz.com/chatState/profile> <http://franz.com/chatState/windowSize> "8"^^<http://www.w3.org/2001/XMLSchema#integer> . 

  <http://franz.com/chatState/profile> <http://franz.com/chatState/historyMatchFmtString> " {FEED}  {HUMAN}:  {QUERY}" . 

When a profile triple exists, the value of the Profile property in the triple takes precedence over the default value.

The following table describes the Profile Configuration properties.

Note 1: The value of {FEED} is a string of concatenated exchanges, where each exchange is string of the form

    {HUMAN}: {QUERY}   {BOT}: {RESPONSE} 

Note 2: the value of {EXPERTISE_MATCHES} and {HISTORY_MATCHES} is a string of concatenated matches, where each match has the form

 citation_id: {CITATION_ID} Hint: {TEXT}

Note 3: In Note 1 and Note 2, "concatenated" means the strings appended together with a newline inserted between. These values all have to be strings, rather than data structures, because they become part of the LLM prompt.

Note 4: llm:chatState may utilize keyword syntax.

chatState Example

Follow these directions to try a chatState with a sample configuration and vector databases.

1. Follow the instructions in Chomsky RAG Example: Download the chomsky47.nq file and create the repos chomsky47 and chomsky47-vec. chomsky47-vec is your Expertise repo. Note that the AllegroGraph.cloud server has the chomsky and chomsky-vec repos already loaded and set up (the names do not include the 47 but are otherwise identical), allowing you to run the example here with much less initial work.

2. Create a vector database called chomsky47-history, your History repo. It will be empty when created. You can use agtool repo create --vector-store ... or use the WebView interface interface.

3. You can use all the default values from the Profile Configuration Properties above, with two exceptions:

Load these two profile triples into chomsky47:

<http://franz.com/chatState/profile> <http://franz.com/chatState/bot> "Noam Chomsky" .  
<http://franz.com/chatState/profile> <http://franz.com/chatState/human> "Interviewer" . 

These values, specific to the Chomsky bot, will override the default values for bot and human.

Now, in the repo chomsky47, execute the following SPARQL query:

 PREFIX llm: <http://franz.com/ns/allegrograph/8.0.0/llm/>  
 # When using the chomsky/chomsky-vec store in AllegroGraph.cloud  
 # uncomment the following line with a valid openaiApiKey:  
 # PREFIX franzOption_openaiApiKey: <franz:sk-XXvalid key hereXX>  
 SELECT ?response {  
  ?response llm:chatState ("Hello" "chomsky47-vec" 4 0.75 "chomsky47-history" 4 0.0)  
 # In AllegroGraph.cloud the vector store is named chomsky-vec. Modify  
 # the above line.  
 }  

The Chomsky bot's response to "Hello" will be bound to ?response.

Now repeat the same query, replacing "Hello" with a series of different queries:

 "My name is Lex.  I was born in 1983 in the Soviet Union. I graduated with a PhD from Drexel University"  
 "Define Universal Grammar in 10 words or less."  
 "Do you believe animals can understand language?"  
 "What about my cat.  I say 'Come here, Kitty' and she responds."  
 "What about humpback whales?" 

Next, ask

 "What do you know about me?" 

The Chomsky bot should respond with something like

 "I recall you mentioned, Lex, that you were born in 1983 in the Soviet Union and later pursued a PhD at Drexel University." 

Now, if we query the repo chomsky47-history with

SELECT ?human ?bot  
{  
  ?s rdf:type <http://franz.com/chatState/Exchange>.  
  ?s <http://franz.com/chatState/user> ?human.  
  ?s <http://franz.com/chatState/bot> ?bot.  
  ?s <http://franz.com/chatState/exchange-index> ?index  
} ORDER BY ?index  

we can retrieve a full transcript of the interview:

Because of the random nature of LLMs, your transcript may not be exactly the same, but generally the conversation should follow the same flow.

The same questions posed to the chomsky bot described in the Chomsky RAG Example, which does not maintain a history, gives similar answers to the questions up to the last, but the answer to the last question is completely different with less useful content.