Large Language Models (LLM) and Vector Databases | AllegroGraph 9.0.1
Author: Franz Incorporated

Copyright (c) 2005 - 2026 Franz, Incorporated
Last updated 27 May 2026 at 10:34

# AllegroGraph 9.0.1 Large Language Models and Vector Databases

- Documentation Index
  - Release Notes
  - Quick Start
  - SSL/TLS Quick Start
  - Introduction
  - Setup and Configuration
    - Download
    - Server Installation
    - Server Configuration and Control
    - WebView
    - Database Upgrading
    - Managing Users
  - Server Management
    - Performance Tuning
    - Deleting Duplicate Triples
    - Purging Deleted Triples
    - Auditing
    - Managing Users
    - Replication
    - Point-in-Time Recovery
    - Transaction Log Archiving
    - Security Overview
    - Security Implementation
  - Tools
    - agtool
    - Data Loading
    - Data Export
    - Backup and Restore
    - Querying
    - RDFS++
    - Materializer
    - Event Scheduler
    - SHACL
  - Details
    - Triple Indices
    - Freetext Indices
    - Encoded IDs
    - Connection Pooling
    - N-dimensional Geospatial Overview
    - 2-D Geospatial
    - Temporal
    - Social Network
    - JavaScript
    - Datatypes
    - Data Conversion (Rapper)
    - Stored Procedures
    - Triple Attributes
    - Lineparse Format
  - Querying
    - SPARQL documentation summary
    - SPARQL Reference
    - SPIN
    - SPARQL Magic Properties
    - Prolog
  - 3rd-party tools
    - Solr text Indices
    - MongoDB integration
    - TopBraid Composer Plugin
    - Cloudera
  - Client APIs
    - Gruff
    - GraphTalker
    - REST/HTTP interface
    - HTTP reference
    - Javadocs (Sesame and Jena)
    - Python API
    - Lisp Reference
    - JavaScript
  - Tutorials
    - N-dimensional Geospatial
    - SPARQL Tutorial
    - Java Sesame
    - Java Jena
    - Python Sesame
    - Lisp
    - Prolog
    - More...
  - Other
    - Conferences and Seminars
    - Source Code on Github
    - Amazon EC2
    - Copyrights
  - Franz, Inc.

- Introduction
- Four aspects of LLM
- Supported vendors and models
- SPARQL query options for LLMs
- LLM Specifications in agraph.cfg
  - OpenAI
  - Ollama
  - SERP
- You may need an API key to use any of the LLM-associated magic predicates
- SPARQL LLM magic predicates (LLMagic)
- Vector databases
- LLM Examples

Documentation Index

## Introduction

Large language models, or LLMs, are a type of AI that can mimic human intelligence. They use statistical models to analyze vast amounts of data, learning the patterns and connections between words and phrases.

AllegroGraph supports integration with Large Language Models.

A Vector Database organizes data through high-dimensional vectors. It is created by taking fragments of text, sending them to the Large Language Model (like OpenAI's GPT), and receiving back embedding vectors suitable for storing in AllegroGraph.

## Four aspects of LLM

1. The supported vendors and models: see Supported models below.

2. Keyword syntax: using Keyword Syntax for Magic Predicates allows simplified and more readable calls to the supported predicates.

3. The selector argument (when the predicate uses vector database) allows reducing what is seached, see the Using the ?selector and ?useClustering arguments with LLM Magic Predicates document.

4. Clustering to speed up vector matching (again, when the predicate uses vector database). see the Using the ?selector and ?useClustering arguments with LLM Magic Predicates document.

## Supported vendors and models

Each version of AllegroGraph supports access to a fixed set of LLMs. The agraph.cfg file created when AllegroGraph is installed contains that list of LLMs as each is described with an LLM directive in that file. You can modify the agraph.cfg file's list of properties for each LLM to set the defaults for each LLM.

When doing SPARQL queries you can set the properties for the LLM operation using query options (expressions beginning with PREFIX) which take precedence over the defaults found in agraph.cfg.

Here are the LLMs supported in the current version of AllegroGraph and the query options which apply to each.

We use an abbreviated name to specify the query options below. The full name which you would use in SPARQL is the name preceeded by `franzOption_`

The `llmVendor` query option must always be specified. The value must be one of the values in the Vendor column below.

for example in SPARQL:

    PREFIX franzOption_llmVendor <franz:openai>

would be used to set the vendor to openai.

## SPARQL query options for LLMs

Vendor | SPARQL Query Option [default value] | directive arguments in agraph.cfg
--- | --- | ---
| openai | lmChatModel [gpt-3.5-turbo] llmEmbeddingModel [text-embedding-3-large] llmEmbeddingLength [*] openaiApiKey | "apikey", "embedding-length"
ollama | llmChatModel [llama3.1] llmEmbeddingModel [llama3.1] llmEmbeddingLength [*] ollamaScheme [http] ollamaHost [localhost] ollamaPort [11434] | "scheme", "host", "port", "embedding-length"
| claude | llmChatModel [Claude Sonnet] claudeApiKey | "apikey"
azure | azureApiKey azureEndpointUrl llmEmbeddingModel llmEmbeddingLength | "apikey", "endpoint-url", "endpointurl"
| deepseek | llmChatModel [DeepSeek-R1-Zero] deepseekApiKey | "apikey"
groq | llmChatModel [deepseek-r1-distill-llama-70b] groqApiKey groqEndpointUrl [https://api.groq.com/openai/v1] | "apikey"
databricks | lmChatModel [default] llmEmbeddingModel [default] databricksApiKey databricksEndpointUrl | "apikey", "endpoint-url"
gemini | geminiApiKey llmChatModel | "apikey"

Notes about the table above.

1. openaiApiKey can be stored by the UI (see below)
2. A databricks endpoint url points to either an embedding or chat service with the model already set so there’s no reason to specify the chat and embedding models. We use default as a placeholder model name but you need not specify it
3. If an LLM has a ApiKey query option then that api key is required.

## LLM Specifications in agraph.cfg

The configuration file agraph.cfg lists all of the LLMs for which there is support in this particular version of agraph. Some default values are given as well and many are incorrect but are present so you can see what to replace when you know the actual location of the LLM servers.

In agraph.cfg the form of an LLM specification is

    <llm NAME>
    DIRECTIVE VALUE ARG1=”value1”, ARG2=”value2” …
    DIRECTIVE VALUE ARG1=”value1”, ARG2=”value2” …
    DIRECTIVE VALUE ARG1=”value1”, ARG2=”value2” …
    </llm>

The Directives are one of the words from the following table. The case (upper/lower) of these words is unimportant and many use camel case (e.g EndpointUrl) to make reading easier.

Only the directives ChatModel and EmbedderModel can have additional arguments specified as shown above.

The values of the arguments can be a number like 123 or a string and if a string it must be surrounded by double quotes to indicate it's a string.

The VendorField directive specifies a parameter that is used to connect to or configure the this particular LLM. Sometimes for convenience a particular parameter is also available as a directive so that it applies to all chat and embedder models. For example in the ollama specficiation

    VendorField host
    Host localhost

says that host is a parameter needed to connect to this LLM and also that the default value is localhost if not given on the ChatModel or EmbedderModel line.

Each LLM’s take certain directives and the agraph.cfg generated when AllegroGraph is installed shows which directives are commonly used and one need only edit what’s in agraph.cfg to make it apply to your setup.

Directive | Value
--- | ---
EmbedderModel | name of an embedder model. Can be repeated. The first one listed is considered the default model in a sparql query where the llmEmbedderModel query option is not supplied.
ChatModel | name of a chat model. Can be repeated The first one listed is considered the default model in a sparql query where the llmChatrModel query option is not supplied.
NeedsApiKey | true if this LLM requires an api key to process a request
ApiKey | The API key passed to the LLM to process this request. Typically this is not put in agraph.cfg for security reasons but it is specified in the appropriate query option for the LLM.
EndpointUrl | Many LLMs require that you specify the location of the LLM server. You use this to make that specification. Alternatively you can use the directive argument endpoint-url (note the hyphen) to do the same thing. This directive thing and this takes precedence of the EndpointUrl directive
Scheme | For ollama you need to specify the scheme, host and port of the server and from that the endpoint url is computed by the agraph code.
Host | For ollama you need to specify the scheme, host and port of the server and from that the endpoint url is computed by the agraph code.
Port | For ollama you need to specify the scheme, host and port of the server and from that the endpoint url is computed by the agraph code.
length | This is deprecated. Use the "embedding-length" argument on the EmbedderModel line instead.
VendorField | This can be specified more than once to say what the valid ARGs are in a DIRECTIVE statement

Currently the supported LLM venders are OpenAI GPT (the default option for chat and embedder models) and Ollama.

### OpenAI

OpenAI is a vendor described at OpenAI GPT. OpenAI requires an API key.

Starting in release 8.3.1, admin users (see the Managing Users document) can specify an OpenAI key in AG WebView by navigating to the webview/welcome/ page (that is [server:port]/webview/welcome/). Doing so will cause this dialog to be displayed:

Admin users can then fill in the OpenAI key (already done in the illustration) and click Save. The key will be used whenever it is required.

(Note the webview/welcome page was introduced in version 8.3.1. In version 8.3.0, OpenAI keys were associated with individual vector repos and with individual SPARQL queries, with a PRFIX) rather than at once. There are many places in the documentation when you are told how to specify the OpenAPI key, and those instructions can be ignored if an admin user specified it on the webview/welcome page. But they will all still work and if an admin users has not set things globally as shown here, all users must specify the key when required.)

The WebView tool supports an LLM Playground where you can try out LLM queries which require the OpenAI API key. That tool makes it very easy to experiment with such LLM queries.

OpenAI has various models. Some are specified in agraph.cfg and you can add more if you like.

The default model for the llm:response, llm:askMyDocuments, and llm:askForTable magic properties (see Use of Magic predicates and functions) is the first ChatModel listed in the openai llm specification in agraph.cfg

### Ollama

Ollama does not require a key but does require an entry in the AllegroGraph configuration as file described here in the Server Configuration and Control document. See the Ollama document for more information.

### SERP

We also use SERP API, which is a service that returns Google and other search results through an API. It is used in conjuction with LLM APIs to validate LLM results (i.e. detect hallucinations). Separate keys are required for each.

## You may need an API key to use any of the LLM-associated magic predicates

Ollama does not require a key but OpenAI and SERP do.

The keys shown below are not valid OpenAI API or serpAPI keys. They are used to illustrate where your valid key will go.

To get an OpenAI API key go to the OpenAI website and sign up. To get a serpAPI key, go to https://serpapi.com/.

There are various ways to configure your API key, as a query option prefix or in the AllegroGraph configuration.

The actual openaiApiKey starts with `sk-` followed by a lot of numbers and letters. The serpApiKey is just a lot of numbers and letters. In the query option prefixes just below, the keys are preceded with `franz:` and the whole thing is wrapped in angle brackets. Similarly for the query options just below that.

As a query option prefix, write (the keys are not valid, replace with your valid key(s)):

    PREFIX franzOption_openaiApiKey: <franz:sk-U01ABc2defGHIJKlmnOpQ3RstvVWxyZABcD4eFG5jiJKlmno>

    PREFIX franzOption_serpApiKey: <franz:11111111111120b7c4d3d04a9837e3362edb2733318effec25c447445dfbf594>

Syntax for config file which causes the option to be applied to all queries (see the Server Configuration and Control document for information on the config file):

    QueryOption openaiApiKey=sk-U01ABc2defGHIJKlmnOpQ3RstvVWxyZABcD4eFG5jiJKlmno

    QueryOption serpApiKey=11111111111120b7c4d3d04a9837e3362edb2733318effec25c447445dfbf594

The first `openaiApiKey` option is applied to each query in the LLM Playground, but not the second `serpApiKey`.

You can also put the value the file data/settings/default-query-options file:

    (("franzOption_openaiApiKey" "sk-U01ABc2defGHIJKlmnOpQ3RstvVWxyZABcD4eFG5jiJKlmno"))

    (("franzOption_serpApiKey" "11111111111120b7c4d3d04a9837e3362edb2733318effec25c447445dfbf594sk"))

Remember your key is valuable (its use may cost money and others with the key can use it at your cost) so protect it.

The WebView LLM Playground prompts you for the OpenAI API key at the start of a session. First click on the Create LLM Playground button:

The following dialog comes up. Enter your OpenAI API key as shown (again that key is not actually valid):

Click Create & Open and a repo which automatically adds a PREFIX with the OpenAI API key to every query. (It does not add a PREFIX with the SERPApi API key. You must supply that yourself if you use the askSerp magic predicate. See the serpAPI example.

## SPARQL LLM magic predicates (LLMagic)

AllegroGraph LLMagic supports LLM magic predicates (internal functions which act like SPARQL operators). These are listed in the SPARQL Magic Property document under the heading Large Language Models*. (You click on a listed magic property to see additional documentation.) See also LLMagic functions listed here.

For example the `http://franz.com/ns/allegrograph/8.0.0/llm/response` magic property prompts GPT for a response. For example, we can ask for a list of 5 US states:

    PREFIX llm: <http://franz.com/ns/allegrograph/8.0.0/llm/>

    SELECT * { ?response llm:response
               "List 5 US States.". }

which produces the response:

    response

    "California"
    "Texas"
    "Florida"
    "New York"
    "Pennsylvania"

## Vector databases

A vector database is a database that can store vectors (fixed-length lists of numbers) along with other data items. Vector databases typically implement one or more Approximate Nearest Neighbor (ANN) algorithms, so that one can search the database with a query vector to retrieve the closest matching database records. (Wikipedia)

See the document Vector Databases and Storage for further information and examples using vector databases.

## LLM Examples

See LLM Examples. The LLM Split Specification document also contains some examples. The LLM Embed Specification has some vector database examples. The Natural Language Queries/Q&A system/ChatBot document shows a natural language interface for querying LLM databases. Building a Knowledge Graph via an LLM shows building a Chatbot on the AllegroGraph.cloud server. The example can also be run in a local AllegroGraph.