Lisp Freetext Indexing Tutorial | AllegroGraph 9.0.1
Author: Franz Incorporated

Copyright (c) 2005 - 2026 Franz, Incorporated
Last updated 27 May 2026 at 10:34

# AllegroGraph 9.0.1 Freetext Indexing Tutorial

- Documentation Index
  - Release Notes
  - Quick Start
  - SSL/TLS Quick Start
  - Introduction
  - Setup and Configuration
    - Download
    - Server Installation
    - Server Configuration and Control
    - WebView
    - Database Upgrading
    - Managing Users
  - Server Management
    - Performance Tuning
    - Deleting Duplicate Triples
    - Purging Deleted Triples
    - Auditing
    - Managing Users
    - Replication
    - Point-in-Time Recovery
    - Transaction Log Archiving
    - Security Overview
    - Security Implementation
  - Tools
    - agtool
    - Data Loading
    - Data Export
    - Backup and Restore
    - Querying
    - RDFS++
    - Materializer
    - Event Scheduler
    - SHACL
  - Details
    - Triple Indices
    - Freetext Indices
    - Encoded IDs
    - Connection Pooling
    - N-dimensional Geospatial Overview
    - 2-D Geospatial
    - Temporal
    - Social Network
    - JavaScript
    - Datatypes
    - Data Conversion (Rapper)
    - Stored Procedures
    - Triple Attributes
    - Lineparse Format
  - Querying
    - SPARQL documentation summary
    - SPARQL Reference
    - SPIN
    - SPARQL Magic Properties
    - Prolog
  - 3rd-party tools
    - Solr text Indices
    - MongoDB integration
    - TopBraid Composer Plugin
    - Cloudera
  - Client APIs
    - Gruff
    - GraphTalker
    - REST/HTTP interface
    - HTTP reference
    - Javadocs (Sesame and Jena)
    - Python API
    - Lisp Reference
    - JavaScript
  - Tutorials
    - N-dimensional Geospatial
    - SPARQL Tutorial
    - Java Sesame
    - Java Jena
    - Python Sesame
    - Lisp
    - Prolog
    - More...
  - Other
    - Conferences and Seminars
    - Source Code on Github
    - Amazon EC2
    - Copyrights
  - Franz, Inc.

- Setting Up
- Basic Queries
- Wild-card and Fuzzy Matching
- Indexing By Predicate
- Ignored Words
- Indexing Subject, Predicate, and Graph Fields
- Searching from SPARQL
- Textual Query Syntax

Documentation Index

As in the other tutorials, you should read this while running an interactive Lisp session. The forms should be evaluated one after another from the top of the tutorial to the bottom.

Freetext indices provide a way to quickly retrieve triples containing a given word, allowing queries similar to those understood by Web search engines.

## Setting Up

We start by creating a triple store with a text index:

    > (create-triple-store "fti")
    > (create-freetext-index "index1")

And add a few triples:

    > (enable-!-reader)
    > (register-namespace "ex" "http://franz.com/example#")
    > (add-triple !ex:Nietzsche !ex:quote
                  !"We have art in order not to die of the truth.")
    > (add-triple !ex:Picasso !ex:quote
                  !"Art is the lie that enables us to realize the truth.")
    > (add-triple !ex:Wilde !ex:quote !"All art is quite useless.")

## Basic Queries

The words in the subjects of each of these triples are indexed. We can now use freetext-get-triples to query them. This function returns a cursor, from which we just collect the subjects:

    > (collect-cursor (freetext-get-triples "art") :transform 'subject)
    ({Nietzsche} {Picasso} {Wilde})

Here, the query is simply `"art"`. The index is case-insensitive, so the triple where the word 'art' is capitalized is also found, and a query of `"ART"` would have returned the same results.

More complicated queries can be made by combining queries with boolean operators:

    > (collect-cursor (freetext-get-triples '(and "art" "truth"))
                      :transform 'subject)
    ({Nietzsche} {Picasso})
    > (collect-cursor (freetext-get-triples '(or "useless" "die"))
                      :transform 'subject)
    ({Nietzsche} {Wilde})

To match an exact phrase, a query can use the `phrase` operator, as in:

    > (collect-cursor (freetext-get-triples '(phrase "art is"))
                      :transform 'subject)
    ({Picasso} {Wilde})

Note that `and` and `or` allow any kind of query to appear as their argument, so `(and (phrase "art is") "truth")` is a valid query.

## Wild-card and Fuzzy Matching

Freetext indices support wild-card matching with the `match` operator:

    > (collect-cursor (freetext-get-triples '(match "reali?e"))
                      :transform 'subject)
    ({Picasso})

The string given to `match` can contain question marks to match single characters, or asterisks to match any number of characters. Be aware that when an asterisk appears early in the string, as in `"*tastic"`, it is impossible for AllegroGraph to make effective use of the text index, and the query will be slower.

Another form of non-exact matching can be done with the `fuzzy` operator. This one matches all words within a given Levenshtein distance (or edit distance) of the given term. The Levenshtein distance between two strings is the amount of edits that have to be made to change the one into the other, where an edit is the insertion or deletion of a single character, or the replacement of a character with another one. `(fuzzy "realise" 1)` will match `realise`, `realize`, `realist`, and so on. If the distance argument is omitted, one fifth of the word length (rounded up) is used.

## Indexing By Predicate

The create-freetext-index function takes a lot more arguments than just a name. It is, for example, possible to only index triples that have certain predicates:

    > (create-freetext-index "name-index" :predicates (list !ex:name))
    > (list-freetext-indices)
    ("name-index" "index1")

This new index will not index the triples with the `!ex:quote` predicate. But it will index these:

    > (add-triple !ex:Nietzsche !ex:name !"Friedrich Wilhelm Nietzsche")
    > (add-triple !ex:Picasso !ex:name !"Pablo Diego José Francisco de Paula
        Juan Nepomuceno María de los Remedios Cipriano de la Santísima Trinidad
        Ruiz y Picasso")

freetext-get-triples takes an `:index` keyword argument that is used to only use a single index. If not given, all existing indices in the store are used.

    > (count-cursor (freetext-get-triples "art"))
    3
    > (count-cursor (freetext-get-triples "art" :index "name-index"))
    0
    > (count-cursor (freetext-get-triples "Pablo" :index "name-index"))
    1

## Ignored Words

Several of our quotes contain the word `is`, yet when we search for that, we get nothing:

    > (collect-cursor (freetext-get-triples "is") :transform 'subject)
    ()

To prevent the index from containing useless information, by default all words shorter than three characters are ignored. On top of that, there is a stop-word list, the content of which is also ignored. By default this list contains a number of common English words. If we are interested in short and common words, we can create an index that indexes them:

    > (create-freetext-index "index2" :minimum-word-size 2 :stop-words ())
    > (collect-cursor (freetext-get-triples "is") :transform 'subject)
    ({Picasso} {Wilde})

## Indexing Subject, Predicate, and Graph Fields

create-freetext-index also allows control over which part of a triple are indexed. The `:index-fields` argument defaults to `(:object)`, meaning only the object is indexed, but may contain any of `:subject`, `:predicate`, `:object`, and `:graph`. Our subjects and predicates contain resources, not literals, though, and by default those are not indexed. We can set `:index-resources` to `T` when creating an index to fix this. Another possible value is `:short`, which will cause only the part of the resource after the last `/` or `#` to be indexed.

    > (create-freetext-index "sp-index"
                             :index-fields '(:subject :predicate)
                             :index-resources :short)
    > (collect-triples (freetext-get-triples "Nietzsche" :index "sp-index"))
    (<Nietzsche quote We have art in order not to die of the truth.>
     <Nietzsche name Friedrich Wilhelm Nietzsche>)

A similar setting exists to control the indexing of literals, `:index-literals`. This defaults to `T`, causing all literals to be indexed, but can be set to `nil` to not index literals, or to a list of resources to index only literals with the given types.

    > (create-freetext-index "typed-literals"
                             :predicates (list !ex:test)
                             :index-literals (list !ex:indexme))
    > (add-triple !ex:A !ex:test !"hello")
    > (add-triple !ex:B !ex:test !"hello"^^ex:indexme)
    > (collect-cursor (freetext-get-triples "hello" :index "typed-literals")
                      :transform 'subject)
    ({B})

Indices can be deleted at any time with drop-freetext-index:

    > (drop-freetext-index "typed-literals")
    > (list-freetext-indices)
    ("sp-index" "name-index" "index1")

## Searching from SPARQL

The AllegroGraph SPARQL engine defines a 'magic' predicate `<http://franz.com/ns/allegrograph/2.2/textindex/match>`, which can be used to generate bindings for the subjects of triples that match a given freetext query.

    > (sparql:run-sparql
       "PREFIX fti: <http://franz.com/ns/allegrograph/2.2/textindex/>
        SELECT ?x WHERE { ?x fti:match 'remedios' }"
       :results-format :lists)
    (({Picasso}))
    :select
    (?x)

This matches only `ex:Picasso` because the only triple in which `"remedios"` occurs is Picasso's name.

## Textual Query Syntax

To express more complicated queries, the `fti:match` predicate understands a simple language, where multiple words mean `and`, and a pipe character means `or`.

    > (sparql:run-sparql
       "PREFIX fti: <http://franz.com/ns/allegrograph/2.2/textindex/>
        SELECT ?x WHERE { ?x fti:match '(art | truth) useless' }"
       :results-format :lists)
    (({Wilde}))
    :select
    (?x)

Furthermore, double-quotes around a piece of text can be used to express the `phrase` operator, which matches only triples that contain the whole phrase. Wild-card matching can be done simply by including question marks and asterisks in words, and fuzzy matching is done by appending a tilde (`~`) to a word, optionally followed by a maximum edit distance.

The function `ag.text-index:parse-query` transforms such a textual query into an S-expression.

    > (ag.text-index:parse-query
       "allegro* \"freetext index\" (fuzzy | levenshtein~2)")
    (:and (:match "allegro*") (:phrase "freetext index")
     (:or "fuzzy " (:fuzzy "levenshtein" 2)))