AllegroGraph 7.0.3
AllegroGraph Server: General Changes
No significant changes.
HTTP Client
No significant changes.
SPARQL
Rfe16530 - Add support for xsd:dateTime subtraction in SPARQL queries
Arithmetic operations in SPARQL's BIND clauses now support xsd:date/xsd:time/xsd:dateTime subtraction that returns xsd:duration values, i.e. query like
SELECT ?x
WHERE {
BIND (("2020-07-29T17:26:08Z"^^xsd:dateTime -
"2020-07-28T17:26:08Z"^^xsd:dateTime) AS ?x)
} will return a value
"P1D"^^xsd:duration Note that results of date/time subtraction are restricted to xsd:dateTimeDuration subset of xsd:duration values for simplicity reasons.
Bug26309 - Missing zero-length property path results if both s and o are collections
It was possible for a query of the form:
select ... {
values ?start { ... }
values ?end { ... }
?start ( <ex:pred1> / <ex:pred2>* ) ?end
} with triple data:
<ex:s> <ex:pred1> <ex:o> to not return the valid solution:
?start = <ex:a>
?end = <ex:b> where
Bug26293: Property path error if object value constrained
In certain cases a property path expresion in which the object value is constrained to a literal, could lead to an error.
For example in this query ?o is constrained to value 2:
SELECT ... {
bind(2 as ?o).
?s <ex:p>+ ?o
} Depending on the active indices and the exact triples in the repository, this could give the error:
The assertion
(loop for x in (list s p o g s-end p-end o-end g-end)
always (typep x (quote (or upi null))))
failed This is now fixed.
Bug26282 - Stack overflow during OPTIONAL query planning
Certain queries containing an OPTIONAL clause could lead to an error ("stack overflow") during query planning. This is now fixed.
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Prolog
No significant changes.
Documentation
No significant changes.
Distributed AllegroGraph
No significant changes.
AllegroGraph 7.0.2
AllegroGraph Server: General Changes
An AllegroGraph server may restrict itself to the number of licensed cores
If the UseLicensedCores configuration directive is specified true (that is UseLicensedCores true is in agraph.cfg) then the agraph process will automatically reduce the number of cores it uses to the value given in the license file. Otherwise the server will fail to start up if there are more cores on the machine than the license specifies.
Bug26226 - Changing passwords in AGWebView could fail
When a password has expired, changing it in AGWebView could fail. This has been fixed.
Bug26221 - Speedup fti phrase match searches
Fixed an inefficiency in FTI phrase matches that caused a slowdown when a word in the phrase appeared in a single triple multiple times. This is now optimized. The speedup seen will vary based on the frequency in which a word from the phrase appears in a given triple. For small string literals and resources, the improvement will be very modest, but for phrase matches in large string literals the speedup can be significant
HTTP Client
No significant changes.
SPARQL
Bug26274 - Insert ..^^xsd:string fails
Queries that insert a triple with an XSD string as object, like:
INSERT {
<ex:s> <ex:prop> "2.0.0"^^xsd:string .
}
WHERE { } would fail with the error message: :string fell through a ecase form. This is fixed.
Bug26273 - Do not skip index when rewriting left-joins to joins
In some cases the system would skip the index component of a plan step when constructing a JOIN plan step. This was breaking the assumption that index is non-nil when removing equivalent variables from JOINs. This is now fixed.
Bug26272 - Count DISTINCT of UNION incorrect
It was possible for queries with COUNT DISTINCT of a UNION to give a number that was too high. For example:
SELECT (COUNT (DISTINCT ?s) as ?s)
{
{ ?s <ex:p1> <ex:o1> }
union
{ ?s <ex:p2> <ex:o2> }
} This is now fixed.
Bug26263 - Type errors when using UNION and FILTER
This is an issue relevant to queries containing UNION in which an unprojected variable occurs both inside the UNION (at most once per UNION subclause) and once outside it, like ?o in:
SELECT DISTINCT ?s {
{ ?s <ex:p1> ?o } UNION { ?s <ex:p2> ?o }
FILTER (?o > 3)
} In such cases it was possible for ?o to become unbound in the second occurrence (in the FILTER clause) even though the first UNION clause would bind it. This would lead to incorrect query results.
This is now fixed.
Bug26259 - Property path error: "the value of p is "
Certain property path expression would fail with the error:
the value of db.agraph.sbqe.property-path::p is
#<db.agraph.utility.dictionary:dictionary ...>
which is not of type (or null upi real). This is fixed.
Bug26240 - SPARQL property path that joins bindings misses results
It was possible for property path expressions to miss certain results if the subject or object of the property path was bound by a previous pattern. This is now fixed.
Bug26236 - Errors when using multi-match clause with multiple vars as filter-pattern
There were situations where a query containing a "VALUES" or "FILTER (.. IN ..)" clause, in combination with other patterns, would give incorrect query results. This has been fixed.
Bug26235 - Assertion error :is-ordered-cursor-for-distinct-variable
It was possible for queries with DISTINCT to fail with the error:
the assertion
(null (getf (db.agraph.cursor::cursor-properties db.agraph.sbqe::cursor)
:is-ordered-cursor-for-distinct-variable))
failed. This has been fixed.
Bug26222 - SPARQL FILTER range errs if there are objects with non-overlapping type
If a SPARQL query included a numerical filter, in the presence of triples whose object value is of a numerical type that does not overlap with that range, this would signal an error at query plan time.
For example:
SELECT * {
?s ?p ?o .
FILTER (?o > 3 && ?o < 10)
} This range (3, 10) of positive integers does not overlap with type xsd:negativeInteger. If the repository contain a triple like:
{ <ex:s> <ex:p> '-123'^^xsd:negativeInteger } the resulting error would be:
Error: `3` is not of type xsd:negativeInteger, exceeds the maximum: -1
[condition type: xsd-datatype-error] This has been fixed
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Prolog
No significant changes.
Documentation
No significant changes.
Distributed AllegroGraph
Bug26220 - Fedshard failure for "count distinct" with "group by"
When using FedShard, running a query that combines COUNT DISTINCT with GROUP BY could result in an error. For example:
SELECT ?s (COUNT(DISTINCT ?o) AS ?count) {
?s ?p ?o
} GROUP BY ?s This has been fixed.
AllegroGraph 7.0.1
AllegroGraph Server: General Changes
Rfe16481 - Increase triggered checkpoint timeout
Previously while adding a replica of a large repo on a large memory machine an error like the following could occur:
Error: instance add failed with code 400, body "failed with message: Could not trigger a checkpoint: Checkpointer process failed to respond"
This has been fixed.
Bug26176 - SHACL validation would consider shape graph triples
Under certain circumstances, SHACL validation would incorrectly consider shape graph triples for validation, resulting in reporting triples non-conforming when the data graph triples actually conformed. (Reporting triples as conforming when in fact they did not conform never occurred.) This has been fixed. SHACL and SHACL validation are described in the SHACL document.
HTTP Client
No significant changes.
SPARQL
Rfe16452 - Let GROUP BY with no result rows give zero result rows
The results of GROUP BY queries with aggregates in an edge case has been changed to closer follow the intention of the SPARQL 1.1 standard.
Take this query and assume there are no triples with predicate
select ?s (count(?o) AS ?count)
where {
?s <ex:p> ?o
} group by ?s AllegroGraph would return one result row with ?s unbound, and count zero.
An alternative behaviour that some other triple stores have, is to return zero result rows. The text of the SPARQL 1.1 standard conflicts with the "agg-empty-group2" test case, so that either behaviour is reasonable.
The current consensus is that having zero result rows was intended, and is expected by users. See e.g. https://afs.github.io/sparql-agg-group-empty.html#group-agg-no-rows
AllegroGraph has been updated to follow this consensus. So for GROUP BY queries with aggregates, in case of zero matching pattern in the body, there will be zero result rows.
Bug26190: Incorrect rewrite of OPTIONAL with non-patterns on LHS
In some cases queries of the form:
SELECT .. {
VALUES ..
..
OPTIONAL { .. }
} could be optimized incorrectly, resulting in missing query results. This has been fixed.
Bug26188 - Executing query failed for query with VALUES
Certain queries of the form:
SELECT .. {
VALUES ?x { <a> <b> <c> }
.
} could lead to an error:
Error: the value of ag.sbqe::index is nil,
which is not of type ag.sbqe::plan-index-type. this has been fixed.
Bug26181 - Incorrect :time-total addenda value for query on remote store
When executing a SPARQL query in AGWebView the Information tab shows query execution statistics, including the time it took to plan, execute, and output query results.
It could happen that the total time would be incorrect (too low). This has been fixed.
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Prolog
No significant changes.
Documentation
No significant changes.
Distributed AllegroGraph
Rfe16457 - improve shard-local kb support
Previously, when defining a distributed store in the cluster config file, there was limited support for shard-local kb's by including localhost in the KB declaration. That is the following declaration would define a kb on localhost and would inherit whatever scheme/port were the default at the point where the declaration was made:
KB localhost/catalogs/tests/repositories/kb4 If these defaults match the scheme/port used by each AG shard, then the KB will be shard-local. Now, any triple-store-spec that does not include a server part can be used to define a shard-local KB. Thus
KB tests:kb4 no longer will inherit defaults from the environment at the point of the declaration, and will inherit the scheme/port of each shard, even if scheme/port differ across all shards.
See the Distributed Repositories Using Shards and Federation Setup document.
Bug26187 - Do not track servers in KB directives
In Distributed AllegroGraph, the servers that comprise a cluster are derived from the directives that refer to servers in the agcluster.cfg file. This includes server, repo, replica, and --incorrectly-- kb directives.
The servers referred to only in KB directives are no longer considered as part of the cluster. This allows remote KBs which do not reside on a cluster host.
AllegroGraph 7.0.0
AllegroGraph Server: General Changes
agtool load changes
The (undocumented) --blank-node-cache-size argument has been removed. (Even though this argument was undocumented, some users may have been told of its existence.) Blank nodes are no longer cached during data loading.
There is a new -w/--workspace option to agtool load. The value should be an existing directory which will be used for temporary files during data loading, particularly for the temporary storage of a table of blank node information. Absent this option, temporary data is stored in /tmp and if that directory is too small, the load will fail.
agtool now supports a triple-count tool
The triple-count tool returns the number of triples in a repository. See here in the agtool document.
agraph-control now takes a --cluster-config control option
The option allows the path of the cluster configuration file to be specified. This file defines distributed repositories (see Distributed Repositories Setup). The agraph-control program is described in the agraph-control of the Server Configuration and Control document.
Adding a triple with a predicate mapping and a datetype mapping might add two triples
Before this change, when adding a triple, if both a predicate mapping and a datatype mapping applied to the object of the triple, two triples would be added. The new defined behavior is that predicate mappings take precedence over datatype mappings.
index-literals argument passed when creating/modifying a freetext index
When creating a freetext index, the index-literals argument determines if and/or under what circumstances a literal will be indexed. The value passed can be either a boolean or a list containing UPIs, future-parts, or strings.
With this change, when passing a string to index-literals, it is expected to be equivalent to the value returned when calling part->value. That is, the string should not contain any embedded angle-brackets (< or >) or double-quotes (") used to denote the string represents a resource or literal, that are not part of the actual value represented by the string.
Rfe16417 - Deprecate style 2 indices
There are two triple index styles available, "style 1" and "style 2", with different storage structures and thus performance characteristics. In practice "style 1" was generally recommended, and "style 2" has been deprecated. The latter has been removed from the documentation and might be removed in a future release.
Rfe16386 - New BriefBacktrace configuration directive.
See the [administrator note](#admin-notes-700} for more information. See Server Configuration and Control for information on configuration directives.
Rfe16364 - New LicenseWarnInAdvance configuration directive.
See the administrator note for more information. See Server Configuration and Control for information on configuration directives.
Rfe16283 - New configuration directive to control the stale DNS retain time
There is a new configuration directive StaleDNSEntryRetainTime which specifies how long AllegroGraph will retain a stale DNS entry. If the DNS mapping from name to IP address may change while the application is running you'll want to specify a very small value (number of seconds) for this directive. See StaleDNSEntryRetainTime in Server Configuration and Control.
Rfe16244 - Update PARSE-NTRIPLES to support N-Triples 1.1 blank node syntax
When importing N-Triple data, certain blank node formats with dots, dashes or underscores were unsupported. For example importing this line:
<http://s> <http://p> _:a-b_c.d . would fail with:
Error: N-Triples parser error while parsing string at line 1
(last character was #\_): Illegal character Now AllegroGraph supports the blank node syntax as defined in the N-Triples 1.1 technical recommendation.
Rfe16239 - agtool load/export/archive from/to s3
agtool can access files in Amazon S3. To do so, AWS authentication must be specified on the agtool command line. See the section Accessing and operating on files on Amazon S3 in the agtool document. Files in S3 must be prefaced by s3://, like the following:
s3://bucketname/a/b/c/filename Rfe14667 - Log THP warning only (don't show in AGWV)
The THP (Transparent Hugepages) warning is now logged to agraph.log only and not the console or AGWebView.
Bug26040 - agtool archive restore to a different repo.
The documentation printed by 'agtool archive restore --help' claimed that if the archive directory contained only one repo backup then you needn't name that backup if you restored to a different repo name. This claim has been removed from the documentation because the behavior isn't supported.
Bug26017 - Literal language in rdf/xml was not properly loaded
Previously, instead of setting the language of the literal object, an additional triple with the object being the language value and the predicate http://www.w3.org/XML/1998/namespacelang was incorrectly added to the triple store.
For example, <rdfs:label xml:lang="fr">Autriche</rdfs:label> loaded as:
<subj> <pred> "Autriche"
<subj> <http://www.w3.org/XML/1998/namespacelang> "fr" Bug26011 - Allegrograph licensed cores check prints wrong core count in error message
Previously if the number of CPU cores exceeded the licensed limit, AllegroGraph would generate an error message like the following:
This machine exceeds the CPU core count specified in the license. Maximum licensed cores: X. This machine has: Y.
The value of X would be the license limit but the value of Y would potentially be incorrect (it would report the number of available threads, rather than the number of cores in the machine, which will be different values if hyperthreading is enabled). This has been fixed.
Bug26000 - Fti recovery crashes if stray fti files exist from a prior instance crash
In some circumstances instance recovery could fail with an error message like the following:
opening of database myrepo has failed (The database system tried to
allocate a new file with the logical name fti-1-chunk-1 which already
exists in database myrepo) This has been fixed.
Bug25999 - Crashed instance process can leave service daemon in confused state
Under some circumstances a crashed instance process would not be automatically restarted by the service daemon. This has been fixed.
Bug25986/Bug25997 - Fix agraph-backup tool initialization issues
When the repo spec was used to specify the repository, the credentials and host values from the spec were ignored and the tool always attempted to connect to 127.0.0.1. This commend thus
$ agtool archive backup test:xyzzy@remotehost:10035/test test.bak could fail with a Connection refused (errno 111) message because the host is not localhost. Specifying the credentials when restoring data also could cause the tool to fail (bug25997):
# Fails with the message 'Invalid arguments (:params,
# :restore-mode-p) found while creating triple-store of class
# db.agraph:remote-triple-store. ...'
$ agtool archive restore-all test.bak test:[email protected]:10035 Both these issues have been fixed. Note that using deprecated arguments like --host and --port could cause problems if the values specified did not match the values in the REPO-SPEC, even default (and thus not-visible) values in some cases. See Older repository specification arguments in the agtool General Command Utility document for examples.
Bug25959 - Fix ACID isolation violation in some ordered cursor operations.
Fixes a bug introduced in v6.5.0 where some ordered triple cursors could produce triples that were not visible in the current transaction. The scope of the problem was limited to triples committed in concurrent transactions that had yet to be written to disk.
Bug25907 - Document how to pass args to agraph using agraph-control
The ability to use agraph-control to pass arguments to the agraph program (which is called by agraph-control) when starting or restarting AllegroGraph existed prior to this change, but was not clearly documented. How to do so is now documented in the agraph-control --help text and in the description of agraph-control in the Server Configuration and Control document.
Bug25901 - Importing triples bypassed security-filters
It was possible for users with security-filters to import triples into a db that those filters were intended to prevent.
This problem is now fixed.
Bug25880 - Remove whitespace trimming in TRIX literals parsing
To restore the behavior specified by the standard, removed whitespace trimming in the TRIX parser while parsing literals.
HTTP API
No significant changes.
SPARQL
Rfe16433 - Remove full-scan-warning
The query warning type "warn-fullscan" and the query option "fullScanWarningSize" have been removed.
In practice these warnings could occur without there being a problem with the query. It would fill agraph.log with warning lines that would be ignored, defeating the purpose.
Usage of the franzOption_fullScanWarningSize query prefix will now lead to a warn-obsolete-query-option entry in the query result metadata:
(run-sparql "PREFIX franzOption_fullScanWarningSize: <franz:210000000>
select ...") has fourth return value:
#<done
[warn] obsolete-query-option: fullScanWarningSize
[time] ...
...>
Rfe16324 - Warn on ASK modifiers
An ASK query does not support LIMIT, OFFSET, or ORDER BY. Now a query warning is signaled if any of these is provided, with the exception of LIMIT 1.
Rfe16279 - Improve checking for unknown constants in the query
The check for unknown constants in the query at plan time is now more advanced, resulting in faster query execution.
For example if a resource <s> only occurs in the subject position of triples, a pattern like ?s <p> <s> will now be marked impossible at plan time.
Another example are references to standard IRIs that are present in every repository automatically, like <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>. Previously a pattern like ?s a ?foo would never be marked impossible, but now if there are no triples with such predicate, the pattern will be marked impossible.
This also has effect on magic properties like <http://franz.com/ns/allegrograph/3.0/temporal/intervalBefore> that under the hood expand into patterns with properties like <t:starttime>, <t:startpoint> and <t:time> (with http://franz.com/ns/allegrograph/3.0/temporal/ as t). If a repository only uses starttime, and never the startpoint/time combination, the plan will be optimized accordingly.
Bug26144 - CAAT ignores :pattern-filters of query patterns
Queries executed with the Chunk-at-at-Time (CAAT) strategy (so with a LIMIT, without DISTINCT or aggregators) could give incorrect results, if a specific query pattern was planned to act as filter for another pattern. This has been fixed.
Bug26143 - Graph <g> {} gives error
A query with an empty named graph clause, like:
select ?s ?o {
graph <ex:g> {}
?s ?p ?o .
} would fail with an error: Attempt to access the name field of {g} which is not a symbol. This is now fixed.
The meaning of graph <ex:g> {} as a pattern, is that some triple must exist in the given fixed graph <ex:g>. The meaning of graph ?g {} is that some triple must exist in any named graph.
Bug26137 - Spurious unknown-variables warning for VALUES in inner SELECT
A VALUES clause inside an inner SELECT, like in this query:
SELECT ?region {
SELECT ?region WHERE {
VALUES ?type { <ex:Region> }
?region a ?type .
}
} would give a spurious warning about the VALUES variable:
unknown-variables: Unknown variable used in VALUES clause: ?type_1_1 This is now fixed, so that there will be no more warnings.
Bug26116 - FILTER CONTAINS LCASE gives UPI-NOT-IN-STRING-TABLE-ERROR
Certain SPARQL expressions that create intermediate strings, like:
select ?s {
?s <ex:p> ?o .
filter ( contains(lcase(?o), 'something'))
} could signal the error: The UPI #(...)is not present in the repository's string table.
This has been fixed.
Bug26093 - GROUP_CONCAT interprets separator arg as format string
If a SEPARATOR argument containing a tilde was supplied to the GROUP_CONCAT aggregator, the resulting string could be incorrect. This has been fixed.
Bug26090 - Boolean value serialized as sparql-json lacks datatype
A boolean value serialized in sparql-json format would look like (here we use the Lisp function sparql:run-sparql but the results would be the same if the query was executed in AGWebView and exported in sparql-json format):
(sparql:run-sparql "select * { bind(true as ?x) }"
:results-format :sparql-json)
-> {"head": {"vars": ["x"]},
"results": {"bindings": [{"x": {"type": "literal", "value": "true"}}]}} But the correct way to serialize a boolean is to include the datatype. Thus this:
"x": {"type": "literal", "value": "true"}` is incorrect and should instead be:
"x": {"type": "literal",
"datatype": "http://www.w3.org/2001/XMLSchema#boolean",
"value": "true"} This has been corrected.
Bug26078 - SPARQL FILTER on numeric range excludes UPI types that partially overlap
If a SPARQL query FILTER expression restricts a triple part to a numeric range, and if there is a triple whose part value falls in that range but is of a data type such that the filter range exceeds the data type range, then the triple would not pass the filter, which was incorrect.
For example:
(add-triple "<ex:s>" "<ex:p>" (value->upi 127 :byte)) and query:
select * {
?s ?p ?o .
filter ((?o > 120) && (?o < 130)
} The query FILTER restricts the triple object to (120, 130). The ?o value 127 should pass this filter. The value 127 is of type :byte, which has range [-128, 127].
The incorrect behaviour is that the FILTER range (up to 130) exceeds the data type range (up to 127) and therefore the value 127 was incorrectly filtered out, meaning the triple would not be returned by this query.
This has been fixed.
Bug26071 - COUNT aggregate on unknown variable should return 0.
If a query aggregates a variable that is not otherwise used, like:
SELECT (COUNT(?bogus) AS ?count) { ?s ?p ?o } this would be evaluated as if the unknown variable ?bogus was replaced by true. The value of ?count would be the total triple count.
This has been corrected, so that the returned ?count is now zero.
Bug26020 - Invalid OPTIONAL reordering
If a query has an OPTIONAL pattern that binds a variable, followed by a non-OPTIONAL pattern that references that variable, like:
SELECT * {
?s <ex:p1> ?o . # (1)
OPTIONAL {
?t <ex:p2> ?o2 . # (2) possibly binds ?o2
}
?s <ex:p3> ?o2 # (3) references (if bound already) or binds ?o2
} the query engine could create an incorrect query plan with evaluation order (1) (3) (2). The evaluation order of (2) and (3) would be reversed: variable ?o2 would be bound based on pattern (3). This is now corrected to have evaluation order (1) (2) (3).
Bug26005 - Incorrect SUM and GROUP_CONCAT default values for empty group
If a SPARQL query has no result rows (or if GROUP BY is used, and a specific group has no result rows) a SUM aggregation and a GROUP_CONCAT aggregation would both return an unbound value.
This was incorrect according to the SPARQL 1.1 spec. Now SUM will return 0, and GROUP_CONCAT will return the empty string.
Bug26004 - Queries that have a constantly true HAVING could fail.
A constantly true HAVING expression in a query, like:
SELECT ?s (max(?o) as ?maxO) {
?s ?p ?o
} GROUP BY ?s
HAVING (3 < 4) could err with Received signal number 7 (Bus error). This has been fixed.
Bug25999 - Crashed instance process can leave service daemon in confused state
Under some circumstances a crashed instance process would not be automatically restarted by the service daemon. This has been fixed.
Bug25988 - SPARQL function REGEX should err on non-string
Previously AllegroGraph was liberal in accepting a constant resource as REGEX pattern. For example:
SELECT (regex('ex:foo', <ex:foo>) as ?match) {} would return "true".
This has been fixed in order to adhere to the SPARQL specification: if the target or pattern is not a string, a type-error is signaled. The above SELECT now returns an unbound value.
Bug25984 - SPARQL function LANGMATCHES should err on non-string
Previously AllegroGraph was liberal in accepting non-strings to function LANGMATCHES and converting them using STR.
For example:
BASE <http://foo/>
SELECT (langMatches(<ex>, <ex>) as ?a)
(langMatches(123, 123) as ?b) {} was interpreted as:
BASE <http://foo/>
SELECT (langMatches( STR(<ex>), STR(<ex>)) as ?a)
(langMatches( STR(123), STR(123)) as ?b) {} and would return "true" twice.
This has been fixed in order to adhere to the SPARQL specification: if either argument to langMatches() is not a string, the result is a type-error. The above SELECT now returns two unbound values.
Bug25982: Fix HAVING constraint on implicit single group without aggregates
If a query uses HAVING without using GROUP BY, and no aggregates (like MIN, COUNT), as for example in:
SELECT * { ?s ?p ?o } HAVING (?s = <ex:s>) then the HAVING clause acts like FILTER:
SELECT ?s ?p ?o { ?s ?p ?o . FILTER (?s = <ex:s>) } However in the first query AllegroGraph would incorrectly ignore the HAVING clause, meaning results that should have been filtered out were returned. This is now corrected.
Bug25941 - Blank node patterns not supported in subject position of BGP triples
A query pattern with a blank node pattern as subject, and a property path, like:
select * {
[ <ex:p2> ?o2 ] ( <ex:p1> | <ex:p2> ) ?o
} would give a parse error at the "(" character. This has been fixed.
Bug25936 - Inefficient join with UNION
The query execution planning for certain queries containing UNION has been improved, leading to faster execution. This impacts also magic properties that are implemented using UNION, like <t:intervalBefore>.
Bug25934 - Problem with BIND (EXISTS { pattern1, pattern2 } as ?x)
If a query binds a variable to an EXISTS expression, where the EXISTS contains multiple patterns, like:
SELECT ?person ?known where {
?person a <ex:Person> .
BIND ( EXISTS {
?employee a <ex:Employee> ;
<ex:knows> ?person
} as ?known )
} then the outcome could be incorrect: the second and later patterns could be ignored, or the query could result in an error like:
Error: Bad variables for set: (?person). This is now fixed.
Bug25933 - Duplicate patterns in blank node + property path expansion
For query triple patterns with a property path, and a blank node as object, like:
?s ^<ex:p> [ <ex:p1> <ex:foo> ;
<ex:p2> <ex:bar> ] the expansion into simple triple patterns could contain duplicates:
1. ?bnode_1 ex:p ?s
2. ?bnode_1 ex:p ?s (duplicate)
3. ?bnode_1 ex:p2 ex:bar
4. ?bnode_1 ex:p1 ex:foo The duplicated pattern resulted in superfluous work during query execution. This is now fixed.
Bug25931 - Sending SPARQL query with typed booleans to remote-triple-stores fails
Making SPARQL queries with typed booleans (e.g. true"^^xsd:boolean) would fail with an error indicating that AllegroGraph is unable to convert a boolean UPI into a future part.
This is now fixed
Bug25925 - SPARQL MAX does not handle unbound values correctly
The SPARQL MAX aggregator would return an unbound value as result, if any of the values to aggregate was unbound. This has been fixed.
Bug25920 - SPARQL SERVICE using https without chunking errs
When a SPARQL query was sent to remote SPARQL endpoint that uses HTTPS, and the endpoint sent results not using chunked encoding, then an error would be signaled:
Error: the value of stream is #<excl::ssl-client-stream ..>,
which is not of type
(or net.aserve::unchunking-stream util.zip:inflate-stream
excl::hiper-socket-stream).
[condition type: type-error] This could happen in a SERVICE <https://...> clause, or by proving a :HOST "https://.." argument to function run-sparql.
This has been fixed.
Bug25919 - SPARQL function STRLEN should err on non-string
Previously AllegroGraph was liberal in accepting non-strings to function STRLEN and converting them using STR.
For example:
SELECT (STRLEN (<http://franz.com>) as ?s)
(STRLEN (123.45) as ?t) {} was interpreted as:
SELECT (STRLEN (STR (<http://franz.com>)) as ?s)
(STRLEN (STR(123.45)) as ?t) {} and would return values 16 and 6.
This has been fixed in order to adhere to the SPARQL specification: taking the length of a non-string now gives a type-error. The above SELECT now returns unbound values.
Bug25916 - COUNT DISTINCT of expressions gives error
A query for COUNT DISTINCT where the value is not a variable (like ?o) but an expression (like "strlen(?o)"), for example:
select (count(distinct(strlen(?o))) as ?olen) {
?s ?p ?o
} used to signal an error:
Received signal number 11 (Segmentation fault) These queries are now supported.
Bug25898 - SPARQL error for property multimatch with filtered object range
Queries with a property path with alternatives, that have a filter on the object variable, where the object values are of a type that is hashed to a UPI, like:
select * {
?s ( <ex:p1> | <ex:p2> ) ?o .
filter (lang(?o) = 'en')
} could result in an error like:
Error: The UPI #(0 0 0 0 0 0 0 0 0 0 0 3) (which is a literal-language)
is not present in the triple-store's string dictionary. This has been fixed.
Bug25821 - SPARQL function SUBSTR should clip LENGTH to string length
If in SUBSTR the length argument is larger than then the string's length, this would result in an error:
SELECT * { bind (substr('abc', 1, 100) as ?s) }
Executing query failed:
In subseq, `end' (100) is beyond the end of the sequence (3). This is now fixed, so that the substring up to the end of the string is returned. In this case: abc.
Bug25806 - Fix ORDER BY on non-projected variable
For certain SPARQL SELECT queries that use ORDER BY on a non-projected variable, the results could be ordered incorrectly. This has been fixed.
Bug25694 - Fix simplifying SPARQL= into UPI=
It was possible for a SPARQL equality filter:
FILTER (?a = ?b) to return false if the values are of different types. For example, if ?a is the integer 2 and ?b the float 2.0. This has been fixed.
Bug25636 - Triple cursor FILTER containing [NOT] EXISTS applied incorrectly
If a query contains a FILTER clause that uses EXISTS or NOT EXISTS, there were cases where that FILTER was evaluated incorrectly and as a result the query would return incorrect results.
Concretely, if in the query plan the filter was integrated in the first triple pattern, it was possible for EXISTS to return true in certain cases where false was the correct answer; and for NOT EXISTS to return false where true was the correct answer.
This has been fixed.
Bug25585 - Support ORDER BY on an aggregate expression
Queries in which the ORDER BY expression refer to an aggregation, for example:
SELECT ?s (MIN(?o) as ?minO) {
?s ?p ?o
}
GROUP BY ?s
ORDER BY (min(?o)) would fail with the error:
:aggregate fell through a etypecase form This is now fixed.
Bug24815 - Fix MINUS with an empty row on the RHS
A query that uses MINUS, where the second BGP contains only unbound values, like:
SELECT ?x {
{ VALUES (?x) { (1) (2) (3) (4) } }
MINUS
{ VALUES (?x) { (UNDEF) }
}
} should have 4 results, namely ?x bound to each integer.
Previously AllegroGraph would return no bindings. This is now corrected.
AGWebView
Bug26010 - Fixed Delete statement confirmation dialog
The Delete statement confirmation dialog in AGWebView is now fixed. Instead of asking for the confirmation, if showed the message: function () { [native code] } and did not allow to confirm the operation. Now, the confirmation dialog is displayed properly and, if the operation is confirmed, it will complete correctly.
Bug25947 - AGWebView doesn't allow all session spec characters
In 6.6.0 AGWebView was too restrictive on the characters allowed in the Session Specification edit box on the main catalog page. This has been fixed.
Bug25856 - Back-up this repository on WebView repo page does not work
Fixed the inner API for remote backup request with target on the server side file system. Fixed the dialog to correctly describe what is expected from user.
Changes to the Lisp API
Improve Materializer support for remote-triple-stores
The keyword arguments :delete-first? and :delete-duplicates-first? are now supported when calling materialize-entailed-triples on a remote triple store via the AllegroGraph Lisp client, and can also be specified via the REST API as well.
Rfe16211 - add IF-DOES-NOT-EXIST parameter to OPEN-TRIPLE-STORE
This change extends the open-triple-store function with an :if-does-not-exist argument. The following values are supported:
:error- throw an error (default);:create- create new repository.
Rfe12356 - HTTP proxy support for AllegroGraph Lisp client
New :proxy parameter for create-triple-store, open-triple-store and triple-store-exists-p functions of the AllegroGraph Lisp client API allow to specify an HTTP proxy to be used for all HTTP requests issued by client. Proxy settings can be specified in the form
[<user>[:<password>]@]<host>[<port>] Alternatively, components of the proxy specification can be provided explicitly by :proxy-user, :proxy-password :proxy-host :proxy-port parameters. Additionally, the list of domain name suffix strings, for which the proxy should be bypassed, can be specified via :proxy-bypass-list parameter.
Bug25974 - load-ntriples doesn't work with namestring source on remote-triple-store
Previously calling load-ntriples (or any of the other load-* functions) would fail for remote-triple-stores if the source argument was a string naming a file instead of a pathname. This has been fixed.
Prolog
No significant changes.
Documentation
rfe16275 - Storage and memory document for AllegroGraph
There is a new Memory Usage document which discusses memory and disk space requirements when using AllegroGraph.
nD Geospatial
No significant changes.
AllegroGraph 6.6.0
AllegroGraph Server: General Changes
Rfe16127 - Make AllegroGraph's scripting model permissions more strict
Superuser permission is now required to upload, list, view, or delete both sitescripts and reposcripts.
Additionally, reposcripts can no longer be loaded into shared backends.
Rfe16096 - Simplify AllegroGraph script naming.
Scripts no longer support hierarchical naming, in favor of a simple flat namespace. For example, when uploading a script:
Legal: http://example.com:10035/scripts/sc.cl
Illegal: http://example.com:10035/scripts/com/app/module/sc.cl Rfe16093 - Incoming IP restrictions mechanism for AllegroGraph
New AllowIP configuration directive provides a way to restrict access to AG instance to certain ranges of IP addresses.
Rfe15992 - Add MaximumBackupAge parameter to the agraph.cfg file
One can specify the maximum age in seconds that a backup of the multi-master controlling instance is considered acceptable to use to grow the cluster. See the Instance Settings section of the Multi-master Replication document and also the Top-level directives for multi-master replication clusters section of the Server Configuration and Control document).
Rfe15886, Rfe15470 - UI Improvements to the triple attribute pages.
Merges the 'Define Attribute' and 'Define Static Filter' pages into a single 'Triple Attributes' page that is more streamlined and user friendly. See the Attributes example in the AGWebView documentation.
Bug25833 - Using CorsAllowRegex would break session/backend creation
Setting the CorsAllowRegex directive (see Server Configuration and Control) in your agraph.cfg would cause the creation of dedicated sessions and shared backends to break with a confusing error message in the agraph.log file.
Bug25832 - Only compare CorsAllowOrigin directive against Origin host
Previously, each value specified by a CorsAllowOrigin directive was matched against the full value of the Origin header: scheme, host, and port. Now, each CorsAllowOrigin directive is matched only against the host, per the documentation.
Bug25812 - agtool query --standard-namespaces
If agtool query is given the argument '--standard-namespaces no' then the AllegroGraph set of standard namespace PREFIX definitions will not be prepended to the text of query. Furthermore if the query is going to an AllegroGraph repository that agtool will open itself then the set of namespaces known to AllegroGraph is cleared before the query is run.
Bug25805 - Fix bug in SNA path-finding functions used with remote-triple-stores.
SNA path-finding functions like depth-first-search and all-bidirectional-search-paths would fail when called on a remote-triple-store if either the START or END arguments represented literals. This is now fixed.
Bug25801 - CSV import hangs on surrogate pairs
The CSV importer has been fixed to work with UTF8 surrogate pairs.
Bug25784 - Agtool archive failed when using an https spec.
Trying to use agtool archive with an https spec would fail due to the tool attempting to make regular http requests to the ssl port. This problem is now fixed.
Bug25736 - improved handling of GRAPH in JSON-LD
A JSON-LD error is now signaled when a string which is neither a resource nor URL is passed to as the --graph argument to agtool load. :root is supported as a special value of GRAPH, resulting in using the toplevel subject as a default graph for all triples added for the JSON-LD object with that subject.
Bug25382 - Fix security flaw in AllegroGraph scripting.
Fixes a bug, introduced in v6.3.0, that allowed any file on a filesystem accessible to AllegroGraph to be loaded as a script. Scripts are now only searched for and loaded from their expected locations relative to the SettingsDir directive.
HTTP Client
Bug25820 - CORS does not work via HTTPS
Previously CORS configuration options were ignored for accesses to AllegroGraph made via HTTPS. This has been corrected.
SPARQL
Rfe16157 - Implement logging of slow queries
A new configuration directive SlowQueryLogThreshold, if set, causes SPARQL queries that have total run time in milliseconds greater or equal to the value, to be logged to agraph.log or to a log file specified by SlowQueryLogFile configuration directive.
Bug25831 - Fix optimization of GRAPH with a non-existant value in SPARQL
The query:
select ?s ?p ?o
{
VALUES ?s { <http://s> }
VALUES ?p { <http://p> }
GRAPH <http://non-existing> {
OPTIONAL {
?s ?p ?o
}
}
} should return 1 result row with ?s, ?p bound and ?o unbound.
However due to an incorrect optimization of unreachable GRAPH clauses there would be no results. This has been fixed.
Bug25803 - Fix augmenting pattern with two IRI equivalence constraints
SPARQL queries that bind a variable multiple times, like:
SELECT * {
BIND (<http://ex/p2> as ?p)
{
BIND (<http://ex/p1> as ?p)
?s ?p ?o
}
} could be optimized incorrectly, leading to the error message:
Executing query failed:
nil is not of the expected type `number' This has been fixed.
Bug25802 - Join query rewriters lose filters
Certain SPARQL queries where a join happens, and one side contains BIND and FILTER, like:
SELECT ... {
?s ?p1 ?o1
{
BIND (<http://p2> as ?p2)
?s ?p2 ?o2
FILTER (.. ?o2 ..)
}
} could be optimized wrongly in that the FILTER would be dropped. This has been fixed.
Bug25794 - BIND variables disappear in join
Certain SPARQL queries where a variable was used in a filter and also projected could return results in which that variable was unbound due to an incorrect query optimization. For example:
SELECT * {
{
bind (4 as ?four)
?s ?p ?o .
filter (?o != ?four)
}
{
bind (3 as ?three)
?s ?p ?o .
filter (?o != ?three)
}
} could return rows with ?three and ?four unbound. This has been fixed.
Bug25761 - Whitespace symbols are not escaped properly when writing in TSV format
According to the TSV SPARQL standard, when serializing RDF terms some whitespace symbols like tabs, newlines and carriage returns are encoded in strings as \t, \n and \r respectively. This has not been the case, as these symbols were serialized as-is, leading to some parsing problems later, specifically a tab character being treated as separator for TSV format. This has been corrected.
Bug25756 - Unused variable in complex property path gives errors
Queries of the form:
select ?o {
?s ( ^<http://foo> | <http://bar> ) ?o
} in which the property path was not very simple, and also one variable is not otherwise used (here ?s), could lead to an error during execution. This has been fixed.
Bug25752 - Fix missing results for DISTINCT + LIMIT
There could be missing results for queries that uses both DISTINCT and LIMIT due to the DISTINCT condition being applied incorrectly. This has been fixed.
Bug25749 - Filter (str(?var) = '..') marked as impossible
If a SPARQL query had filter pattern comparing a STR() value to a literal string, like:
filter ('http://g' = str(?g)) this would result in no results and the warning:
Warning: `?g` cannot be a literal so the FILTER ?g is one of 1 thing will always fail This has been fixed.
Bug25714 - SPARQL IRI() and URI() functions must return valid, absolute results
Previously, the SPARQL IRI() and URI() functions could return IRIs and URIs that were syntactically invalid and/or relative. This has been corrected. AllegroGraph will now signal SPARQL type errors which will be treated as UNBOUND for assignment and as FALSE for FILTER evaluation.
Bug22350 - Fix missing results when using CAAT with DISTINCT
When a SPARQL query containing DISTINCT or REDUCED was executed using the chunk-at-a-time strategy, there could be results missing due to incorrect handling of intermediate query results. This has been fixed.
AGWebView
Rfe16012 - Don't show AllegroGraph Utilities menu unless logged in.
When not logged in, AGWebview displayed the Utilities menu even though none of its menu items work without being logged in. Now, the menu is not presented until a user is logged in.
Changes to the Lisp API
Rfe16071 - Implement breadth-first-search-if functionality
The four breadth-first search SNA functions:
- breadth-first-search
- breadth-first-distance
- map-breadth-first-search-paths
- all-breadth-first-search-paths
have all been enhanced such that the end argument can be a function of one parameter. If end is such a function, then the search code will call it on each node found during the search and will record a successful search if the function returns true.
Bug25759 - Change load-trix :default-graph argument to :graph
To ensure consistency with other load- functions, :default-graph argument to load-trix has been deprecated and will be removed in the subsequent release. :graph should be used instead.
Bug25738 - AGWebView users are notified when query results may be stale
When reloading queries via the 'Saved' or 'Recent' menus--but also when executing new queries--AGWV will now report when there's a possibility the results displayed are out of date.
Prolog
No changes
Documentation
No changes
AllegroGraph 6.5.0
AllegroGraph Server: General Changes
Rfe15938 - Deprecate --tlog arg to agtool view-tlog
The --tlog argument to agtool view-tlog is deprecated. One need only specify the tlog filename to view.
Rfe15836 - Improve efficiency of dynamic reasoner on rdf:type queries
Increased the speed of the dynamic reasoner when looking for all <SUBJECT rdf:type OBJECT> triples.
Rfe15774 - Make agtools use repo and server specs
All agtool commands are in the process of being modified to use a Repo Spec when referencing a repository and a Server Spec when referencing a server but not a repository. (Repo Specs also identify servers. Server Specs are used when a repository is not relevant to the command, such as with agtool user.) In this release, the following commands have been modified to accept Repo Specs or Server Specs:
- archive
- create-db
- export
- generate
- materialize
- query
- replicate
- upgrade
These changes are backward compatible. Previous options are supported but deprecated and may cause warnings to be signaled.
Rfe15748 - Add support for backing up a single remote repository
agtool archive now supports remote repository backup. This operation requires a user with AllegroGraph superuser privileges.
Rfe15745 - Check for no-commit mode before creating a new text index
Creating a new text index requires indexing all existing triples and preparing the new index configuration and datafiles for commit.
Previously creating a new text index on a no-commit repository (such as a warm standby replica) would perform the initial indexing and then fail to commit due to no-commit mode. Now the check for no-commit mode is made early to avoid doing work that will end up being discarded.
Rfe15695 - agtool load now reports the filename with buffer-size exceeded errors.
When the length of a line in an input file exceeds the buffer size an error is reported. Previously this error did not specify which file the error resulted from. This has been changed to report the filename.
Rfe15595 - Add AG directives to choose suppored SSL methods and SSL ciphers
It is now possible to specify SSL protocols and ciphers that will be accepted by AllegroGraph when using HTTPS. This can be achieved by using two new config directives - SSLProtocol and SSLCipherSuites. Configuration directives are described in the Server Configuration and Control document.
Rfe15517 - Change agtool load to use repository specifications
agtool load now accepts a repository specification (REPO_SPEC) argument when specifying a repository. A REPO_SPEC names the repository and also its catalog, and the server scheme, host, port, so all associated arguments which specified those values (--scheme, --host, --port, and --scheme) are deprecated.
Rfe15471 - Support For loading JSON and CSV data in agtool load.
agtool load can now load JSON and CSV files. See See the section Loading Raw Data in the agtool load document to learn more about the supported workflows and parameters.
Rfe14214 - Add remote store support to agload
Remote store support has been added to agtool load. It is no longer necessary to run agtool load on the same machine as the AllegroGraph server and as the same user. A username and password must be specified when loading into a remote repository, and that user must have write permission in the repository into which data will be loaded.
Rfe13797 - Reconsider triggering checkpoints on tlog rollover
Automatic checkpoints on tlog rollovers are now disabled.
Rfe10713 - Two phase commit (2PC) support
AllegroGraph can now participate in two phase commit (2PC) operations. See the Two-phase Commit document for details.
Bug25668 - Turtle parser can fail to use same blank node context for all triples in a parse
The turtle parser could fail to unify blank nodes with the same label that appears in RDF lists and outside of those lists. For example, the following data would load with two different nodes instead of one:
prefix : <ex://#>
:a :p ( _:x1 _:x1 _:x1 ) .
_:x1 :p 'object' . This has been corrected.
Bug25653 - Opening of a repl that should be running
A multi-master replication instance repo will remain open until it is explicitly stopped or for the period of time when the AllegroGraph server is not running. If an instance dies for some reason and a request comes in to open its repo then the repo now will open and rejoin the cluster. Previously an explicit start was required to start the instance and open the repo.
Bug25641 - Unhelpful error message from inability to translate a character
The error message that occurred when a data import was unable to translate a character was unhelpful. This has been corrected.
Bug25596 - RDFS++ reasoner does not follow sameAs for domain or range assertions
The dynamic RDFS++ reasoner was not inferring triples that could be derived from rdfs:domain and rdfs:range statements in addition to owl:sameAs statements. This has been corrected.
Bug25589 - Improve domain and range RDFS++ reasoning when subject not supplied
AllegroGraph's RDFS++ dynamic reasoner was failing to find all rdf:type inferences that used rdfs:range or rdfs:domain when the subject portion was not supplied. This has been corrected.
Bug25588 - Rdfs++ reasoner did not infer range/domain via subProperties
AllegroGraph's dynamic RDFS++ reasoner was not making rdf:type inferences when these required rdfs:domain and rdfs:range reasoning through chains of rdfs:subPropertyOf or owl:inverseOf linkages. This has been corrected.
Bug25586 - Error during recovery of partially-created text index
Previously it was possible for a crash to occur when restarting a repository instance if a partially-created text index existed during at the time of the last checkpoint. This has been fixed.
Bug25578 - xsd:integer casting operator does not work on xsd:longs
The xsd:integer() casting operator was not working on values of type xsd:long. This has been corrected.
Bug25574 - Bidirectional search was inefficient when there is no path
AllegroGraph bidirectional search functions did more work than needed when there was no path to be found between two different nodes. This has been corrected.
Bug25511 - Dynamic restriction reasoner requires subclassing unnecessarily
The dynamic RDFS++ restriction reasoning now recognizes owl:Restrictions that are simple rdfs:subClassOf owl:Restriction and also those that are of rdf:type owl:Restriction.
Bug25496 - agtool upgrade will detect a running repo and exit
agtool upgrade will check if a repo is running and if so print a message that an upgrade is not necessary and then exit with a no-error status of zero. (If a repo can be opened successfully, it does not require upgrading. Once a repo is opened in a version of AllegroGraph later than the version in which is was created, it should not be opened in the earlier version again.)
Bug25390 - Anyone can delete roles
A bug that allowed a user to delete roles without logging in has been fixed.
Bug25353 - Fix various issues when using `agtool export --parallel ...'
Fixes a few issues when exporting in parallel (using --parallel or a --workers argument greater-than 1), and using the default --blank-node-handling strategy of `together'
NQX export could be incomplete due to worker threads incorrectly detecting and skipping triples containing blank-nodes. Now, blank-part detection when exporting NQX works correctly.
Only N-Quads export would check for blank nodes in the graph slot, meaning for other quad formats, some blank nodes would not be properly separated into their own file. Now, NQX, TriX, and TriG formats also check the graph part when saving statements containing blank nodes to a separate file.
Bug25201 - agexport output depends on the value of LANG
It was possible for the value of the LANG environment variable to alter the output of agtool export. This has been corrected.
Bug25114 - Fix printing nquad default graphs
When exporting triples in NQUAD format from a federated or distributed triple store, there could be an unintended "default graph" marker like "default-graph1748246237" after the subject, predicate and object. This has been fixed.
Bug24954 - Fix exporting in NQX format of remote stores
Exporting a remote triple store using "agtool export" in "nqx" format:
agtool export -o nqx --user test:xyzzy ... (with both "-o nqx" and a value for "--user")
would result in the error:
http-triple-cursors cannot change whether or not they return
extended-triples after they have been created. This has been fixed.
Bug23424 - Rdfs++ reasoner does not infer subProperties of transitive properties
The dynamic RDFS++ reasoner was failing to infer some triples when they relied on following rdfs:subProperties of owl:transitive properites. This has been corrected.
Bug19476, Bug18483 - restriction reasoning improvements
AllegroGraph's dynamic restriction reasoning was missing some inferences when either the predicate portion of a query was wild or when they required additional rdfs:subClassOf inference on inferred triples. These have both been corrected.
HTTP API
The %D HTTP log directive change
The %D HTTP log directive (meaning response time in microseconds) will now use the high precision clock provided by the operating system rather than the one second resolution clock.
Rfe15876 - Allow specification of Accept-Type in the query parameters
In addition to supporting the normal HTTP Accept-type header, AllegroGraph now also allows an accept parameter to be included in the query portion of an HTTP SPARQL query request. This allows a client to specify the desired content type even when it does not control the headers that it sends. For example, if QUERY is the percent-encoded SPARQL query string, then this sample curl command would return the number of rows that the query produces:
curl "http://REPOSITORY_URL?query=QUERY&accept=text%2finteger"
Rfe15737 - Add config directive to set HTTP keep-alive timeout
Added a new config directive HTTPKeepAliveTimeout to control the HTTP keep alive timeout. See Server Configuration and Control for information on config directives.
Rfe15731 - Add user-agent header to outgoing HTTP requests
To avoid problems with network proxies that refuse HTTP requests without a User-Agent header, this header is now included in outgoing HTTP requests to remote SPARQL endpoints.
SPARQL
Rfe16032 - Add some list and set handling functions to SPARQL
AllegroGraph now supports three new SPARQL functions:
- http://franz.com/ns/allegrograph/6.5.0/fn#makeSPARQLList,
- http://franz.com/ns/allegrograph/6.5.0/fn#makeSPARQLSet,
- http://franz.com/ns/allegrograph/6.5.0/fn#lookupRdfList.
The first two take a list of arguments and associate then with a variable (gensym). The third looks up an RDF list in the current SPARQL dataset and associates the contents with a variable. In each case, this variable can then be used with other SPARQL magic properties and functions to retrieve and manipulate the list or set.
For example:
select ?member {
bind(<http://franz.com/ns/allegrograph/6.5.0/fn#makeSPARQLSet>(1, 2, 4, 1, 2) as ?set)
?member <http://franz.com/ns/allegrograph/4.11/sna/members> ?set .
} Would return 1, 2, and 4.
Rfe16007 - Evaluate the WHERE clause of SPARQL UPDATE queries in REDUCED mode
If a repository contains duplicate triples, then evaluating the WHERE clause of SPARQL update queries could become very memory intensive. Evaluating these in REDUCED mode ameiliorates this problem.
Rfe15967 - Eager filter evaluation could lead to missing query results
If a SPARQL query was supplied parameter values (e.g., via the :with-variables argument to the Lisp API's run-sparql command) and these values produced type-errors, then the FILTER analysis machinery could become confused and produce an incorrect filter. This has been corrected. As an example, the following query and parameters should produce one result but before this bug was fixed, it would produce no results:
select ?target {
optional {}
FILTER( ! bound(?target) || ! ( isLiteral(?target) && (lang(?target) = 'en')))
} parameters: ((?target . !<ex://test>)). The type-error occurs when the SPARQL lang() function tries to determine that language of the resource !<ex://test>.
Rfe15891 - Provide URI function in SPARQL to parse freetext queries
AllegroGraph now includes a URI function that can be used to see how freetext indexing queries will be parsed. For example:
prefix test: <http://franz.com/ns/allegrograph/6.5.0/>
select ?query ?expression {
values ?query {
'"soccer game"'
'soccer | f򴢯l'
'socc?r ping* '
}
bind(test:parseFreetextQuery( ?query ) as ?expression)
} would output:
---------------------------------------------------------------
| query | expression |
===============================================================
| "soccer game" | (:phrase "soccer game") |
| soccer | f򴢯l | (:or "soccer " "f򴢯l") |
| socc?r ping* | (:and (:match "socc?r") (:match "ping*")) |
--------------------------------------------------------------- which shows how each of the queries would be parsed for use by the fti:match magic property.
Rfe15890 - Enhance fti:match and fti:matchExpression to also bind predicates
AllegroGraph's built in Freetext Index magic properties can now also bind the predicate of any matching triples. For example, a query like:
(?subject ?text ?predicate) fti:match 'sample query' . Will search all text indices for matches on "sample" AND "query" and then bind ?subject, ?text, and ?predicate to the columns of each matching triple.
Rfe15717 - Query timeouts should only happen during execution not results output
Previously, AllegroGraph would check for query timeouts both during query execution and during result generation. This meant that a timeout could cause partial results to be sent over HTTP. AllegroGraph now only checks for timeouts while the query is executing which means that either no response will be sent or the entire result set will be sent.
Rfe15693 - Optimize some str(?a) = str(?b) filters
AllegroGraph will now execute queries involving str/str equality and inequality FILTERs where the variables in the calls to str are resources more efficiently. For example, if ?a and ?b are resources or plain literals, then this query will be able to execute without needing to look up the string values in the string dictionary:
SELECT * {
:entry :has ?a .
:reference :contains ?b .
FILTER( str(?a) = str(?b) )
} Rfe15692 - Speed up query planning for queries with only simple BGPs
AllegroGraph now avoids computing estimates for queries that contain only single pattern BGPs. E.g.,
prefix f: <ex:foo#>
select * {
graph ?g { ?s2 f:record ?o }
{ select ?s2 { ?s2 f:record ?o2 } }
}
Rfe15686 - Handle query + filter patterns more efficiently
This change allows the query + filter handling improvements from rfe15628 (implemented in release 6.4.2) to be used more often. In particular, the filter pattern can now be the last pattern in a BGP and the filter pattern can also introduce new bindings.
Rfe15529 - Let run-sparql intern query variables in one package
SPARQL query variables (the third return value from run-sparql) were sometimes located in the query.variables package, and sometimes in the db.agraph.sbqe package. Now they are always in query.variables.
Rfe13652 - Improve performance of some DISTINCT SPARQL sub-queries
AllegroGraph has long evaluated single variable DISTINCT queries using a specialized and more efficient querying technique. This technique did not, however, apply to sub-queries. This meant that these two equivalent queries would perform significantly differently:
# Faster
SELECT DISTINCT ?p { ?s ?p ?o } and
# Slower
SELECT ?p { { SELECT DISTINCT ?p { ?s ?p ?o } } } The query engine logic has been modified so that the two queries will perform similarly.
Rfe10047 - Basic support for date arithmetic in SPARQL
AllegroGraph now supports adding and subtracting xsd:durations from xsd:dates, xsd:times, and xsd:dateTimes. For example:
## Find subjects modified since yesterday
SELECT * {
bind( now() - 'P1D'^^xsd:duration as ?yesterday )
?s :hasDate ?date .
FILTER( ?date > ?yesterday )
} AllegroGraph supports adding durations to date/times, adding date/times to durations and subtracting durations from date/times.
Bug25709 - Predicate statistics invalidation is incomplete
The query engine could incorrectly optimize away SPARQL type casting operations. This has been fixed.
Bug25669 - Logical ORs of inequality filters could produce too many solutions
SPARQL FILTERs that logically OR'd multiple inequality constraints together could be incorrectly processed leading to too many solutions. This has been corrected.
Bug25634 - Impossible OPTIONALs can lose solutions in some cases
If a query began with an OPTIONAL clause and that clause found no bindings and the query then required subsequent processing (e.g., via a BIND() clause), then solutions could be lost. This has been corrected.
Bug25564 - SPARQL queries that used constant expressions could fail over HTTP
AllegroGraph v6.4.2 introduced the eager evaluation of constant expressions (like bind(iri('string') as ?value)). This improvement sometimes caused failures when a query plan with these constants was reused because the value of the expression was not being saved by the plan which would lead to UPI is not in string table errors. This has been corrected.
Bug25545 - UUID() and STRUUID() should not be evaluated eagerly
The SPARQL UUID() and STRUUID() functions were being evaluated at plan time rather than query time. This has been corrected.
Bug25501 - Fix bug in t:intervalCotemporal using start and end times
The SPARQL magic property for t:intervalCotemporal worked correctly when using points on the time line (t:startpoint and t:endpoint) but not when using actual times (t:starttime and t:endtime). This has been corrected.
Bug25465 - Constraint analysis could cause queries with SERVICE clauses to fail
AllegroGraph's planning step could fail to process queries with SERVICE clauses that included FILTERs if those FILTERs attempted to query the current repository. This has been corrected.
The problem was introduced in AllegroGraph v6.4.1.
Bug25454 - Problem with SPARQL division by zero
If the operands to a SPARQL divide arithmatic expression were of type xsd:double, then AllegroGraph could incorrectly signal a SPARQL type error if the divisor was 0 instead of returning positive or negative infinity.
Bug25396 - SPARQL parser bug with unnamed aggregate expressions
AllegroGraph v6.4.1 introduced a bug such that the SPARQL parser would fail to parse a GROUP BY expression that did not provide a new variable. For example:
SELECT (count(?s) as ?count) {
?s ?p ?o .
} GROUP BY str(?o) Queries that did introduce a variable like:
SELECT ?foo {
?s ?p ?o .
} GROUP BY (str(?o) AS ?foo) would parse correctly. This has been corrected.
Bug25391 - SPARQL (count(distinct *) ...) can get the wrong answer
It was possible for SPARQL queries using COUNT(DISTINCT *) to produce the wrong answer. This has been corrected.
Bug25352 - Blank nodes from planning and parsing could incorrectly overlap
If a query used both anonymous blank nodes and property paths, then it was possible for the blank node bookkeeping to incorrectly assign identifiers. For example, a query like:
select * {
[ a ?Class ] .
?x :a/:b ?c .
} could be parsed as
select * {
_:b0 a ?Class .
?x :a _:b0 .
_:b0 :b ?c .
} instead of
select * {
_:b0 a ?Class .
?x :a _:b1 .
_:b1 :b ?c .
} This problem has been corrected.
Bug25342 - Alternation in Property Paths could return incorrect bindings
If a SPARQL query used alternation in a property path, it was possible that AllegroGraph would bind the subject side of the path to the variable on the object side. This has been corrected.
Bug25334 - SPARQL parser's variable classification was too strict
AllegroGraph was signaling an error if a GROUP BY variable was renamed via a SELECT expression. For example, the following valid query would fail to parse
prefix : <http://example.org/>
SELECT (?entry AS ?other) (MIN(xsd:date(?date)) AS ?minDate) {
?item a :InternalProcedure .
?entry (:logicalSuccessor)+ ?entry .
?entry :eventDate ?date
} GROUP BY ?event This has been corrected.
Bug25198 - Variable mismatch for (run-sparql "ask ..") as :alists
There was a mismatch in the return values for:
(run-sparql "ASK .." :results-format :alists) as the alist (first value) could be:
((?result . t))
where the variable names (third value) would be: (?found)
Now both use the db.agraph.query.variables::?result symbol.
Bug24650 - Improve SPARQL parser's validation of VALUES clauses
AllegroGraph is now more strict when evaluating VALUES clauses where the number of variables in the clause does not match the size of each row of values.
Bug23066 - XSD casting could fail when used with expressions
If an XSD casting operator was used on the result of an expression, the result could be returned as nil. This has been corrected. For example, the following query used to fail with SPARQL type errors:
select ?value {
values (?a ?b) { (1 2) }
bind( xsd:double( ?a + ?b ) as ?value )
} Bug22548 - Add page rank centrality to Prolog and SPARQL
AllegroGraph now supports computing page rank centrality via the page-rank-centrality Prolog function and also via a new magic property: (link http://franz.com/ns/allegrograph/4.11/sna/pageRankCentrality).
Bug19805 - Dynamic restriction reasoner could fail to infer some triples
The dynamic restriction reasoner was only inferring rdf:type triples when either the subject or the object (or both) was supplied. It would not make inferences when both sides were wild. This has been corrected. Note that inferring all type triples in a query like:
SELECT ?o ?c { ?o a ?c } can be expensive due to the amount of computation required.
AGWebView
Bug25510 - Bug in reposcript handling in AGWebVeiw
Previously if you saved or loaded a reposcript from the Utilities | Scripts page, the page would change to one which does not show reposcripts at all. This has been fixed.
Bug25336 - 'Cancel' button does not work in 'Import RDF from a text area input' form
The 'Cancel' button on the 'Import RDF from a text area' dialog used not to work unless an upload had been started. This has been corrected.
Bug25164 - Show metadata in WebView for Parallel SPARQL queries
Parallel SPARQL queries run in WebView against a federated triple store would not show query metadata in the "Information" tab. Now the tab shows a list of the query metadata for each shard of the federated store.
Changes to the Lisp API
Rfe15933 - Remove variable get-triples-list-limit
The variable *get-triples-list-limit*, if not nil, used to limit the number of triples returned by function get-triples-list. As get-triples-list already takes an explicit limit keyword argument, the variable has been removed.
Rfe15879 - Remove RUN-SPARQL :ENGINE argument, VALID-QUERY-ENGINES documentation
Since there is only one SPARQL engine in the current release, the :engine keyword argument to run-sparql and the operators get-sparql-query-engines and valid-query-engines have been removed.
Rfe15870 - Error when a query returns a cursor and there is no query environment
If run-sparql returns a cursor, then it is imperative that the code using that cursor be wrapped with the with-query-environment macro. The lack of wrapping can lead to hard to diagnose problems.
Rfe15828 - Remove preserve-strings and return-[non-]encoded-triples arguments
The Lisp API has long included a preserve-strings argument to the various functions that add triples to a repository (such as add-triple, load-ntriples, and others). When true, both a string version and an encoded version of triples whose objects were typed literals were added. Similarly, functions that retrieved triples (such as get-triples) included arguments to request either only the encoded version or the non-encoded version of said triples (return-encoded-triples and return-non-encoded-triples respectively). These three arguments have been removed from the API.
Rfe15603 - Improve behavior of db when closing an encapsulated triple store
The behavior of variable db when closing or deleting a triple store was not always intuitive. The new behaviour is that (close-triple-store :db db) only updates the binding of the variable db if it had db as its value before the call.
The new value of db depends on the type of db:
if
dbis an encapsulated store (like a reasoning store) then db is afterwards set to the inner store (in a sense unwrapping the encapsulation);otherwise db is set to
nil.
Rfe15282 - Allow warmup of individual index flavors
warmup-triple-store now accepts an indices keyword argument to allow specifying which indices to warm up. When the include-triples is true, indices specifies which indices to warm up. :indices nil or unspecified means warm up all indices.
Bug25553 - optimize-indices on a remote-triple-store ignored the wait-p argument.
In the lisp client, if optimize-indices was called on a remote-triple-store with :wait-p t, a non-True boolean value was passed in the HTTP request generated by the call. As a result, the call would return before the optimization operation had completed.
This is now fixed.
Bug25401 - Fix hanging (get-triples :return-extended-triples t) on remote store
Retrieving triples from a remote-triple-store could indefinitely hang if the number of triples was large (typically tens of thousands). This has been resolved.
Prolog
Bug24993 - Default graph in query results
Prolog queries that return the default graph object in a binding caused an error if CSV, TSV or TTL was used as the output format. Such queries also failed to display results in AGWebView. This has been corrected. Any variable bound to the default graph will be treated as unbound.
Bug22548 - Add page rank centrality to Prolog and SPARQL
AllegroGraph now supports computing page rank centrality via the page-rank-centrality Prolog function and also via a new magic property: (link http://franz.com/ns/allegrograph/4.11/sna/pageRankCentrality).
Documentation
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.4.6
AllegroGraph Server
Bug25699: Restoring upi-string got illegal value during recovery/replication
AllegroGraph 6.4.0 introduced a bug that could result in the following error message during crash recovery or when using replication:
restoring upi-string got illegal value [xxx] This could prevent a repository from starting up after a crash. This has been fixed.
Bug25653 - A multi-master repl repo that died will restart the next time it is opened
A multi-master repl instance repo (see Multi-master Replication) will remain open until it is explicitly stopped or for the period of time when the agraph server is not running. If an instance dies for some reason and a request comes in to open its repo then the repo now will open and rejoin the cluster. Previously an explicit start was required to start the instance and open the repo.
AGWebView
Rfe16002 - Move query from URL to body
Previously in AGWebView when running a query or downloading query a result, the query was in placed the request URL. This caused problems with long queries with HTTP proxies that limit the length of URLs. Now the query text is placed in the request body.
AllegroGraph 6.4.5
AllegroGraph Server
Bug25586 - Error during recovery of partially-created text index
Previously it was possible for a crash to occur when restarting a repository instance if a partially-created text index existed during at the time of the last checkpoint. This has been fixed.
SPARQL
Bug25564 - SPARQL queries that used constant expressions could fail over HTTP
AllegroGraph v6.4.2 introduced the eager evaluation of constant expressions (like bind(iri('string') as ?value)). This improvement sometimes caused failures when a query plan with these constants was reused because the value of the expression was not being saved by the plan which would lead to UPI is not in string table errors. This has been corrected.
Bug25545 - Uuid() and STRUUID() should not be evaluated eagerly
The SPARQL UUID() and STRUUID() functions were being evaluated at plan time rather than query time. This has been corrected.
AllegroGraph 6.4.4
AllegroGraph Server: Multi-master replication changes
Rfe15786 - Changes while adding a multi-master replication instance
In earlier releases, when a new MMR instance was added, it could not be removed until the addition completed. This caused problems when the addition took excessive time. Now additions can be removed at any time after they are added. Also, the Replication Cluster Management page in AGWebView would not always display correctly while a MMR instance was being added. That has been fixed.
AllegroGraph 6.4.3
AllegroGraph Server: Multi-master replication changes
Rfe15762 - In repl mgr do timeout for repl/stop http calls
Sending an HTTP request to non-existent machine can sometimes take a long time to timeout. The repl/remove and repl/stop commands for Multi-master Replication now accept a timeout parameter to control how long they wait for the affected host to respond before giving up or moving on.
Rfe15737 - Add config directive to set HTTP keep-alive timeout
The new config directive HTTPKeepAliveTimeout specifies the HTTP keep alive timeout. Configuration directives are described in the Server Configuration and Control document. HTTPKeepAliveTimeout is a top-level directive.
Rfe15712 - Improve replication temporary commit error
When it's not possible to commit at the moment due to replication parameters not being satisfied, an error will be signaled that will not backtrace to agraph.log and that will return a 409 "conflict" error from HTTP.
Bug25397 - Replication restart of two instances at the same time
Previously when two replication instances start up at nearly the same time they can get into a state where they fail to link up. This has been fixed.
Bug25364 - Return x-durability header in commit via HTTP
When an HTTP command results in a commit, "x-durability: missing=N" (where N is the number of of instances that are left to commit to satisfy durability) is supposed to be HTTP response headers but it was not. This has been fixed.
AllegroGraph 6.4.2
AllegroGraph Server: General Changes
Rfe15599 - Disallow creation of repos within the system catalog
Attempts to create repositories in the system catalog will now be met with the following error message:
Repositories cannot be created within the 'system' catalog Rfe15528 - Increase speed of finding distinct predicates or graphs in a reasoning triple-store
It is now much more efficient to find distinct predicates or graphs when RDFS++ or restriction reasoning is enabled.
Rfe15492: Use triple-store-spec for certain agtools
The following agtools (see agtool) have had their syntax changed to use a triple-store specification instead of the options --port, --user, --password, --host, and --catalog. The affected tools are: cancel-purge-deleted-triples, define-attribute, delete-static-attribute-filter, get-metadata, lookup-attribute-definition, memory-lock, memory-unlock, purge-deleted-triples, purge-rate-limit, set-purge-rate-limit, set-static-attribute-filter. The triple-store specification is
http(s):[//[USER:PASSWORD@]HOST[:PORT]][/catalogs/CATALOG]/repositories/REPO Items in brackets ([]) may be left out in some circumstances depending on what is required for the tool.
Rfe15482 - Change if-exists default for agtool repl create-cluster
In agtool repl create-cluster --if-exists argument used to default to supersede and now defaults to error. Existing scripts using agtool repl create-cluster may now fail to work until --if-exists supersede is added to the argument list.
Rfe15481 - Improve turtle serializer output for numbers
AllegroGraph's Turtle and TriG serializers now emit integers, decimals and double floats using the more concise numeric representation rather than the full XSD typed literal one.
Rfe14740 - agtool can now manipulate users and roles
Added agtool user and agtool role commands to manipulate user accounts and roles respectively. For more infomation, execute:
agtool user --help
agtool role --help See the Managing Users document for further information.
Rfe11183 - Add --optimize option to agload
agtool load now accepts the --optimize option to request full (level 2) index optimization after loading completes. See the agtool load document. Index optimization is discussed in the Optimizing indices section of the Triple Indices document.
Bug25289 - Dropping a text index deletes files too early
Dropping a freetext index could result in instance startup errors if the instance crashed before a checkpoint occurred. This has been fixed.
Bug25288: Free Text Index state updates not recorded in tlog during commit
Starting in AllegroGraph v6.3.0, changes in text index configuration (such as creation and removal) were not stored in the transaction log during commit. This could result in disappearance of an added text index, or reappearance of a deleted text index after a repository crash restart. This problem is now fixed.
Bug25266 - sna:egoGroup function does not find its generator in all cases
The functional form of the sna:egoGroup magic property did not look up and register its SNA generator function if the generator was stored using triples. This has been corrected.
Bug25236 - Turtle serializer could emit invalid qnames
It was possible for the Turtle serializer to produce invalid output if resources contained characters that needed to be escaped. This has been corrected.
Bug25207 - Using Japanese tokenizer can result in errors
Using the Japanese text index tokenizer on a system that does not have the mecab package installed would result in an error like the following:
tagger.cpp(153) [load_dictionary_resource(param)] param.cpp(71)
[ifs] no such file or directory: /etc/mecabrc This has been fixed.
Bug25201 - agtool export output depends on the value of LANG
It was possible for the value of the LANG environment variable to alter the output of agtool export. This has been corrected.
Bug25138 - agtool CMD will fail if run from a directory it cannot write to.
If the current working directory is not writable agtool query and agtool materialize would stop running with an error the first time they try to write to their respective log files. Now this behavior has been changed to warn the user that it was unable to write to the log file and then continues processing. It continues logging to the terminal as it would normally do. See agtool General Command Utility for a description of agtool.
Bug25115 - 'agtool export' errors when using --parallel and exporting NQX
Attempting to use agtool export with --output nqx and --parallel mode would result in an error like the following:
error `#<consumer-cursor @ #x1000991a312> does not support extended-triples`
occurred in worker 11
interrupting... This has been fixed.
HTTP API
Bug25246 - Improve HTTP socket reuse for remote repositories
Queries on remote triple stores now reuse the same underlying HTTP socket connection for efficiency.
SPARQL
Rfe15644 - Improve probing decision
AllegroGraph decides at query time whether it is more efficient to scan or probe to evaluate a triple pattern. However, if a pattern had certain constraints on some of its variables, then AllegroGraph was unable to probe. Now AllegroGraph is able to probe in more cases which can greatly improve performance for particular queries.
Rfe15628 - Handle query + filter patterns more efficiently
AllegroGraph now handles SPARQL queries where a pattern in a BGP can act as a filter on the existing set of solutions. As an example, consider this query:
:target :select ?s .
?s :has ?property .
?property rdf:type :animal . Prior to this change, AllegroGraph would first find all <?s :has ?property> triples and only then filter the results using the <?property rdf:type :animal> pattern. Depending on the shape of the dataset, this could generate a large number of intermediate results. AllegroGraph is now able to treat the <?property rdf:type :animal> pattern as a filter while evaluating the <?s :has ?property> pattern which can help speed the query.
Rfe15607 - Sort language tagged literals sensibly
The SPARQL 1.1 standard does not define an ordering on language tagged literals. AllegroGraph now orders language tagged literals by first comparing their languages and then their string values. Plain and xsd:string typed literals come before any language tagged literal.
Rfe15600 - Remove and warn about unused BINDings
AllegroGraph now issues a query warning if variables introduced by a BIND clause are not used elsewhere in the query. For example:
select distinct ?o {
bind(23 as ?number)
?s ?p ?o .
}
Rfe15541 - Warn when a BIND form is extending an empty set of solutions
The SPARQL BIND construct operates by extending an existing set of bindings. If there are no bindings, then there is nothing to extend and so the assignment will uniformly fail. AllegroGraph now warns when it encounters this situation. For example,
SELECT ?ABC {
BIND(?s AS ?ABC)
?s ?p ?o .
} Previously, AllegroGraph was incorrectly rearranging the above query so that it would return bindings for ?ABC. Now the above query will leave ?ABC unbound and emit a :warn-empty-bind-clause warning.
Rfe15524 - Improve SPARQL property path efficiency
The data structures used to compute SPARQL property paths have been improved to provide better performance.
Rfe15523 - Improve SPARQL dataset efficiency
SPARQL datasets are now slightly more efficient when processing queries.
Rfe15522 - Use existing constant bindings when evaluating subsequent ones
AllegroGraph now uses existing constant BINDings when evaluating subsequent ones during query planning rather than waiting until query execution.
Rfe15515 - Support variable attributes in SPARQL attributes block
AllegroGraph SPARQL now supports variable attributes for SPARQL update operations. For example, the query:
prefix attr: <http://franz.com/ns/allegrograph/6.2.0/>
insert {
attributes ?attr { ?s ex:newPredicate ?o . }
} where {
?attr attr:attributes (?s ex:oldPredicate ?o }
} would replace each triple whose predicate was ex:oldPredicate with one having the predicate ex:newPredicate and the same attributes.
Rfe15501 - Make better use of FILTER constraints in property path queries
AllegroGraph now uses constraints derived from SPARQL property path queries in more cases. This can help to improve the performance of some queries.
Rfe15413 - Improve handling of constant BIND expressions
AllegroGraph now analyzes constant expressions in BIND forms and can use the results to improve the efficency of some queries.
For example, AllegroGraph previously would execute only the first of the following two logically equivalent queries efficiently:
# BIND with a constant
select * {
bind( <http://example.com#Mammal> as ?Class )
?a rdf:type ?Class .
}
# BIND with a constant _expression_
select * {
bind( iri('http://example.com#Mammal') as ?Class )
?a rdf:type ?Class .
} Now both of the queries will execute efficiently.
Rfe15381 - Allow federated geospatial query if one store has no geospatial mappings at all
When executing SPARQL geospatial query on federated-triple-stores an error was thrown if federated stores had different geospatial type mappings. This was an overly strict limitation, so that error is now only thrown when regular type mappings differ.
Bug25271: Fix "agtool query" on SPARQL endpoint
Running a query on a SPARQL endpoint using "agtool query", e.g.
./agtool query http://dbpedia.org/sparql would, after printing results, signal the error:
Error "http://dbpedia.org/sparql" is not a triple store occurred during setup This has been resolved.
Bug25268 - Correct handling of query log line length
It was previously impossible to set the query log line length using the logQuery query option. The log length can now be set with the new logLineLength query parameter.
Bug25247 - Some SPARQL sub-queries with external filters could fail.
AllegroGraph could fail to correctly execute SPARQL queries when they involved a sub-query and FILTERs that applied to triple-patterns within the sub-query. For example:
select ?s ?timestamp {
{
select distinct ?s ?timestamp {
?s allegro:hasCreationDate ?timestamp .
}
}
FILTER (?timestamp >= '2018-04-12T20:32:53.213Z'^^xsd:dateTime>)
FILTER (?timestamp <= '2018-04-15T20:32:53.213Z'^^xsd:dateTime>)
} This has been corrected.
Bug25170 - Property path analysis of missing constants is too stringent
AllegroGraph was marking some property path queries as if they were impossible when elements of the path were not found in the repository. This led to some queries failing to return results. The problem has been corrected.
Bug25197 - Fix handling 'select *' results from remote endpoints
Queries of the form select * { .. } sent to a non-AllegroGraph endpoint could fail with a "result variables are in wrong order" error even though the query results sent by the endpoint were valid. This has been fixed.
Bug25169 - Impossible NOT EXISTS clauses should not make query impossible
The logic that notices when parts of a SPARQL query cannot succeed and marks them as impossible was incorrectly treating NOT EXISTS filters as if they were EXISTS filters which led to some queries failing to return results. This has been corrected.
Bug25151 - Problem with complex NOT EXISTS filters
Queries that used [NOT] EXISTS in a FILTER could lead to an error if the query inside the [NOT] EXISTS contained language filters.
Bug22482 - Improve property path estimates with fixed subject/object
AllegroGraph now does a much better job of estimating the cost of transitive closure (i.e., zero-or-more and one-or-more) property path queries. These improved estimates can lead to greatly improved plans.
AGWebView
Rfe15410: Add repository name and group columns to the AGWebView cluster table
On the replication manager page in Webview columns were added to display the repository name and catalog as well as the group of each instance. See the Managing Clusters Using AGWebView section for information on the WebView cluster table.
Bug25146: Ok button never reenabled after failed text area load
In AllegroGraph WebView when importing statements from a text area input the OK button didn't reenable after the loading process had failed. This has been corrected.
Changes to the Lisp API
Rfe15547 - Add type checking to the upi function
The upi function now includes type checking so that it will signal a clearer error message if passed an argument of the wrong type. The function is now also defined on UPIs in addition to future-parts.
Rfe15499 - delete-triple improvements
delete-triple now accepts a triple object as argument, as alternative to a triple id.
In either case the return value now indicates whether the deletion was successful.
Rfe15483 - Add part-to-namespace to the Lisp API
The Lisp API now includes part-to-namespace. This function takes a UPI, a future part, or a string representing a URI (with or without angle brackets) and returns the Prefixed Name and Local Part (as multiple values) if a registered namespace can be found to abbreviate the resource. For example:
(part-to-namespace "<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>")
==>
"rdf"
"type"
Rfe14068 - Let run-sparql return query metadata for remote triple stores
Calling the run-sparql function on a remote triple store now returns query metadata like timing information as fourth value, just as for local triple stores.
Rfe8199 - Add a with-open-triple-store convenience macro
The Lisp API now includes with-open-triple-store macro which opens a triple-store, executes the body, and then closes it.
Bug25305 - Export cursor-peek from db.agraph package
The cursor-peek function was introduced in AllegroGraph v6.4.1 but was not exported from the db.agraph package. The symbol is now exported and also available in the db.agraph.user package.
Bug25192 - materialize-entailed-triples was not supported in the remote Lisp client
Both materialize-entailed-triples and delete-materialized-triples are now available in the remote Lisp client.
Bug25168 - Fix leaked http-cursor finalizer clash
Cursors can be leaked, for example by using return-from inside iterate-cursor. These cursors are eventually garbage-collected. There was a finalizer race condition in the collection of HTTP connection based cursors, that would lead to stream-closed-error being signaled to the user unexpectedly. This has been resolved.
Bug25168 - Improve cursor leak message for get-triples on remote-triple-store
If get-triples is called on a remote-triple-store outside with-query-environment, the cursor will open an HTTP socket to the triple store. If the cursor is leaked it will be garbage-collected eventually, but until that moment the socket will be kept open. This can negatively impact performance, therefore a warning will now be logged:
[2018-03-12T09:06:27.250 p13791 W cursor] Cursor created by
(get-triples) on remote-triple-store "db" was leaked,
will be closed. Bug25166 - remote-triple-stores reason on the client rather than the server
AllegroGraph now ensures that the dynamic reasoner will execute on the server rather than the client when using a remote-triple-store. This prevents a large number of get-triples from executing via HTTP when using the RDFS++ reasoner.
Prolog
No significant changes.
Documentation
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.4.1
AllegroGraph Server: General Changes
New agtool commands and some changed arguments
The *agtool command-line utility has new commands available. Further some commands have new arguments and some arguments have changed. See the agtool General Command Utility document for details. Most commands now have a --scheme argument which can be http or https.
agtool repl create-cluster adds --if-exists and --force arguments
agtool repl create-cluster previously would only create new, empty repositories and if an existing repository were specified, it would be deleted and a new, empty repository with the same name would be created. (This behavior differed from the behavior when creating clusters with AGWebView, where a repository had to exist and need not be empty when it was converted to an initial replication instance.) Now agtool repl create-cluster accepts an --if-exists argument with allowable values supersede (the default, meaning delete the existing repository and create a new one with the same name), overwrite (same as supersede), use (convert an existing, possibly non-empty repository), and error (signal an error if the repository specified already exists).
If use is specified, then any encoded id definitions will be examined and if any do not allow for 2^60 distinct strings and so will not work with next-encoded-upi-for-prefix, the conversion will fail and a warning will be printed. If the --force argument with value true is supplied, then the conversion will be done but none of the offending encoded id prefixes must be used again.
The Cluster Replication Manager page now displays statistics
These help users to monitor the state of the cluster. See the Managing clusters in AGWebView once created section in the Multi-master Replication document.
When adding a cluster instance in AGWebView, a progress log is displayed
When adding an instance in Webview you'll see that the log of actions is updated with progress reports describing what is being done. Also the AGWebView replication manager page has new columns and some columns have been renamed. See here for information on using AGWebView to manage clusters.
Rfe15465 - Disallow setting bulk-load-mode in replication instance
In a replication instance one cannot set bulk load mode since replication depends on the transaction logs being an accurate reflection of the changes to the repository and logging is disabled in bulk loading mode. Specifying the --bulk option to agtool load or clicking the Use bulk load mode checkbox when adding triples from an uploaded file in AGWebView will cause the load to fail.
Rfe15451 - Don't fail if unable to write to agload.log
If the current working directory is not writable agload would stop running with an error the first time it tries to write to the log file agload.log. Now this behavior has been changed to warn the user that it was unable to write to the log file and then continues processing. It continues logging to the terminal as it would normally do.
Rfe15426 - Expose and document the --parameter option for agtool load
agtool load now accepts the --parameter option for controlling the configuration of newly created repositories. See the --parameter option in the agtool load document for details.
Rfe15418 - Change StringTableCompression default to 'none'
AllegroGraph now in the default creates new repositories with string compression disabled. This results in faster out-of-the-box operation. String compression can be enabled by using the StringTableCompression directive in agraph.cfg. See the description of StringTableCompress in the Server Configuration and Control document for information.
Note that the string compression type can only been set when creating a new repository. The StringTableCompression parameter is further discussed here in the Performance Tuning document.
Rfe15190 - Remove --repl argument from agtool archive backup and agtool recover
The --repl argument for agtool archive backup and agtool recover is no longer present because it is no longer needed (whether or not a repository is a replication repository is now automatically determined).
Rfe14770 - Error message from agload loses most of the detail
When using agtool load the error messages signaled while loading triples were truncated. This is now corrected.
Rfe14389: AllegroGraph no longer supports Callimachus
AllegroGraph no longer supports Callimachus. Stores that were previously set up to integrate with Callimachus will open normally but they will no longer execute the Callimachus integration code.
Bug25087 - TempDir is not being used for placing temporary query files
Temporary query files were being placed in the repository's directory rather than in the location specified by the TempDir configuration parameter. This has been corrected.
Bug25075 - Bidirectional search can return bad paths
If there were many possible overlapping bidirectional paths between two nodes, AllegroGraph could return incorrect paths. This has been corrected.
Bug25102 - Subscript UPIs did not print correctly.
AllegroGraph was printing the prefix of a subscript UPIs twice. This has been corrected. Previously, they would print as
<http://franz.com/ns/allegrograph/4.14/index#_http://franz.com/ns/allegrograph/4.14/index#_4> Now they print as
<http://franz.com/ns/allegrograph/4.14/index#_4>
Bug25062 - Fix uninformative error when running agraph-control as root.
When running agraph-control as root when there was no RunAs directive in agraph.cfg would result in the following uninformative error:
Got error while executing plan: `nil' is not of the expected type
`(array character (*))' for foreign type `(* :char)' Now the error messages generated for no RunAs user and non-existent RunAs user, respectively, are:
# bin/agraph-control --config lib/agraph.cfg start
Running AllegroGraph as superuser is not recommended for security reasons.
It is recommended that another less privileged user be used, specified
via the `RunAs' directive in agraph.cfg.
# bin/agraph-control --config lib/agraph.cfg start
RunAs user `UserThatDoesNotExist' does not exist in the passwd database.
Please run the AllegroGraph configuration script or adjust the `RunAs'
directive in lib/agraph.cfg
HTTP API
Rfe15418 - Allow specifying string table compression method
Added a stringTableCompression parameter to the repository creation HTTP service.
SPARQL
Rfe15421 - Add control over duplicate triple reduction in SPARQL queries
AllegroGraph now has a new query option: enableDuplicateReduction which controls whether or not the SPARQL engine reduces the number of duplicate triples it processes. See SPARQL Query Options in the SPARQL Reference document.
If the store has no duplicate triples (based on subject, predicate, object, and graph) then this option has no effect.
SPARQL is defined on sets of triples (i.e., no duplicates) but AllegroGraph can store duplicates which can lead to duplicate rows of bindings returned by a query.
If this option is set to true, then AllegroGraph will ignore consecutive duplicate triples returned from the storage layer. Since the storage layer makes no guarentees on the order of triples it returns, this can lead to different results from the same query.
Note: Previous versions of AllegroGraph used duplicate reduction by default. This version turns if off by default. If your triple-store has duplicate triples, then this may change the results of your queries.
Rfe15419 - Include query modifiers when printing a SPARQL query plan
AllegroGraph now includes query modifiers like ORDER BY or LIMIT when it prints a query plan.
Rfe15391 - Warn when the variable HOST of a SERVICE clause is not bound
AllegroGraph now warns when the HOST variable of a SERVICE clause is unbound in the query.
For example, both of these queries will produce query warnings:
# Query 1
select * {
{
:myService :isAtURL ?url
} union {
service ?url { ?s ?p ?o }
}
}
# Query 2
select * { service ?unknown { ?s ?p ?o } } Rfe15354 - Improve handling of queries containing GRAPH and JOIN clauses
AllegroGraph will now handle queries involving GRAPH and JOIN clauses more efficiently.
Rfe15336 - Sanity check on JSON/XML data returned by SPARQL endpoint
If a SPARQL query includes a SERVICE call, the XML and JSON result data is now subject to more syntactic and semantic error checking. Nevertheless invalid result data from third parties might still lead to unexpected outcomes.
Rfe15273 - Add Javascript API for defining SPARQL extensions
AllegroGraph's server side Javascript engine now supports the following:
- sparql.extensionFunctions() - return a list of SPARQL extension function names.
- sparql.defineExtensionFunction(uri, function) - define a new extension function associated with
uri. - sparql.magicProperties() - return a list of magic property URIs.
- sparql.defineMagicProperty(uri, parameters) - define a new SPARQL magic property.
For details, see the Javascript documentation.
Bug25145 - Illegal keyword error when executing some queries
AllegroGraph could signal an Illegal Keyword error when executing some complex queries. This has been corrected.
Bug25113: SPARQL planner stopped optimizing certain queries
If a query has a FILTER like
FILTER( ?var1 = ?var2 ) and it can be determined that both ?var1 and ?var2 are resources, then AllegroGraph can perform certain query optimizations. This feature was inadvertently broken in AllegroGraph 6.4 and has now been reinstated.
Bug25097 - Graph not in dataset warning signaled inappropriately
AllegroGraph v6.3.0 and following signal a query warning if the graph in a GRAPH clause is not in the dataset as in, for example, this query:
select *
from named <http://example.com#graph> {
graph <http://example.org#otherGraph> { ?s ?p ?o }
} AllegroGraph was signaling this warning more often than necessary. This has been corrected.
Bug25064 - GROUP BY expressions in a SUB-QUERY causes an error
AllegroGraph was signaling an error if a SPARQL query used GROUP BY expressions in a SUB-QUERY. This has been corrected. An example query that used to fail would be
select * {
{ select ?strWho (sum(?amount) as ?sum) {
?who ?p ?amount .
} group by (str(?who) as ?strWho)
}
}
Bug25057 - Values and OPTIONAL could lead to too many results
If a SPARQL query used a VALUES form inside the query and the VALUES form had only a single variable and a single binding for that variable and the query also used an OPTIONAL form that made use of the same variable, then AllegroGraph could return results with the variable unbound. This has been corrected.
As an example, AllegroGraph previously could have returned results with ?o2 unbound for the following query:
SELECT ?s ?o1 ?o2 {
?s ?p1 ?o1
OPTIONAL { ?s foaf:knows ?o2 }
VALUES (?o2) { (:b) }
} Bug25055 - Improve SPARQL parser error message for bad aggregation expression
AllegroGraph was signaling an error for bad aggregation expressions rather than indicating that the query failed to parse. For example, this query is invalid because it is illegal to GROUP BY an aggreagate:
select (sum(?o) as ?sum) (sample(?s) as ?ss) {
?s ?p ?o .
} group by (sum(?o) as ?xxx) having (?xxx > 2) AllegroGraph now signals a SPARQL parser error for queries like this.
Bug25034 - Catch additional SPARQL parse errors
SPARQL is defined such that repeated variables in the projection are ignored. Previously, AllegroGraph would include each repeated variable. Now, it will include only the first instance and will warn if the list includes repetition.
For example, this query:
SELECT ?s ?g ?s { graph ?g { ?s ?p ?o } } Will project only ?s and ?g and will warn that "The following variable is projected multiple times: ?s. Only the first projection will be included".
The variables in expressions and aggregation specifications may not overlap with other projected variables. AllegroGraph now catches errors like:
SELECT ?o (str(?o) as ?o) {
?s ?p ?o .
} or
SELECT ?s (count(1) as ?s) {
?s ?p ?o .
} GROUP BY ?s Previously, the above queries would have passed the parser.
AGWebView
Rfe15395 - Highlight all instances of the token under cursor in the query editor
AGWebView's SPARQL query editor now highlights any tokens that match the one currently under the cursor. This makes it easier to ensure that variable names are not mistyped.
Changes to the Lisp API
rfe15430 Use of next-encoded-upi-for-prefix in multi-master cluster instances
In order to use next-encoded-upi-for-prefix when the repository is a multi-master cluster instance, the template associated with the encoded id prefix must generate exactly 2^60 distinct strings. If it does not, the call to next-encoded-upi-for-prefix will signal an error. As noted just below, the "plain" will work as a template. See register-encoded-id-prefix.
Rfe15466 - Plain format encoded ids
register-encoded-id-prefix will treat the format argument of "plain" (a string) in this special way: The encoded id (the string after the @@) is a variable length integer in the range from 0 to (2^60 -1), much like blank nodes. This is a template suitable for use in multi-master cluster instances. See the Encoded IDs document.
rfe15430 Use of next-encoded-upi-for-prefix in multi-master cluster instances
In order to use next-encoded-upi-for-prefix when the repository is a multi-master cluster instance, the template associated with the encoded id prefix must generate exactly 2^60 distinct strings. If it does not, the call to next-encoded-upi-for-prefix will signal an error. As noted just above, the "plain" will work as a template. See register-encoded-id-prefix.
Rfe15379 - Remove in-line-data argument from the run-sparql
The in-line-data argument to run-sparql is no longer supported. If you need to execute the same basic query with different sets of BINDINGS, then use the VALUES form in your SPARQL query string.
Rfe15373 - Export peekable-cursors
The Lisp API now includes these new functions:
A peekable-cursor is like a regular cursor except cursor-peek can be used to see what cursor-next-row will return without modifying the state of the cursor. See the documentation for details.
Bug25119 - load-turtle not accepting a :base-uri arg in remote lisp client
The remote lisp client did not support the base-uri argument to load-turtle. This has been corrected.
Bug25070 - upi->value uses db when decoded encoded IDs
The upi->value function was using db to decode encoded IDs even when a specific value for db was passed into the function. This has been corrected.
Bug25043 - get-triples-nd-* functions were incompletely documented
The various nD get-triples functions (like get-triples-nd-within-bounding-box) return :part-not-found as a second return value if it can be determined that one of the search parameters is definitely not in the repository. This is now documented.
Bug23635: Make Lisp client :default-graph-uris keyword work with remote-triple-stores
SPARQL queries made against remote-triple-stores via run-sparql, that passed the :default-graph-uris keyword argument, would not return expected results. This has now been fixed.
Prolog
No significant changes.
Documentation
No significant changes.
nD Geospatial
Bug25129: Bus error on geospatial search with a non-existing predicate
The bus error on the server happened when trying to obtain triples from polygon using predicate that was not previously added to the store.
Behaviour for obtaining triples from polygons was modified to properly handle this case.
AllegroGraph 6.4.0
AllegroGraph Server: General Changes
Rfe15302 - Remove support for using rapper as an external parser
The use-rapper argument to the AllegroGraph data import commands has been removed because AllegroGraph now supports all of the standard formats.
The rapper command line argument to agtool load has similarly been removed.
Rfe15219 - Add timeout and limit arguments to the SNA path-finding functions
The SNA path finding functions now have a timeout argument and a limit argument. The first specifies the number of seconds to execute the search whereas the second indicates that searching should stop once that many paths have been found.
The SNA path functions will return (as a second value) the status of the search. This will be either nil if the search was not interrupted, :timeout if the search reached the timeout, or :limit-reached if the limit on the number of paths was reached.
Note that the return format for the REST API to the SNA path functions is now versioned. The desired version if specified via a version parameter in the query string. If the version is 1, then the previous result format of a list of lists representing paths will be returned. Otherwise, the new format will be used. The new format consists of a JSON object with two keys: "paths" and "status". "paths" contains the list of paths and "status" contains the reason that the search completed. It will be one of "complete", "timeout", or "limit-reached".
Bug24980 - Turtle serializer escapes the PNAME_NS incorrectly
The Turtle serializer was escaping both the namespace portion and the local portion of a qualified name using the same logic. This was incorrect as the escaping for the two differ. The problem has been corrected.
HTTP API
No significant changes.
SPARQL
Rfe15337 - Have the NOW() SPARQL function include a timezone
The SPARQL NOW() function now returns dateTimes in GMT. Previously, the dateTimes had no explicit timezone.
Rfe15259 - Add a query option to limit the size of query log entries
AllegroGraph's SPARQL engine now truncates query log message to 512 characters by default. This limit can be changed by adding ":LINE_LENGTH" to the logQuery query option either in the AllegroGraph configuration file or on a per-query basis. For example, this PREFIX line will limit log lines to 100-characters:
PREFIX franzOption_logQuery: <franz:yes:100> Rfe15258 - Add maximumValuesCountForService query option
AllegroGraph uses the SPARQL VALUES clause to pass partial results to SPARQL endpoints. In general this can help the endpoint compute its results more efficiently but when there are many values the cost of transmitting the data can outweigh these benefits.
The maximumValuesCountForService provides a mechanism to limit the total number of VALUES sent.
Rfe15250 - Improve treatment of very large VALUES clauses
The query planner now handles large VALUES clauses significantly faster. In addition, the query executor now decides whether to probe or scan based on the number of VALUES and the expected number of rows that a cursor will need to traverse. These two changes can make some queries significantly faster.
Rfe11137: Support the TIMEZONE() and TZ() functions
AllegroGraph now supports the SPARQL 1.1. TIMEZONE() and TZ() functions.
Bug25003 - FILTERing on an expression like ( ! ?a = ?b ) caused an error
The above expression evaluates as (!?a) = ?b and AllegroGraph was not correctly handling the expression of the inner negation operator. This has been corrected.
Bug24985 - FILTER EXISTS and SUB-QUERY can lose solutions
If a FILTER EXISTS clause started with a sub-query then AllegroGraph could fail to find some solutions. This has been corrected.
Bug24657 - GRAPH filter can be applied to the wrong clause
If a graph variable appeared in the subject, predicate, or object position of a triple-pattern, it was possible for AllegroGraph to apply a GRAPH dataset constraint to it incorrectly. For example, AllegroGraph could have incorrectly determined that the ?g1 :predicate1 ?object pattern in the query below should be evaluated as if it was inside a GRAPH clause which could lead to incorrect results.
select * {
GRAPH ?g1 { ?s1 ?p1 ?o1 } .
GRAPH ?g2 { ?s2 ?p2 ?o1 } .
?g1 :predicate1 ?object .
} This has been corrected.
Bug24151 - Correct a SPARQL FILTER analysis issue
AllegroGraph's SPARQL engine would signal an error if a FILTER contained an expression like
str( ?varOne ) = ?varTwo This has been corrected.
Bug20593 - Support the SPARQL 1.1 BNODE( identifier ) function
AllegroGraph has long supported the BNODE() function with no arguments. It now also supports the version that takes a single plain literal or XSD string argument.
Bug24076 - Improve handling of DBPedia.org and other Virtuoso servers as SPARQL endpoints.
AllegroGraph works around an apparent problem with Virtuoso.
AGWebView
Rfe11480 - Added Shutdown instance operation to Repository Control section in AGWebView
See the desription of the repository page in the WebView document.
Bug24949 - Overview page broken for remote sessions
Opening a session to a single, remote store in WebView used to produce the following error message:
Loading failed: Service not supported in a session that does not wrap a single local store.
instead of the overview page. This has been corrected.
Bug24929 - Help messages appear out of view.
Help messages were often shown at the top of the Webview page, out of sight of the current page location. They will now be displayed in tooltips.
Changes to the Lisp API
Rfe15325: Make *db* a required thread binding
To help avoid accidental sharing of a repository connection between threads, each new thread will start with *db* bound to nil. Previously, *db* would retain its global value in new threads.
Given this sample code:
(open-triple-store "test") ;; sets global value of *db*
(mp:process-run-function "new thread"
(lambda ()
;; The new thread wants its own connection to the repository
(open-triple-store "test"))) Prior to v6.4.0, the call to open-triple-store in "new thread" would overwrite the global value of *db*. In v6.4.0, "new thread" will have its own binding of *db* and the global value will not be affected.
Prolog
No significant changes.
Documentation
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.3.0
AllegroGraph Server: General Changes
Catalogs may no longer be named root or system.
The names root and system are reserved as catalog names. root interferes with the default catalog (which is referred to as the root catalog) and the system catalog is used for auditing and other system data. If you have catalogs named root or system, contact [email protected] if you have difficulty upgrading to 6.3.0.
'agtool archive' no longer accepts --catalog, catalog now part of repo name
The agtool archive program (see Backup and Restore) now specifies catalogs and repositories together using the catalog:repository format. The --catalog option to agtool archive is no longer accepted.
Rfe15210 - Log which user starts a session
When a session is started, AllegroGraph now logs the name of the user that started the session. (Although not documented, the AllegroGraph log has always said when a session was started and stopped.) A sample log message:
[2017-09-21T15:59:48.028 p21266 back] Session on port 43704 started by user "test".
Rfe15194 - Support XQuery and XPath Math Functions
AllegroGraph now supports the following XPATH math functions
- acos
- asin
- atan
- atan2
- cos
- exp
- exp10
- log
- log10
- pi
- pow
- sin
- sqrt
- tan
See http://www.w3.org/2005/xpath-functions/math/ for details.
Rfe15132 - agtool archive list can operate on backup directories
agtool archive list now can operate not only on single archive file but can be given backup directory as source. In that case it displays details about every repository inside backup.
Additionally it can be given the optional DB-IN-ARCHIVE argument to display details about single repository in ARCHIVE even if other repositories are present.
New option --summary allows agtool archive list to only list repositories inside ARCHIVE, without printing details about every one.
agtool archive is described in the Backup and Restore document.
Rfe12362 - CORS support
CORS (Cross-Origin Resource Sharing), if enabled, allows scripts run on a web page from one server to make HTTP requests to the (different) server where AllegroGraph is running. CORS is not enabled by default. You enable it with various donfiguration directives described in the CORS directives section of the Server Configuration and Control document.
Rfe12201 - Switching a repository between commit and no-commit modes
This now requires superuser privileges.
Bug24927 - Turtle serializer tried to escape capital A
The turtle serializer could try to escape the letter capital A which could lead to invalid output. This has been corrected.
Bug24910 - Unhelpful Turtle parser error message
Improved the error message the Turtle parser produced when the object of a triple was missing.
Bug24905 - Audit log does not display in AG 6.2.3
The Audit log display feature in AGWebView was inadvertently broken in release v6.2.3. This has been corrected.
Bug24842 - Custom services should not use user namespaces
Custom services were being executed in the context of the current user rather than in the context of the code that defined the service. This has been corrected.
Bug23757 - If a script file generates errors (warnings) while loading, don't start the session.
Previously, any errors or warnings that occurred when compiling a Lisp script were written to the AllegroGraph log file and the session would still start. Now, any problems are still written to the log file but the session will not start and the text of the problem(s) will be returned in the HTTP response.
HTTP API
Bug20845 - /version/info web service removed
The /version/info web service caused the server to report detailed internal information about the server and the machine on which the server was running. It was decided that the information was not in fact useful and could expose security vulnerabilities. The URL now returns 404.
SPARQL
Defining your own magic properties
AllegroGraph lets you define your own Magic Properties using the Lisp API. The Defining Magic Properties Tutorial describes how to do this and provides numerous simple examples.
Rfe15182 - All of AllegroGraph's query warnings are now documented.
See the Query Warnings section of the SPARQL Reference document.
Rfe15152 - Propagate impossible query conditions outward in the plan
When a query contains a triple pattern that cannot possibly match, it will mark the pattern as impossible. The impossible marking will propagate outward (e.g., from the pattern to the BGP to the enclosing JOIN and so on). This allows queries with typos or other errors to fail quickly.
Rfe15135 - Warn if any SPARQL type errors occur during query execution
SPARQL will now log a warning if any type errors are encountered during query execution. For example, it is an error to try and cast the string '1.23e4' to an integer because the lexical representation of an XSD integer consists of "a finite-length sequence of decimal digits (#x30-#x39) with an optional leading sign." (cf https://www.w3.org/TR/xmlschema-2/#integer). Therefore, a query like:
select * { bind(xsd:integer('1.23e4') as ?int) } will leave ?int unbound. Previously, this type error would have been surpressed. Now, AllegroGraph will issue a warn-sparql-type-errors warning.
Rfe15112 - Improve cancelQueryOnWarnings option
The cancelQueryOnWarnings query option now cancels queries immediately for most plan or query warnings. This option defaults to off but can be turned on by placing
QueryOption cancelQueryOnWarnings=yes in the AllegroGraph configuration or by prefixing a SPARQL query with the PREFIX
PREFIX franzOption_cancelQueryOnWarnings: <franz:yes>
Rfe12249 - Improve magic predicates for Solr and MongoDB
The magic predicates that interface AllegroGraph's SPARQL engine with Solr and MongoDB have been improved. The three magic predicates:
- http://www.franz.com/ns/allegrograph/4.5/solr/match
- http://www.franz.com/ns/allegrograph/4.5/solr/matchId
- http://franz.com/ns/allegrograph/4.7/mongo/find
can now take an optional second parameter in their list of subject arguments. This second parameter will be bound to the matching objects of any triples found by the search in the case of solr:matchID and to the associated linking identifier in the cases of solr:match and mongo:find. For example, the following query will bind ?s to the subjects found and ?text to the object:
prefix solr: <http://franz.com/ns/allegrograph/4.5/solr/>
select ?s ?text {
(?s ?text) solr:matchId 'boatswain' .
} Rfe11193 - Improvement performance of SPARQL expression compilation
The SPARQL query expression compiler is now approximately 45 times faster than before. This means that queries that use many expressions (e.g., in FILTERs, BIND, ORDER BY, etc.) will perform more efficiently.
Note that aggregation expressions still use the older and slower compiler.
Rfe11678 - Full-scan warning should take dataset into account
AllegroGraph signals a warning when a SPARQL query needs to perform a full scan and the store contains more than one million triples. AllegroGraph was incorrectly signaling this warning when the query was executed against a SPARQL dataset. This has been corrected.
Bug24898 - Fix problem with magic SNA centrality predicates and neighbor caches
Some queries that used AllegroGraph's SNA centrality magic predicates and neighbor caches could cause an error. This has been corrected.
Bug24861 - Sparql parser confused by "?o<='foo'^^<...> "
The SPARQL parser was incorrectly treating the inequality operator as the start of a URI. This has been corrected.
Bug24681 - Parsing should fail if a blank node label is used in more than one BGP
AllegroGraph did not signal a parsing error if a SPARQL query used the same blank node in more than one basic graph pattern (BGP). This has been corrected.
AGWebView
Rfe15198 - Index management page
An index management page has been added to AGWebView. The new page can be accessed from the Repository Overview page by following a link labeled 'Manage triple indices'. The list of indices has been removed from the overview page and the 'purge deleted triples' link has been moved to the index management page. See The Manage triple indices page for more information.
Rfe14563 - Repository reports
A set of reports describing disk usage and other repository properties has been added to WebView. The reports can be accessed through links on the repository overview page. See the Reports section in the Repository Overview page description for more information.
Rfe13419 - Add warmup triple-store to AGWebView
Add new option Warmup store in Repository Control section in repository overview page. See the description of the Repository Overview Page in the WebView document.
Rfe11915 - Creating freetext index prefills stop words with default ones.
The AGWebView dialog for creating a freetext index now starts with a default selection of stop words (rather than starting with no stop words and the option to add a default selection). See the Free-Text Indices page in the WebView document for more information.
Rfe11256: Import RDF statements directly.
It is now possible to import RDF statements directly by typing or pasting them into a text box in AG WebView. See the Load and Delete Data section of the Repository Overview page description.
Bug24824 - Spurious warning for ASK queries in AGWebView
AllegroGraph WebView was showing a query warning for ASK queries when the "Limit to 1000 results" checkbox was checked. This has been corrected.
Bug24480: "Ignore errors" option was ignored when loading triples from server file
The "Ignore errors" error handling option was not being honored when loading a server-side file using AG WebView. This has been fixed.
Bug23607 - Improve accuracy of query timing results in AGWebView
Previously, the timing results in AGWebView represented the time it took to find the first result. Now, it represents the time it takes to find all results. The total time to display may still be longer as it will also include the time it takes the web browser to render the returned data.
Changes to the Lisp API
Remove :plain datatype mapping
The :plain datatype mapping converted typed string literals into the equivalent plain literal. E.g., it would convert "bear"^^xsd:string into "bear". :plain mappings are no longer necessary because AllegroGraph handles typed strings more efficiently automatically. Mappings are specified with setf and datatype-mapping and predicate-mapping.
Note that any existing datatype or predicate mappings of type :plain will be automatically removed from the store.
Bug24857 - path-finding over HTTP fails when paths include literals
If an SNA path included typed literals, then a remote-triple-store would convert them into the wrong UPI type leading to UPI not in string-table errors. This has been corrected.
Prolog
No significant changes.
Documentation
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.2.3
AllegroGraph Server: General Changes
Rfe15101: Restore tlmgr to tlmgr.restored
The tlmgr file contains information about replication jobs. Since the existence of replication job information can result in transaction log files accumulating, the tlmgr file is no longer restored as is. Now it is renamed to tlmgr.restored during a restore. After a restore, replication jobs must be reestablished by the user.
Rfe12408 - Add graph filtering support to agmaterialize
agtool materialize can now limit materialization to a specified list of graphs via the --graph (short form -f -- note that -g specifies the default graph for inferred triples) command line argument. The inferred graph (specified with --inferred-graph or -g) will always be included in the list. See the Materializer document.
Bug24729 - Spaces in filenames prevent server-side file import.
Importing a server-side file with spaces in its path would fail with an error message about illegal characters in a URI. This issue affected both WebView and the HTTP API, but not agtool load and has been corrected.
HTTP API
Bug20772 - Loading JSON format may fail to commit all triples
Using the 'commit=number' parameter when importing statements through the HTTP API could result in some statements not being committed. The issue only affected JSON imports and has been corrected.
SPARQL
Rfe15077 - Signal a query warning when CONSTRUCT tries to generate invalid triples
AllegroGraph will now signal a query warning if any triples produced by the CONSTRUCT template are invalid. Formerly, these triples were silently discarded.
Rfe15075 - Improve parsing efficiency of large USING [NAMED] datasets
The SPARQL parser now parses UPDATE commands with very large dataset specifications using the USING and USING NAMED clauses more efficiently.
Rfe15074 - SPARQL parser has non-linear behavior for large VALUES clauses
Improved the SPARQL parser performance for queries with 1000s of elements inside of VALUES clauses.
Rfe15061 - Improve efficiency of SPARQL queries with large datasets
AllegroGraph now determines at query time whether it would be more efficient to issue getStatements calls for each graph in a dataset or to issue a single getStatements call and filter the results to include only graphs in the dataset. Depending on the triple-store and the dataset, this can be significantly more efficient.
Rfe15006 - Improve GRAPH and PROPERTY PATH query planning and execution
AllegroGraph now handles property path queries inside of GRAPH clauses more efficiently when the graph is a variable and the clause contains no property path queries that use any of the ZeroOrMorePath, OneOrMorePath, or ZeroOrOnePath operators.
Rfe14450 - Improving cross product warning
The warning issued when AllegroGraph detects a cross product in a SPARQL query is now more descriptive.
Bug24779: Constructed triples should not allow blank nodes in the predicate postion
AllegroGraph was incorrectly allowing CONSTRUCT to produce blank nodes in the predicate position of triples. This has been corrected.
Bug24762 - SPARQL DELETE/INSERT/WHERE and Chunk processing could fail
If the chunk size was too small, a SPARQL DELETE/INSERT/WHERE command could fail to correctly process triples.
Bug24755 - Problems in intervalAfterDatetime and intervalBeforeDatetime
The intervalAfterDatetime magic property could fail to find solutions when using a constant datetime rather than a variable. Additionally, both intervalAfterDatetime and intervalBeforeDatetime could fail to find solutions when the intervals were defined on explicit times rather than using named time points.
Bug24709 - FILTER EXISTS and unbound variables on the RHS
AllegroGraph used to return incorrect results for FILTER EXISTS expressions when the RHS contained any unbound variables. This has been corrected.
Bug24704 - Improve strategy used to find unique values in a store
AllegroGraph uses several techniques to determine all of the unique values stored for subject, predicate, object, or graph. Depending on the quads in the store and the shape of the query, one of these techniques can be much more efficient than the others. It was possible, however, for AllegroGraph to choose the wrong technique in some cases. This has been corrected.
Bug24665 - SPARQL queries with :dateTime inequality filters could lose results
Query results could be lost if a SPARQL query used a non-inclusive inequality filter on xsd:dateTimes or xsd:times. This has been corrected.
Bug24455 - Incorrect results when join variables might be unbound
AllegroGraph would sometimes return incorrect results from joins where a part of the join key was unbound.
This has been corrected.
Bug24215 - Unbound values in join keys
When merging solutions during a join, AllegroGraph would sometimes not produce bindings for variables that are only bound on the RHS of the join and are a part of the join key.
For instance when running this query:
select * {
{ values(?x ?y) {(1 undef)} }
{ values(?x ?y) {(undef 2)} }
} AllegroGraph would return (?x = 1, ?y = undef) instead of the expected (?x = 1, ?y = 2).
This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
Bug24796 - intern-resource error on future-part with unregistered namespace abbreviation
A call to intern-resource on a future-part whose namespace abbreviation was not defined would signal a bus error. This will now signal a condition of type cannot-resolve-namespace-for-future-part-error.
Prolog
No significant changes.
Documentation
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.2.2
AllegroGraph Server: General Changes
Rfe14926: Reserve parentheses characters
Parentheses are no longer allowed in triple store or catalog names. These characters are reserved for use in triple store specifications.
Bug24700 - Unfriendly error message when session ports are unavailable
Previously, if a request to create a session cannot be completed due to lack of an available session port, an unfriendly error message was reported. This has been fixed and now a clear error message is returned.
Bug24687 - Deleting a tlog when free list is full throws error.
Fixes a bug introduced in v6.1.4 where a tlog that was deleted due to the free list being full would be repeatedly deleted by AllegroGraph despite it no longer existing. This bug could prevent the freeing/deletion of old tlogs, causing them to accumulate.
Log entries that indicate this bug, would be similar to the following:
[2017-05-22T13:49:21.800 p17318 tlarchiver] Store "bug24687", free list is full, will just delete "tlog-2477c96f-0579-ba54-b60b-7083f2293fa8-27"
... later ...
[2017-05-22T13:51:34.996 p17318 tlarchiver] Store "bug24687", free list is full, will just delete "tlog-2477c96f-0579-ba54-b60b-7083f2293fa8-27"
[2017-05-22T13:51:34.999 p17318 E tlarchiver] Store "bug24687", Attempt to delete tlog-27 gave error There is no file named /dbs/bug24687/tlog-2477c96f-0579-ba54-b60b-7083f2293fa8-27: No such file or directory [errno=2].: No such file or directory [errno=2].
Bug24673 - FTI merger can create empty data file
The FTI merger process sometimes fails and logs an error message similar to the following:
Process 'repo FTI Merger' raised: opening
#<mapped-file-simple-stream
#P"/agdata/data/rootcatalog/repo/fti-2-chunk-114-data" mapped
closed @ #x100054959b2> resulted in error (code 22): Invalid
argument. this has been corrected.
Bug24663 - TriG parser uses the wrong graph on the first blank node
In TriG nested blank nodes can be specified in a graph other than the default graph. When the TriG parser encountered the first nested blank node inside a non-default graph the graph for the triples derived from the parsing of that nested blank node would incorrectly be set to the default graph. This has been fixed to use the correct graph.
Bug24642: Incorrect text index query warning
AllegroGraph v6.1.4 added a check and warning for text index queries which could not possibly match. Under certain circumstances this check would warn even though the query could match. This has been corrected.
Bug24623 - SNA path finding in remote-triple-stores did not suppport blank nodes
If a path contained blank nodes, then the SNA path finding functions would signal an error if used on a remote triple-store. This has been corrected.
Bug24615 - Static filter function not re-esablished when attaching to a store
Previously when opening an existing triple store, the static attribute filter was not reestablished. This has been fixed.
Bug24588 - load-nqx does not properly track line numbers
When an error occurs loading extended triples via load-nqx and agload the incorrect line number is reported. This has been corrected.
Bug24548: TransactionLogArchive directive doesn't create catalog and repository directories when tlarchiver is initiated
AllegroGraph did not previously create the TransactionLogArchive directory when starting a database instance. Instead a "no such file or directory" error was logged when the tlarchiver ran. This has been fixed. The directory (and any subdirectories) are now created when a database instance starts, if needed.
Bug24545 - Store specification containing a remote URL could fail
A "missing value for argument db.agraph::user" error would occur when starting a session using a store specification containing a remote URL without authentication information. This has been fixed.
Bug24342 - load-turtle loses default-attributes for RDF collections
Previously the Turtle parser would fail to include attributes when parsing an RDF collection such as:
prefix : <ex://foo#>
:a :p (1 2 3) . This has been fixed.
Bug21595 - agload bad error message when given a filespec that does not exist
The agload program reported misleading error messages when attempting to load files that did not exist or contained wildcards. This has been improved by reporting an accurate and more useful error message. See Data Loading for more information on agload.
HTTP API
Rfe14967 - Ensure that the statements REST API uses namespace abbreviations
The statements REST API was not using user defined namespaces to abbreviate RDF resources in formats that supported them like turtle or TriG. This has been corrected.
Rfe14930 - Improve error message when a file cannot be accessed
The agload process for loading a local file via the HTTP interface reported misleading and inaccurate error messages when the local file was inaccessible. This has been improved by returning an accurate and useful error message in the HTTP response.
SPARQL
Rfe14978 - Improve efficiency for SPARQL datasets with many graphs
AllegroGraph now operates on SPARQL datasets with many graphs more efficiently.
Rfe14975 - SPARQL dataset validator is inefficient for large dataset specifications
The SPARQL engine validates any dataset specification before it begins execution. This change improves the performance of the validation which helps query speed especially when the dataset has more than a few thousand graphs.
Rfe14974 - SPARQL parser is inefficient for large dataset specifications
SPARQL queries with thousands of FROM and FROM NAMED clauses in their dataset description now parse much more efficiently.
Rfe14931 - Improve how property path queries implement filtering
SPARQL Property Path queries now handle FILTERs more efficiently.
Rfe14899 - Warn if query LIMIT/OFFSET and externally imposed LIMIT/OFFSET conflict
Pursuant with the changes made for Rfe14597 (described below), AllegroGraph will now warn if a query's internal and external LIMITs differ or if a both an internal and external OFFSET are supplied.
Rfe14597 - Reduce confusion between internal and external query limit and offsets
Previously, any externally imposed LIMIT or OFFSET would override the LIMIT or OFFSET in the query string. This was most evident in AGWebView because the web interface uses a dropdown selector to pick the LIMIT and this LIMIT overrode any specified in the query.
Now, AllegroGraph will:
- use the minimum of the internal and external LIMITs.
- use the sum of the internal and external OFFSETS.
Bug24684 - Property Paths and MINUS could interact badly
It was possible for queries that used MINUS and Property Paths to generate incorrect answers. This has been corrected.
Bug24610 - Trivial equivalence constraints cause an error.
Queries containing trivial equivalence constraints, such as filter(?x = ?x), used to cause the following error:
missing value for argument db.agraph.sbqe::var2 This has been corrected.
Bug24609 - User attributes set via the SPARQL prefix could be ignored
Depending on how a query was executed, any user attributes set via the userAttributes SPARQL query prefix option could be ignored. This has been corrected.
Bug24608 - Spurious join results.
When performing joins where the left hand side contained unbound values (resulting from the use of OPTIONAL or UNDEF keywords) AllegroGraph could sometimes return the same solution row multiple times. For instance, the following query:
SELECT * {
{ values (?a ?x ?y) {(0 1 2)} }
optional { values (?a ?y ?z) {
(0 2 3)
(0 undef 4)
(undef 2 5) }
}
} Would return 4 results, including two copies of the (0 1 2 5) row. This has been corrected.
Bug24599 - concatenated-cursors don't support extended triples
When a SPARQL query which uses attributes-related magic properties is issued against a graph-filtered triple store, an error like the following can occur:
Executing query failed: #<concatenated-cursor @ #x1000732dc52>
does not support fat-triples This has been fixed.
Bug24538 - Invalid effective boolean value of constant strings in filters
A query like:
SELECT * {
?s ?p ?o
FILTER ("bruce")
} was returning no results when it should have returned every row in the store. Non-blank strings are supposed to have an effective boolean value of "true". Blank strings have an effective boolean value of "false". This has been fixed.
Bug24530 - SPARQL-JSON result format does not handle encoded boolean properly
Previously, boolean values in SPARQL-JSON results were represented as the strings "0" and "1". Now boolean values are correctly represented as "false" and "true".
AGWebView
Rfe14968 - Remove the deprecated planner option from AGWebView
Remove the long deprecated planner option from AGWebView's query page. This option has had no effect for some time.
Rfe13239 - Provide "no limit" option for query results in AGWebView
Subject to the resources of the web browser, AGWebView can now return an unlimited number of results from a query. By default, AGWebView will display 1000 results or use the limit specified by the query (if it is smaller than 1000). Because of this, the More Results button has also been removed. See the WebView document for information on AGWebView.
Bug24616 - N-Triples parser errors mention NQX
When trying to load a .nt or .nq file through AGWebView and a parsing error occurs the error message states that a "NQX" parsing error occurred when it should have stated it was an N-Triples (for .nt) or N-Quads (for .nq) parsing error. This has been fixed to report the correct format type.
Changes to the Lisp API
Bug24669 - serialize-nqx does not work on remote-triple-stores
Serialization of attributed triples was not working correctly for remote-triple-stores. This has been corrected.
Bug24658 - load-nqx fails on remote-triple-stores
An error would occur if no default attributes were passed to load-nqx on a remote-triple-store. This has been corrected.
In order to reduce future confusion, the :attributes argument to the various load functions in the Lisp API has been changed to :default-attributes. Note this is an incompatible change. Code using :attributes will signal an error.
Bug24549: open-store-from-specification does not work
Previously, attempting to open a triple store using a URL could fail like so:
(open-triple-store "http://localhost:10443/repositories/kennedy")
Error: Invalid character in triple-store name
"//localhost:10443/repositories/kennedy".
The name may not contain these characters /, \, ~, :,
Space, $, {, }, <, >, *, +, |, [, ]. This has been fixed.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
Bug24680 - Tonativestring is broken on Python 2
The Python client sometimes failed while processing values with non-ascii characters, showing the following error message:
UnicodeEncodeError: 'ascii' codec can't encode characters in position ??: ordinal not in range(128) This has been corrected.
Bug24443 - getStatements sometimes breaks for raw Python values
The getStatements function can be called with a raw Python value, instead of an RDF term, as its third argument (object filter). For instance, the following call:
conn.getStatements(None, somePredicate, True) would retrieve all triples with the predicate of somePredicate and the object equal to "true"^^xsd:boolean.
This mechanism did not work correctly if the second argument (predicate filter) was None. In that case an AttributeError exception would be raised.
This has been corrected.
Java Client
Rfe14990 - Provide methods to download query results
The Java client now contains methods to download and save query results to a file. The new methods are:
- AGRepositoryConnection.downloadStatements
- AGRepositoryConnection.streamStatements
- AGQuery.download
- AGQuery.stream
Both methods have multiple overloads, allowing for the output path and desired output format to be passed in a variety of ways.
Rfe14970 - Speed up AllegroGraph java client connection pools.
Depending on the connection pool configuration, many unnecessary requests could be made to the AllegroGraph server when borrowing/returning connections to/from a connection pool.
These calls have been cleaned up, resulting in improved connection pool performance.
Bug24648 - inferredGraph parameter missing from AGMaterializer
When materializing triples through the Java API it is now possible to specify the graph into which the generated triples will be added.
Bug24644 - Fix TriG import via the Java client.
The AllegroGraph Java client would fail when attempting to import TriG data due to the content-type "application/x-trig" being unsupported in the server.
AllegroGraph now supports TriG data imported with a content-type of either "application/trig" or "application/x-trig".
nD Geospatial
No significant changes.
AllegroGraph 6.2.1
AllegroGraph Server: General Changes
Rfe14881 - Cross attribute comparisons
AllegroGraph 6.2.0 added a restriction against using an attribute filter operator with two different types. An attempt to perform such an operation would result in a "Comparisons between attributes of different types is not supported" error message.
This restriction has been removed. Note, however, that ordered comparisons between different attribute types is still not allowed since there is no defined ordering between two ordered attributes. Triple attrinutes are discussed in Triple Attributes.
Bug24567 - Initial superuser account file owned by root if configure-agraph run as root
Previously, if AllegroGraph was installed, configured, and started as root, the directories and account file for the initial superuser remained owned by root. This resulted in the following error the first time the superuser made access to AllegroGraph:
mkstemp failed: Permission denied [errno=13] This has been corrected.
Bug24557 - Fix bug restoring repositories with a non-default stringTableSize.
When restoring repositories with a non-default stringTableSize directive to v6.2.0, a bug in an upgrade step would cause the repo's string table to be set to the default size of 16M. If the stringTableSize was larger than this default, the string table would be truncated, causing data loss and possible corruption.
This bug has been fixed.
Bug24548 - TransactionLogArchive directive doesn't create catalog and repository directories when tlarchiver is initiated
AllegroGraph did not previously create the TransactionLogArchive directory when starting a database instance. Instead a "no such file or directory" error was logged when the tlarchiver ran. This has been fixed. The directory (and any subdirectories) are now created when a database instance starts, if needed.
Bug24342 - load-turtle loses default-attributes for RDF collections
Previously the Turtle parser would fail to include attributes when parsing an RDF collection such as:
prefix : <ex://foo#>
:a :p (1 2 3) . This has been fixed.
HTTP API
No significant changes.
SPARQL
Bug24538 - Invalid effective boolean value of constant strings in filters
A query like:
SELECT * {
?s ?p ?o
FILTER ("bruce")
} was returning no results when it should have returned every row in the store. Non-blank strings are supposed to have an effective boolean value of "true". Blank strings have an effective boolean value of "false". This has been fixed.
Bug24530 - SPARQL-JSON result format does not handle encoded boolean properly
Previously, boolean values in SPARQL-JSON results were represented as the strings "0" and "1". Now boolean values are correctly represented as "false" and "true".
Bug24529 - Some queries using VALUES produce wrong results.
Some queries using the VALUES keyword and joins or unions produced incorrect results. More specifically some rows were missing from the result.
Two examples of such queries are:
select ?x { { values(?x) {(1)} } union { values(?x) {(2)} } } and
select ?x ?y ?z { { values (?x ?y ?z) {(1 2 3)} }
{ values () {()} }} The first one would produce only a single result (2), while the second one would not produce any results. This has been corrected.
Bug24528 - Wrong join results when using the CAAT strategy
When using the chunk-at-a-time execution strategy some queries that performed left joins with the identity set would produce spurious results. One such query is:
select ?a ?b ?c {
{ ?a ?b ?c } union { ?c ?b ?a }
optional { bind(42 as ?x) }
} This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
Bug24545 - parse-remote-store-specification returns inconsistent results
A "missing value for argument db.agraph::user" error would occur when starting a session using a store specification containing a remote URL without authentication information. This has been fixed.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
Rfe14695 - Replace pycurl with requests.
The HTTP library used by the AllegroGraph Python client has been changed from pycurl to requests. This change means that the Python client no longer depends on any libraries that require native extensions. This results in a simpler installation process, as it is no longer necessary to have a working C development environment on the target machine.
Note that pycurl will still be used if it is installed, since it might offer better performance in some scenarios.
Bug24582 - AGRAPH_PROXY variable not used
Proxy settings specified through the AGRAPH_PROXY environment variable used to be ignored.
This has been corrected.
Java Client
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.2.0
AllegroGraph Server: General Changes
Rfe14846 - Revised triple attribute filter operators
There are extensive changes to attribute filter operators, allowing better control when comparing user attributes with triple attributes. New operators include
or, not, empty, overlap, attributes-overlap, subset, superset,
equal, attribute-set\<=, attribute-set<, attribute-set=,
attribute-set>, attribute-set>= The existing operator attribute>= has been renamed attribute-set>= (the old name is still accepted but users are advised to switch to the new name). See Triple Attributes for information on attribute filter operators.
Rfe14806 - Audit attribute and static filter definitions
If auditing is enabled an audit record is made for defining an attribute and for defining the static attribute filter. See the Auditing document for more information on auditing.
The rdf:type of the new records are:
<http://franz.com/allegrograph/4.11/audit-log#defineAttribute>
<http://franz.com/allegrograph/4.11/audit-log#deleteAttribute>
<http://franz.com/allegrograph/4.11/audit-log#defineStaticAttributeFilter> Rfe14773 - Changes to agraph-control
The agraph-control program that starts and stops AllegroGraph has a new command restart, which requests that the server shut itself down, if running, and then start back up again. The reload command, which reloaded the configuration file, is no longer supported. See the agraph-control section of the Server Configuration and Control document.
Rfe14797 - Add permissions for user-attribute specification
Added user permissions user-attributes-header (if true, user can send attributes via HTTP as the value of the x-user-attributes header), and user-attributes-prefix (if true, user can specify the user attributes via the SPARQL prefix franzOption_userAttributes. See the Managing Users section of the WebView document for information on user permissions.
Rfe14733 - String comparison operator for attribute filter
Added the attribute= operator which can be used to compare attributes to other attributes, attributes to strings, or strings to strings. See Triple Attributes for information on attribute filter operators.
Rfe14671 - New configuration option StringTableCompression
The StringTableCompression configuration option allows specifying the compression type (including no compression) for the string table. The default is lzo999 compression. See Server Configuration and Control for information on configuration options.
Rfe14661 - agtool combines many command-line programs
The *agtool program replaces the programs agraph-backup, agmaterialize, agquery, agexport, agload, agraph-recover, agraph-replicate, and convertdb. agtool takes as its first argument the task to perform (usually indicated by a shortened version of the old program name but sometimes quite different -- so agraph-backup becomes archive and convertdb becomes upgrade) and then the various argument and options for that task. For example, in order to load the Ntriples file mytriples.nt into the mystore database in the server listening on port 10035, in earlier releases you ran this command:
% agload --port 10035 mytriples.nt mystore now you use agtool as follows:
% agtool load --port 10035 mytriples.nt mystore (Other argument/option combinations will do the same as those shown.) For most tasks, the arguments and options are the same as with the earlier tools, with the following differrences:
agraph-backup (archive), agmaterialize (materialize), The --ext-backup-command is no longer supported. Otherwise arguments and options are the same.
agraph-recover (recover), convertdb (upgrade), these changes: -c accepted for --catalog, -p accepted for --port.
agload (load), this change: -u not accepted as synonym for --base-uri.
agquery (query) and agexport (export), this change: --username is now --user and -u is a synonym of --user.
agraph-replicate (replicate), this change: -c accepted for --catalog, -u accepted for --user.
See the agtool General Command Utility document for more information and links to all the documentation for the individual tasks.
Rfe13254 - Tools for saving and loading metadata
Metadata, that is attribute and static filter definitions, can be saved from a database and loaded into a database. The methods for doing so include using the new --save-metadata option to agtool export (see Repository Export) and the new --metadata option to agtool load (see Data Loading), along with new Lisp functions and REST API, discussed under Lisp API and HTTP API below.
Bug24425 - install-agraph no longer creates a log/ directory
install-agraph would incorrectly create a log/ directory alongside the lib/ and bin/ directories it creates. It no longer does so, leaving the creation of the log directory to tools like configure-agraph, agraph-control, or the agraph server itself.
HTTP API
Rfe13254 - load/save attribute metadata
GET and POST metadata commands allow obtaining metadata (attribute and static filter definitions) from a repo and loading them into another repo. See the Attributes section of the REST/HTTP interface document for details.
SPARQL
Rfe14812 - Support attributes in INSERT DATA and INSERT/WHERE
The syntax of SPARQL INSERT DATA and DELETE/INSERT/WHERE commands has been extended to support the fine-grained specification of attributes on a per-triple basis. See Triple Attributes for more information on attributes.
Rfe14726 - New prefixes for attributes
The defaultAttributes SPARQL query option specifies the attributes to be associated with each triple added via the INSERT DATA, DELETE/INSERT/WHERE, or LOAD commands. The userAttributes query option specifies the user attributes to be used when processing the query. See Triple Attributes for more information on attributes.
Rfe13305 - Old-style SPARQL GEO queries are no longer supported
Because AllegroGraph's magic properties support a richer range of geospatial features than the old-style SPARQL GEO syntax, the old syntax was deprecated in 2014 and has now been removed from the code. Any queries that use the old-style syntax will need to be rewritten.
Magic properties support both the deprecated 2D geospatial encodings and the newer nD ones.
Bug24481 - Wrong results for VALUES with a single variable
When processing a VALUES clause in a SPARQL query AllegroGraph would return incorrect results if there was exactly one variable bound by that clause and the list of values contained either:
An UNDEF value: it would be silently ignored (but other values would still be used).
Duplicates: each value will be matched exactly once, regardless of the number of times it appears inside VALUES.
This has been corrected.
Bug24470 - Spurious results for SELECT DISTINCT on a bound variable
When executing a query which tries to find unique values of an already bound variable like in the example below:
select distinct ?s WHERE {
bind(ex:A as ?s) .
?s ?p ?o .
} AllegroGraph would ignore the binding and return all results that match the pattern. This error could be observed only if the single-set execution strategy was used.
This has been corrected.
AGWebView
Rfe14576 - Make deleting namespaces easier on AGWebView Query page
You can now delete namespaces on the AGWebView Query page.
Rfe14569 - New vmstat-like server performance charts added to WebView
A new menu item in the WebView Utilities menu brings up a page of performance charts on the server machine. The several charts show approximately the same machine load data printed by the vmstat command, charted for the past five minutes. AGWebView is documented in the WebView document. The new Server Load page is documented here in that document.
Changes to the Lisp API
Rfe14797 - New user-attributes-prefix-permission-p keyword argument to run-sparql
The function run-sparql now have a user-attributes-prefix-permission-p keyword argument. It must be given a true value in order to run a SPARQL query including "PREFIX franzOption_userAttributes".
Rfe14754 - Allow #\: to be used as the catalog and repository delimiter
In addition to using the slash character (#\/), the colon (#\:) can now be used to designate a triple-store in a different catalog. I.e., the following are equivalent:
(open-triple-store "test" :catalog "production")
(open-triple-store "production/test")
(open-triple-store "production:test") Note that the format AllegroGraph uses to display triple-stores will now be catalog:name rather than catalog/name.
Rfe14705 - The triple-store-user (db.agraph.user) package uses the sparql package
The triple-store-user package, also named the db.agraph.user package, now uses the sparql package which means that symbols in the sparql package like run-sparql can be used more easily as they do not need to be package qualified.
Rfe14703 - :return-fat-triples keyword argument is now :return-extended-triples
The return-fat-triples keyword argument to get-triples, get-triples-list, and freetext-get-triples has been renamed return-extended-triples. The new functions extended-triple-triple and extended-triple-attributes return the triple and the attributes (respectively) of an extended triple. The new function copy-extended-triple makes a copy of an extended triple.
Rfe13254 - New functions to load/save attribute metadata
The functions get-metadata. set-metadata, load-metadata, and save-metadata allow retrieving metadata (attribute and static filter definitions) from a store and adding metadata to a store. The metadata can be transmitted as a string or written/read from a file.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.1.6
AllegroGraph Server: General Changes
Rfe14785 - Allow default session idle timeout to be set in agraph.cfg
Two new configuration parameters have been defined for controlling the default and the maximum allowed session idle timeout. DefaultSessionTimeout specifies the default timeout and MaximumSessionTimeout specifies the maximum timeout. See the description of those directives in the list of top-level directives in Server Configuration and Control for details.
Bug24519 - Path-finding broken on remote-reasoning-triple-store
Path-finding operations (such as depth-first-search) were broken for remote-reasoning-triple-stores since v6.1.4. This has been fixed.
Bug24389 - Bus error when adding triples with unspecified graph to security-filtered-triple-store
Previously, adding a triple to a security-filtered-triple-store could result in a bus error. This has been fixed.
Bug24221 - String table configuration mismatch during restore.
An error would occur when trying to restore a database that was backed up in AllegroGraph prior to version 6 if the catalog it was restored to had a string table size different than the default 16M entries.
The error was:
> String table configuration mismatch: Configured number of slots X
does not match recovered number of slots (16777216) This has been corrected.
HTTP API
No significant changes.
SPARQL
Bug24463 - Binding produced index out of range error during query
Some queries using BIND used to fail with the above error message. This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Prolog
No significant changes.
Documentation
Bug24427 - Javadocs missing from agraph server distribution
Previously the Javadocs for the AllegroGraph Java client were missing from the installed documentation set. This has been fixed.
Python Client
Rfe14803 - Support for decimal literals
Literal objects constructed from Python's Decimal values will now have the default datatype of xsd:decimal.
Rfe14694 - Drop support for cjson
The Python client no longer uses the cjson library. Python 2.6 users should install the simplejson library instead to avoid a significant degradation in performance. Other versions of Python are unaffected.
Bug24428 - Creating literals from time objects with timezone
RepositoryConnection.createLiteral used to raise an error when passed a datetime.time instance that contains time zone information. This has been corrected - such calls will now return literals of type xsd:time with time value converted to UTC.
Bug24405 - Non-ASCII characters in addData()
The Python client used to raise an exception (specifically a UnicodeDecodeError) if the data passed to addData was a Unicode string containing non-ASCII characters.
This has been corrected.
Bug24322 - Python clients reads imported file into memory
When uploading data to the server the Python client used to read the whole file into memory before sending it. This could cause out of memory errors for larger files. To avoid this issue, the client now sends files in chunks. This method requires only a limited amount of memory.
Java Client
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.1.5
AllegroGraph Server: General Changes
Bug24442 - Fix intermittent query failures that report a "broken pipe" error
A bug was introduced in v6.1.3 where requests could fail inappropriately under certain conditions, with a "broken pipe" error message appearing in agraph.log or reported in AG WebView. This has been fixed.
Bug24412: agload incorrect line number reporting on invalid character
The agload program (see Data Loading (agload)) could report incorrect line numbers if errors were encountered when loading large N-Triples and N-Quads files in parallel. This problem could only occur if the files were larger 4MB. This has been corrected.
HTTP API
No significant changes.
SPARQL
Bug24352 - In FILTERs can be handled very inefficiently
SPARQL queries that constrain a particular binding to a set of values could operate very inefficiently in some circumstances. Such queries include ones that use the IN FILTER or that use the VALUES clause to supply a list of possible matches. The inefficiency has been corrected.
AGWebView
Bug24423 - WebView hangs if HTTP tracing is enabled.
Previously AGWebView would hang on page load if HTTP tracing was enabled. This has been fixed.
Changes to the Lisp API
No significant changes.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.1.4
AllegroGraph Server: General Changes
Rfe14558 - If possible increase the file descriptor limit at startup
If the agraph server is started as root it will set the number of file descriptors agraph can use to 65536 thus not being limited by the default operating system setting (which is usually quite low).
If the file descriptor limit is raised you'll see a line like the following in agraph.log just after the server starts:
[2016-11-08T17:07:23.956 p13419 daemon] Set fd limits soft: 65536, hard: 65536 Rfe11277 - Warn when a freetext query tries to match on things that cannot possibly match
AllegroGraph now generates a query warning when a freetext search expression cannot possibly find matches. For example, the default stop words in an AllegroGraph freetext index include the and an so it is impossible to search for matches on the expression (or 'an' 'the'). This means that a query like
select * { ?s fti:match 'the || an' . } will return no results. Previously, AllegroGraph would have returned no results without generating a warning. Now, AllegroGraph will warn that The freetext expression the || an cannot succeed.
Rfe11046 - Release fds for free text index disk tries
Making use of a free text index will require far fewer file descriptors than before.
Bug24395 - TriG parser errors could print that they were from the Turtle parser
Some TriG parser errors incorrectly indicated that the current parser was using Turtle syntax rather than TriG. This has been corrected.
Bug24349: Transaction log archiving to same filesystem broken
Transaction log archiving was broken in v6.1.2 when archiving to the same filesystem. This problem is now corrected.
Bug23354 - Check SSL cert when frontend starts
If the SSL certificate information you put in your configuration file is not correct AllegroGraph will log an error and exit during server startup. Previously you didn't find out about SSL certificate errors until the first https request came to the server.
Bug24332 - AllegroGraph doesn't fail gracefully if it cannot claim desired ports
Previously, if AllegroGraph was unable to bind the TCP port specified by the Port configuration directive, it would leave some lingering processes after printing the error message. This has been fixed.
Bug24297 - <http://www.w3.org/2005/xpath-functions#:translate> was broken
The XPath translate function could not be used between AllegroGraph 6.0.2 and 6.1. This has been corrected.
If a query attempted to use the translate function, it would get a parse time error.
HTTP API
No significant changes.
SPARQL
Rfe14722 - Improve planning speed for SPARQL queries that use IN filters
The code generated for evaluating SPARQL IN queries has been optimized for some common cases. This will result in faster plan times for such queries.
Bug24383 - Sparql Update can fail if the WHERE part uses string literals
If the WHERE portion of a SPARQL DELETE/INSERT/WHERE command used string literals that were not already interned in the triple-store in FILTER expressions, then it was possible for the UPDATE to fail. This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
Rfe14736 - Add graph parameter to SNA path finding functions
The various SNA path finding functions like all-depth-first-search-paths and bidirectional-distance all take a new graph/s parameter. This can specify either a single graph or a list of graphs that will be used to restrict path search to only triples from these graphs.
Rfe14721 - Path finding functions accept lists of predicates as generators
The Lisp API to AllegroGraph's Social Network Analysis path finding algorithms has been extended. Previously, the functions took a generator that must have been previously defined. Now, you can pass a list of predicates in the place of a generator and these will be used to create a generator on the fly. Note that this added flexibility does come at the cost of between 5 and 10 percent slower performance so predefined generators are to be preferred.
The syntax of the predicate list is as follows. Each element in the list must be one of
predicate URI- (:undirected
predicate URI) - (:subjects-of
predicate URI) - (:objects-of
predicate URI)
For example, we can find the paths between ex:A and ex:B using undirected links on either ex:knows or ex:likes via the following:
(all-breadth-first-search-paths
!ex:A !ex:B
'((:undirected !ex:knows) (:undirected !ex:likes))) This change also adds a new REST API to the path finding algorithms via sna/paths. See the HTTP reference for details.
Bug24293 - Some plain literals could be printed incorrectly
Changes in AllegroGraph 6.1 made it possible for some plain literals to print incorrectly in some situations when using remote-triple-stores. For example, the literal a/b/c could print as a/b"/c.
This has been corrected.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
Rfe14693 - Allow data import from strings
A new method named addData has been added to the connection class. This method makes it possible to add RDF data stored in a string to the store. All RDF data formats supported by addFile can be used.
Rfe14692 - Add more RDFFormats
The only formats supported during data import (addFile) used to be N-Triples and RDF/XML. It is now possible to use other formats, namely Turtle, TriG, N-Quads, NQX and TriX. The addFile method is now capable of determining the correct format based on the filename. When using this capability it is also possible to import gzip compressed files.
bug24379: HTTPS in Python does not work with Basic auth
The Python client used to only allow HTTPS connections if a client certificate was used for authentication.
It is now possible to connect to HTTPS servers by passing either a host value starting with 'https://' or a protocol value of 'https' when creating the server object.
Java Client
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.1.3
AllegroGraph Server: General Changes
Rfe14363: Add binary-only, non-interactive modes for install-agraph
Two command-line options have been added to install-agraph to support non-interactive installation:
--no-configure: do a binary-only install. Skips the configure-agraph step.--supersede: skip interactive prompt and overwrite installation directory if it exists.
Installation and configuration are described in the Server Installation document.
Bug24253 - Triple-store statistics could become stale when new datatypes were added
It was possible for the statistics AllegroGraph keeps on each predicate to become stale if triples were added with heretofore unseen XSD datatypes. This has been corrected.
Bug24245 - Text index word filters and wildcard search do not work together
A freetext index that used word filters like drop-accents or the stemmer could fail to find matches when using wildcard searches. This has been corrected.
HTTP API
No significant changes.
SPARQL
Bug24273 - The SPARQL RAND() function was being evaluated at plan time
AllegroGraph was mistakenly evaluating the SPARQL RAND() function at plan time rather than query execution time. This could lead to incorrect results and has been corrected.
Bug24214 - Querying using the direct reification magic property returns at most one match
AllegroGraph's direct reification magic property only returned the first matching triple. This has been corrected so that all matches are returned.
AGWebView
Rfe14205: Maintain and provide information on live HTTP requests
A new page, Admin -> Requests, was added in AGWebView. It shows all the currently running and recently finished HTTP requests handled by AllegroGraph and its subprocesses.
Bug24256 - Regular users cannot see catalogs
The catalogs list displayed on the main WebView page used to contain only the catalogs for which the current user has 'read' access to all repositories therein.
The list now contains catalogs for which the user has 'read' access to at least one repository therein.
Changes to the Lisp API
No significant changes.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
Bug24300: Python shutdown hang
Previously if a session was created but not closed before Python was ready to exit, the shutdown of Python could be delayed by a very long time. This problem has been fixed. However, it is always best to call closeSession() on a session when it is no longer needed.
Bug24290 - Changes to Python tutorial
The Python tutorial is now configurable using environment varaibles which provide information about the catalog, username, password, etc. See the Python Tutorial for more information.
Java Client
Bug24238 - Java API Unicode issues
The Java client used to incorrectly process non-ascii characters in results of CONSTRUCT queries. Such characters were replaced by question marks. A similar problem could be observed with results of some other calls (e.g. evalInServer) on platforms where the default encoding used by Java is not UTF-8. A notable example is Windows, where the encoding tends to be CP-1252 (although that depends on the language).
Both issues have been corrected.
nD Geospatial
No significant changes.
AllegroGraph 6.1.2
AllegroGraph Server: General Changes
Rfe14561 - Remove the deprecated --encoding option from agload
Previously, the -C/--encoding option was used to allow agload to process N-Triples and N-Quads files encoded in a character encoding other than UTF-8.
This option has been removed. All N-Triples and N-Quads files with non-standard encoding must be converted with a separate tool, such as iconv, prior to being processed by agload. agload is described in the Data Loading (agload) document.
Bug24233 - Install-agraph.sh errors on tutorial directory when overwriting
When installing AllegroGraph to an existing installation directory, an error would occur when attempting to install the tutorial directory. This is now fixed.
HTTP Client
No significant changes.
SPARQL
No significant changes.
AGWebView
Rfe14407 - Use RDF syntax for displaying namespaces in AGWebView
The display style of namespace abbreviations in AGWebView was changed so that they can now be copied and pasted into a SPARQL query.
Bug24194 - Free text search URI incorrectly parsed slashes in WebView
In WebView, URLs of freetext search and script pages were incorrectly parsed if their arguments contained slashes. This has been corrected.
Bug24172 - Namespace sorting and duplicate inconsistencies in WebView
In AGWebView, namespace abbreviations on the Namespaces and Query pages are now sorted alphabetically. Also, adding a namespace with a duplicate prefix is now forbidden. To modify a namespace, delete it and add it, modified, again.
Bug22807 - WebView error when asking for the plan of a SPARQL update command
AllegroGraph signaled an error when asked to display the plan of a SPARQL Update command. This has been corrected.
Changes to the Lisp API
Bug23492 - with-query-environment is not documented
Documentation has now been added for with-query-environment. That macro ensures all cursors created in its body are discarded when the body completes.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
Javascript Client
Bug24207 - Javascript client library could leak cursors
It was possible for scripts using the Javascript client library to leave cursors in an active state. Although these cursors are eventually freed, they use resources and can cause lock contention. The problem is now corrected: Javascript scripts will always close all cursors automatically.
nD Geospatial
No significant changes.
AllegroGraph 6.1.1
AllegroGraph Server: General Changes
Rfe14493 - Print progress messages during long restores.
During a database restore by agraph-backup a progress report is periodically printed to stdout, showing the fraction of the data file which has been processed, the recent processing rate, and an estimate of the completion time. See Backup and Restore.
Bug24168 - Wrong triple count after a delete with duplicate suppression
Deleting an uncommited triple could cause AllegroGraph to report an incorrect triple count (possibly negative) if duplicate suppression was active at commit time.
This has been corrected.
Bug24153 - owl:sameAs interferes with owl:transitiveProperty during materialization
The presence of owl:sameAs assertions could lead to AllegroGraph failing to infer all owl:transitivePropery triples during materialization. This has been corrected. (Note that this problem only occurred using the materializer and not the dynamic RDFS++ reasoner).
Bug24147 - Support "null" as a synonym for the default graph
The remove-graph-uri, insert-graph-uri and using-graph-uri parameters of the SPARQL and Prolog HTTP endpoints now allow the default graph to be referred to by using the string "null" as the URI.
This resolves a problem with the Java client in which SPARQL Update queries issued through the Jena API did not work as expected. Specifically, a WHERE clause would fail to match any triples unless a graph was specified explictly in the query.
HTTP Client
No significant changes.
SPARQL
No significant changes.
AGWebView
Bug24062 - AGWebView uses "Repository" and "Store" interchangeably
In WebView, both "repository" and "store" (and also "triple store) were used to describe repositories. Now, "repository" is used in all places. The documentation has been updated to reflect the change,
Bug24173 - TypeError when going back to query view with saved query result in AGWebView
In AGWebView, going back to a recent query view with cached result caused a harmless error to appear on the browser console. This has been corrected.
Changes to the Lisp API
Bug24179 - Unicode issues in the remote Lisp client.
The remote Lisp client in AllegroGraph 6.1 used to fail in an unpredicatable manner if non-ascii characters were present in query results.
This has been corrected.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
nD Geospatial
No significant changes.
AllegroGraph 6.1
Server Changes
Rfe14282 TBC 5.1.4 plugin
AllegroGraph plugin for TopBraid Composer has been updated to be compatible with TBC 5.1.4. See TopBraid Composer Plugin for more information.
Rfe14279 - agraph-backup should shut down instances
Previously, agraph-backup would leave instances open and lingering after performing a backup or restore operation. This could result in hangs due to memory exhaustion when running backup-all or restore-all operations across many repositories. The likelihood of hang would depend on a combination of repository size and the value of the InstanceTimeout directive.
Now, agraph-backup will attempt to shut down the instance process of all repositories it operates on, as soon as the operation has completed. agraph-backup is described in the Backup and Restore document.
Rfe14273 - Reduce repeated query log messages
AllegroGraph was repeating some messages unnecessarily when logging query execution. This repetition has been reduced.
Bug24130 - Access to the system initfile should be protected
It was possible for any user to access the contents of the AllegroGraph init file. This has been corrected. Now only users with superuser privileges will be allowed to view the file.
Bug24128 - Blank nodes in RDF/XML collections
When parsing an RDF/XML file containing a collection with a blank node as one of its elements, AllegroGraph used to generate an invalid triple (using "li" as the predicate, instead of one of 1, 2, _3, ...).
This has been corrected.
Bug24126 - RDF/XML parser incorrectly handles empty elements.
The RDF/XML parser used to incorrectly handle empty literals encoded as property elements with empty bodies. The parser would generate a blank node instead of an empty literal.
This has been corrected.
Bug24079 - agload with '--error-strategy ignore' and unrecognized file types
agload will now print a warning before ignoring a file of an unrecognized type regardless of the value of the --error-strategy argument. agload is described in the document Data Loading. Example:
agload --error-strategy ignore repo wrong-file
Cannot guess the format of wrong-file. Use --input to specify it.
Load finished 0 sources in 00:00:01 (1.00 seconds). No triples added.
Terminating agload processes, please wait... Compare with previous behavior:
agload --error-strategy ignore repo wrong-file
Load finished 0 sources in 00:00:01 (1.00 seconds). No triples added.
Terminating agload processes, please wait... Bug24066 - Sessions did not close their databases after idle timeout expires
When a session expired after a given timeout, it would not close its database until another session was started. This has been corrected.
Bug23964 - Incorrect escaping of IRIs in Turtle output
AllegroGraph used to incorrectly escape some characters when using the Turtle format. Furthermore, spaces and a few other characters that are not legal in IRIs were not escaped at all. Other characters, such as double quotes, were escaped using an invalid syntax (\" instead of \u0022).
Qualified names were affected by a similar problem.
This has been corrected - all characters are now properly escaped in both IRIs and qualified names.
Bug23960 - gz file import is broken
Importing .gz or .bz2 files using single-threaded import used to cause an error as the file was not correctly uncompressed.
This has been corrected.
Bug23828 - CSRF vulnerability in file upload
The import dialog in AGWebView was not adequately protected against Cross-Site Request Forgery (CSRF) attacks. When a logged-in user visited a malicious web site, an attacker could import arbitrary triples into any store the user can access.
This has been corrected - a CSRF attempt will now result in a password prompt.
Bug23820: Prevent repository access when a restore is in progress
Previously, an attempt to access a repository while it was in the process of being restored could result in random errors and problems, including being unable to close a repository. This change improves the way in which AllegroGraph notes that a repository is being restored, and improves the check for restore-mode so that these issues are prevented.
Bug23705 - A triple-store must be open to define SPIN magic properties
Changes in AllegroGraph 6.0 required that a triple-store be open before any SPIN magic properties could be defined. This has been corrected.
Bug23339 - Canonical escaping in N-Triples
AllegroGraph used to produce invalid escape sequences in URIs when printing triples in the N-Triples format. Some reserved characters, such as double quotes, were escaped with a backslash. Others were not escaped at all.
This has been corrected. All reserved characters are now properly escaped using the \u notation.
In addition, escaping of strings in N-Triples has been modified to make the output match the canonical form described in the standard. The most notable change is that non-ascii characters are no longer escaped.
Bug23181 - Consistency bug when backing up deleted bitmap
If a triple was deleted while backup was in progress, that triple would also be marked as deleted in the restored store. This has been fixed, backup now represents a consistent state of the store at the time it was started.
Bug22115: Fix configure-agraph handling of passwords.
configure-agraph could crash when entering passwords that contain certain non-alphanumeric characters in them. This bug has now been fixed.
Bug21655 - serialize-rdf/xml can emit illegal xml characters
In XML, control characters (e.g. NUL, BEL, STX, ...) cannot be represented in any form. The RDF/XML serializer (with the Lisp function serialize-rdf/xml, used by agexport when passed --output rdfxml) used to silently output such characters, producing files that could not be parsed.
This has been corrected - if the serialized data contains any invalid characters an error will be reported, as in the following example:
bash-4.2$ agexport -o rdfxml store output.rdf
open the triple-store store on port 10035 and export to ...
Warming up the store
Error Unable to escape character U+0008 (Backspace). occurred during processing
logging error to /.../agexport.log
Bug20220 - Fix bug when validating the --loaders argument to agload
The --loaders argument to agload accepts as a value a positive integer between 1 and 32. On machines with a large number of cores, agload would default to a number of loaders greater than 32 and then error due to the value being too large. This validation bug is now fixed.
HTTP Client
Rfe14445 - Add REST API for [catalogs/CATNAME]/respositories/REPONAME/exists
AllegroGraph's REST API now includes
[catalogs/CATNAME]/respositories/REPONAME/exists which can be used to determine whether or not the named repository exists. It will return a 404 code if the repository does not exist and a 200 response with the body true if it does.
Bug23942 - Handle HTTPS scheme for SPARQL UPDATE LOAD command
The SPARQL UPDATE LOAD command would fail when using the HTTPS URL scheme. This has been corrected.
Bug23817 - Running a query that used MINUS repeatedly via REST could error
Under certain circumstances, if a SPARQL query that used MINUS was evaluated more than once using the HTTP REST API, AllegroGraph could signal an error. This has been corrected.
SPARQL
Rfe14278 - Improve SPARQL SERVICE efficiency for AllegroGraph endpoints
AllegroGraph now uses a more efficient binary data format when communicating with AllegroGraph-based SPARQL endpoints. This can lead to large performance improvements for SPARQL SERVICE calls.
Rfe14257 - DISTINCT|REDUCED mode should be passed on to SERVICE calls
When the main query uses DISTINCT or REDUCED, any use of SPARQL federation will pass on the modifier to endpoints. This can greatly reduce overall IO.
Rfe14254 - SERVICE should remove duplicates from any VALUES it passes along
If the main query is using the DISTINCT or REDUCED modifier, then the SERVICE requests should remove duplicate rows from the list of VALUES that is passed to the endpoint. This can make the query much more efficient.
Rfe14086 - New SPARQL functions for performing haversine calculations.
These SPARQL functions accept various combinations of latitude/longitude coordinates and nD encoded literals. They compute distances on the Earth's surface.
- haversineLatLonLatLon
- haversineLatLonLoc
- haversineLocLoc
The individual names indicate the required argument types: a Lat or Lon must be a number interpreted as degrees. A Loc must be an nD literal that encodes a latitude and a longitude. Its nD encoding must be registered in the open database, but is not needed in any of the function calls themselves.
Each of these functions accepts an optional additional argument specifying the units of the computed result. It may be :km (default) or :miles expressed as a resource in the keyword namespace, i.e., <http://franz.com/ns/keyword#km> or <http://franz.com/ns/keyword#miles>.
See the SPARQL functions list for a list of SPARQL function with links to further documentation.
Rfe11390 - Modify SPARQL 1.1 lexer/parser to match changes to standard
The SPARQL parser used to fail when the input contained any:
- Qualified names containing backslash escapes in the local part, such as
ns:something\#\. - Qualified names containing percent escapes in the local part, such as
ns:something%23%2E - '.' not followed by whitespace, for example:
?s ?p ?o.filter(?o > 42) - Some expressions containing longer products, such as:
select ?x {bind (1+1*1*1 as ?x)}
All mentioned cases are now parsed correctly, in accordance with the SPARQL grammar.
Bug24161 - Correct possible segmentation faults during SPARQL DELETE operations
Under some circumstances, a SPARQL DELETE command could cause a segmentation fault. This has been corrected.
Bug24103 - Correct problem in nD magic property SPARQL queries
It was possible to get an error when executing a query that combined nD magic properties and other patterns. This has been corrected.
Bug24030 - An empty IN filter is ignored if there are any other IN filters
If a SPARQL query contained two or more IN filters that were associated with the same triple pattern and one of the IN filters was empty, then both IN filters were ignored. This has been corrected.
Bug24026 - Fix query failure when using DISTINCT, SERVICE and magic predicates
It was possible for a query to fail during planning if it used multiple SERVICE clauses, the service clauses used magic predicates, and the query asked for DISTINCT results. This has been corrected.
Bug24020 - Property path queries in saved plans with Chunk-at-a-Time engine can lose solutions
It was possible for some solutions to be lost on subsequent runs of a saved query plan using the Chunk-at-a-Time execution engine if the query used SPARQL OneOrMorePath or ZeroOrMorePath property path expressions. This bug sometimes appeared when running queries in AGWebView because it directs AllegroGraph to use saved plans. This has been corrected.
Bug23988 - Fix query failure in federations of reasoning and non-reasoning stores
Some queries could fail with an error in a federated triple-store composed of a mix of reasoning and non-reasoning leaf stores. This has been corrected.
Bug23949 - Filtering on multiple language tags could fail
The FILTERing machinery in AllegroGraph's SPARQL engine could fail to correctly process collections of language tag checks. For example, a query with a filter like:
FILTER( lang(?x) = 'fr' || lang(?x) = 'en' ) could fail. This has been corrected.
Bug23897 - Query warnings are incorrectly cached
Query warnings saved in SPARQL plans were not being updated to take account of changes in the underlying store.
Bug23893 - Federated-triple-stores and statistics caching could lead to poor plans
The caching of store statistics could lead to incorrect plans when querying a federated-triple-store. This has been corrected.
Bug23889 - Correct breakage in solR magic property queries
The SPARQL magic properties for solR queries were broken in AllegroGraph 6.0. This has been corrected.
Bug23884 - Constant pattern only CONSTRUCT could fail when executed via a saved plan
If a CONSTRUCT query template contained only constant patterns (i.e., no variables), and there was a limit on the query, and the query was being executed via a saved plan, then only the first execution would return results and subsequent executions would return nothing. This has been corrected.
Bug23882 - Algebraic query combines some BGPs too aggressively
The query planner was sometimes combining BGPs and inadvertently producing cross-products which could lead to slower query performance. This has been corrected.
Bug23881 - FILTER clause could be lost in complex SPARQL queries
It was possible for a FILTER clause to be lost during query planning. This has been corrected.
Bug23873 - Neither limit nor planner should require eval permission
Users without 'eval' permissions were unable to choose a planner or limit for a query. This has been corrected.
Bug23870 - Restriction reasoning could cause error during SPARQL query
Depending on the way it was invoked, a SPAQRL query could signal a error when restriction reasoning was used. This has been corrected.
Bug23841 - Do not allow triples in query results to be deleted
It was possible to remove triples from the database by clicking the X icon next to CONSTRUCT or DESCRIBE query results. This possibility has been disabled. CONSTRUCT and DESCRIBE queries do not return enough information to uniquely identify a triple in the triple-store. Offering to delete them is error prone and ambiguous. To delete triples, either use a SPARQL DELETE statement or the node view (click any part of a triple in the results).
Bug23755 - Problem with alternating property path in sub-queries
A sub-query that used a property path pattern with alternation could incorrectly bind the predicate of matched triples to output variables. An example of the problem would have been:
select ?s ?o {
{ select ?s ?o {
?s :hasChild | :hasParent ?o .
} }
} This query would bind the predicate of matched triples to ?o. This has been corrected.
AGWebView
Rfe14371 - Allow the number of loaders to be selected in AGWebView
It is now possible to specify the number of loaders when invoking agload through AGWebView.
Rfe13428 - Display query information for CONSTRUCT, DESCRIBE and ASK queries
The query information tab is now displayed alongside results of CONSTRUCT, DESCRIBE and ASK queries.
Bug24047 - WebView: download duplicates ignores mode
The Export duplicate statement command in Store Control ignored the choice of SPO/SPOG. This has been corrected.
Bug23948: Fixed paged behavior after using the browser's back button
Previously, the Download and Load more buttons would function incorrectly if used after clicking the browser's back button. This has been corrected.
Bug23888 - Upload progress indicator not showing
AGWebView will now display progress information when uploading a file. This was previously not visible due to a bug. Progress is reported as a text line similar to this:
Uploading file triples.nt(1475 Kb uploaded) Bug23887 - Automatic scrolling in the AGWebView view server log pane broken
The server log in AGWebView would not auto-scroll when new log data appeared. This has been corrected.
Bug23857 - Load more does not work in Free Text search
Loading the next page of Free Text search results did not work. This has been corrected.
Bug23856 - Repeated text search breaks navigation
Page navigation would sometimes break when a user entered the same query twice in the Free Text search field. This has been corrected.
Bug23840 - Wrong even/odd row coloring when odd limit chosen
Row coloring could appear incorrect when using an odd limit and paging in the query view. This has been corrected.
Bug23839 - Planner selector grayed-out after language detection
It was possible to disable the planner selector in the query view if language autodetection was used. This has been corrected.
Bug23827 - Process list in WebView crashes on Firefox
The process and job list views of AllegroGraph WebView did not display any processes or jobs on Firefox. This has been corrected.
Bug23811 - AGWebView import fails for RDF/XML
Multi-core file import in AGWebView would fail if the chosen format was RDF/XML. This has been corrected.
Bug23810 - Importing via file upload no longer works in AGWebView
The multi-core file import feature in AGWebView was not working unless the file type was set directly. This has been corrected. In addition, AllegroGraph now recognizes application/n-triples and application/n-quads as media-types for the import of N-Triples and N-Quads (respectively).
Bug23800 - Query progress spinner only appears on every other query in Agwebview
The "Loading..." indicator was not displaying the first time that a new SPARQL query was executed via AGWebView's query editor. Now the loading indicator is shown every time.
Bug23774 - CONSTRUCT queries not displaying results in AGWebView
Results from CONSTRUCT queries were not being displayed in Safari under OS X. This has been corrected.
Bug23752 - Agwebview Lisp editor fails when digits are at the end of a line
The Lisp and Prolog script editors would become unresponsive if the last token on a line was a number. The JavaScript console showed the following error:
TypeError: undefined is not an object (evaluating 'stream.peek().toUpperCase') This has been fixed.
Bug23141 - AGWebView's CONSTRUCT doesn't show the MORE button
Previously, if the number of results of a CONSTRUCT query exceeded the limit set by the user, there was no way to show more results. Now, there is a MORE button at the bottom of the results which functions similarly to the one used for SELECT queries.
Bug22314 - Federated-triples-stores, encoded IDs and AGWebview don't mix
Attempting to use AGWebView to query a federated triple-store whose leaf nodes used encoded IDs led to an error. This has been corrected. Note: Correct behavior of a federation of stores that use encoded IDs assumes that each leaf node has the same definitions for encoded ID prefixes.
Changes to the Lisp API
Rfe11567 - Support store specifications in the Lisp client
In addition to using the catalog keyword argument, AllegroGraph now accepts store names in the form CATALOG-NAME/STORE-NAME. E.g.,
(open-triple-store "general/test-data") would open the store test-data in the catalog general.
You can also supply a triple-store specification in calls to open-triple-store. This will build a store instance from the specification, bind it to *db* and return it. In this case, the store instance will be given a name like store12341.
Finally, you can query store specifications directly using any of the HTTP clients by using a URL encoded version of the specification in place of a simple store name. E.g.,
curl -X GET -v "http://localhost:11040/repositories/%3Ca%3E%2B%3Cb%3E?query=select%20*%20%7B%20%3Fs%20%3Fp%20%3Fo%20%7D%20limit%205" would send the query
select * { ?s ?p ?o } limit 5 to the federation of triples stores a and b (as specified by <a>+<b>).
Note that in this case, AllegroGraph will not start a new session so Prolog functors or other things defined in the servers initialization file will not necessarily be present.
Bug24105 - Deprecate serialize-rdf-n3 in favor of serialize-turtle
AllegroGraph can serialize data in turtle format but not in N3 so it is better to use the correct name. The now deprecated function serialize-n3-rdf will be removed in a future release and should no longer be used. serialize-turtle should be used instead.
Bug24006 - pcache Lisp module conflicts with AllegroGraph
A package conflict made it impossible to load AllegroCache and pcache in the same Lisp execution as AllegroGraph (without tiresome package manipulation). The problem has been fixed in this release.
Bug23846 - warmup-triple-store does not work in remote-lisp-client
Calling warmup-triple-store on a remote store used to have no effect. The call did not fail, but no operation was performed. The remote client will now correctly pass the request to the server it is connected to, preparing it for subsequent operations.
Prolog
Bug23877 - Prolog select exposes a security hole
There was a remote code execution vulnerability in Prolog query handling code where users without eval permissions could execute arbitrary code on the server. Executing Prolog queries now requires eval permissions. (This bug was fixed in various maintenance releases including version 6.0.2. We repeat it here because of its importance.)
Documentation
Bug23874 - Documentation search in local AllegroGraph docs doesn't work
The documentation search field in the upper left corner of AllegroGraph documentation pages now uses Google to search for the given phrase within:
franz.com/agraph/support/documentation/<version>/ where <version> is vX.Y[.Z] (v6.1 in the case of the current release). This means that even if you have a local copy of the documentation the search is done in the web copy on the Franz Inc. website.
Python Client
RepositoryConnection.createCoordinate arguments renamed
The lat and long arguments of RepositoryConnection.createCoordinate have been renamed latitude and longitude.
Bug23901 - RepositoryResult.asList() always returns None
The asList method in the RepositoryResult class was buggy and always returned None. It now correctly returns all statements from the result object.
Java Client
No significant changes.
Geospatial
Rfe12345 - Unnecessary geospatial functions removed
The functions geospatial-subtype-index-in-this-image and geospatial-subtype-index-in-db are neither useful nor necessary in the geospatial API. They have been unexported and removed from the documentation.
Bug23933 - Geospatial subtypes could be mistranslated on remote stores
It was possible for encoded geospatial data to be mistranslated when connecting to AllegroGraph via a remote-triple-store in the Lisp environment. This has been corrected.
Bug23848 - nd:inCircle :units incorrectly defaults to :miles, not :km
The :units argument to get-triples-nd-within-radius in Lisp and the similarly named functor in Prolog, and nd:inCircle in SPARQL, incorrectly defaulted to :miles instead of :km. This has been fixed so the behavior now agrees with the documentation. Since a kilometer is less than a mile, the change may cause previous queries that did not specify :units now to return fewer results. The extra results before the change were bogus.
Bug23767 - automate-nd-datatype-mappings could fail under parallel load
Spurious metadata conflict errors could be signaled if a store using automatic nD-datatype mappings was being accessed from multiple clients (for example, via agload). This has been corrected.
Bug23584 - nD query support for ISO6709 time
In an nD query, the min and max limits for a :time type ordinate may be expressed as a real number, as a dateTime UPI, or as a string literal in ISO6709 syntax (either as an xsd:dateTime typed string or an untyped string). These are supported in both SPARQL and Lisp/Prolog queries. Previously ISO6709 was not accepted in all contexts.
AllegroGraph 6.0.2 user-visible changes
Prolog
Bug23877 - Prolog select exposes a security hole
There was a remote code execution vulnerability in Prolog query handling code where users without eval permissions could execute arbitrary code on the server. Executing Prolog queries now requires eval permission. User permissions are discussed in the section Managing Users in the WebView document.
AllegroGraph 6.0.1
Server Changes
Bug23796: Disk-chunks-search.*.tmp files leaked when closing database
Starting in v6.0, files matching the pattern disk-chunks-search.*.tmp would remain open after closing a database. If a database was opened and closed frequently, this would eventually exhaust all available file descriptors in the process. This has been fixed.
Bug23715 and Bug23748 - FTI bugs
Bug23715: Creating a text index in which a word occurred in more than two million triples could sometimes crash the server.
This resulted in a log message similar to the following:
Process 'db-name FTI Merger' raised: opening #<...> resulted
in error (code 22): Invalid argument. Bug23748: Indexing a very long document might cause a crash.
An error would sometimes occur when trying to add a triple with very long text to a text index. The error looked like this:
Error: opening #<mapped-file-simple-stream ...>
resulted in error (code 22): Invalid argument. Both bugs have been corrected.
Bug23010 - Rolling back a freshly staged text index leaves a file around
When a text index was deleted or its creation rolled back, a relatively large (~5MB) file used to be left behind and only deleted at the next checkpoint. This has been changed. The file is now deleted immediately as long as there are no live sessions that could still "see" the index before deletion.
HTTP Client
Bug23589 - Sending a SPARQL Update command to the Prolog REST API generates a backtrace
Sending a Prolog query that is not a valid list (e.g. because it is actually a SPARQL query) to the Prolog REST API caused the system to produce a backtrace and return error 500 with the following message:
Attempt to take the cdr of db.agraph.query.prolog:select
which is not listp. This has been corrected: the server now answers with code 400 and the following message:
MALFORMED QUERY: Prolog queries must be lists.
SPARQL
Bug23789, Bug23788 - issues with nD geospatial SPARQL queries
SPARQL n-Dimensional geospatial magic properties were signaling an error when used inside of a GRAPH clause. They were also signaling an error when the magic property was used with no variable bindings (i.e., when all of the arguments in the pattern were fixed). Both of these issues have been corrected. (Repeated under nD Geospatial below.)
AGWebView
Bug22920 - Agwebview result list has dynamic height
The window with SPARQL query results in AGWebView used to have a height of 300 pixels. The height is now dynamically adjusted to fill the available viewport.
Bug22604 - Improve handling of invalid user files
Previously, an invalid user file (e.g., one that was zero length or corrupted) would cause AGWebView to become unusable. This has been corrected. Now, invalid files will be ignored. Note that this means they will not be displayed in AGWebView's user list.
Changes to the Lisp API
Bug21770 - Some old SPARQL functionality now deprecated
Using either of the extendedp and permitted-verbs arguments to run-sparql will now signal a warning. These parameters will be dropped completely in a future release of AllegroGraph. This change also removes the special variable sparql:*use-extended-sparql-verbs-p*.
Prolog
Bug23745 - Improve future-part handling in Prolog select queries
It was possible for a Prolog select query to fail to find results if future-parts were used inside lisp expressions in the query. This has been corrected.
Bug23726 - Prolog select queries that involved type mapping could fail
Under some circumstances, a Prolog select query that used constants requiring type mapping (e.g., '1234'^^xsd:integer) could fail to return results. This has been corrected.
Documentation
No significant changes.
Python Client
Bug22303 - python-client: missing not-equal method for URIs
Trying to test if two URI objects in the Python API are different used to be impossible (i.e. code such as uri1 != uri2 raised an exception). The missing comparison method has been implemented. In addition, it is now possible to order URIs, literals and BNodes. Comparing values of two different types yields an arbitrary but consistent result under Python 2 and causes an exception under Python 3. This follows conventions established by Python's built-in types.
Java Client
No significant changes.
nD Geospatial
Bug23789, Bug23788 - issues with nD geospatial SPARQL queries
SPARQL n-Dimensional geospatial magic properties were signaling an error when used inside of a GRAPH clause. They were also signaling an error when the magic property was used with no variable bindings (i.e., when all of the arguments in the pattern were fixed). Both of these issues have been corrected. (Repeated under SPARQL above.)
AllegroGraph 6.0
Server Changes
Use agraph for the Lisp client fasl name
AllegroGraph 6.0 now uses agraph.fasl instead of agraph[major-version-number].fasl for the name of the Lisp fasl file. Both the direct Lisp client fasl file and the various remote Lisp client fasl files (on the various platforms suitable for use as remote clients) are named agraph.fasl. Users need to manage the locations of the various individual agraph.fasl files to be sure the correct one is loaded when used.
Rfe13969 - Add HDFS input support to agload
agload now supports loading files from HDFS sources. For this to work, there must be a working "hdfs" program in your PATH and HDFS input files must be specified on the agload command line using the hdfs:// prefix. For example:
agload repo hdfs:///user/bruce/lubm-50.nt.gz agload is described in the Data Loading document. (This change was also in release 5.1.1, as noted below, but we repeat it here as some users may not have installed version 5.1.1.)
Rfe13826 - Improve estimates on federated remote stores
Cardinality estimation of query result sets on remote stores has been greatly improved which leads to more efficient query plans on federated remote stores.
Rfe13807 - Added deleted triple purge operation
When triples are deleted they are not immediately removed from indices. They must remain in all indices until all transactions which may potentially need to see those triples (such as transactions that started before the triples were deleted) have completed (committed or rolled back). A triple that cannot possibly be accessed by any live transaction is referred to as an inaccessible triple. The existence of inaccessible triples in indices wastes space and impacts query performance. Therefore AllegroGraph now provides a facility to purge these triples from indices. See Purging Deleted Triples.
Rfe13806 - Configuration option to control index style
AllegroGraph v6.0 adds a new style of index. The old style is called style 1 and the new style is called style 2. Style 2 indices are faster than style 1 when performing point queries where all of s/p/o or s/p/o/g are fixed (where the result is typically a single triple), but these indices take longer to construct and to optimize. Style 1 indices are better for queries which end up scanning many triples and typically produce results with many triples.
The Style2Indices catalog configuration directive can be used to control which index flavors should use style 2. The directive value can be a comma- or space-separated list of index flavor names, or the value none. The default value is none. Configuration options are discussed in Server Configuration and Control.
Examples
Style2Indices none
Style2Indices posgi
Style2Indices spogi, posgi The Lisp API function add-index and the REST interface command PUT /repositories/[name]/indices/[type] allow for specifying the index style as well.
Rfe13641 - Change default for temporaryFileDiskSpace
AllegroGraph now defaults to using no more than 25% of the available filesystem space for temporary query files (at the time the query begins). This setting can be changed using the temporaryFilesystemSpaceLimit query option either in the AllegroGraph configuration file or via a PREFIX option to a SPARQL query. If a query attempts to use more space than this limit allows, it will be canceled. temporaryFilesystemSpaceLimit is a new name for the temporaryFileDiskSpace option. The old name has been deprecated. See agcontrol.
Rfe12000 - Minor improvements in how typed string literals are stored
AllegroGraph now stores typed string literals like "hello"^^xsd:string more efficiently.
Rfe11116 - Support SHA384 digest function in SPARQL
In addition to MD5, SHA1, SHA256, and SHA512, AllegroGraph's SPARQL engine now supports SHA384.
Rfe11078 - Improve agload verbosity behavior
Previously, passing -v or --verbose to agload (see Data Loading) could actually decrease its level of verbosity. This has been fixed. The new verbosity levels are:
0 (-v not supplied): Only report periodic load rate information.
1 (-v) : As above, plus print the job option summary
before starting the operation.
2 (-v -v) : As above, plus print the name of every file
that has been processed. (This change was also in release 5.1.1, as noted below, but we repeat it here as some users may not have installed version 5.1.1.)
Bug23660 - AuditEventsToEmail parser too strict
Previously the events list in an AuditEventsToEmail configuration directive had to be comma-separated with no intervening whitespace. This has been relaxed.
Bug23654 - agraph-backup restore fails if StringTableSize specified
An error would occur when trying to restore a database if its StringTableSize did not match the one in the catalog into which it was being restored. The error was:
Fetching user information failed: String table configuration mismatch:
Configured number of slots (X) does not match recovered number
of slots (Y). This has been corrected. agraph-backup is described in Backup and Restore.
Bug23604 - Improve bidirectional search speed when no paths are found
AllegroGraph's bidirectional search algorithm was expanding more nodes than necessary when no paths between the start and end were found. This has been corrected.
Bug23598 - Error when accessing a suspended account
Previously, if an account was suspended but AccountUnsuspendTimeout wasn't configured, the following error would be signaled when accessing the suspended account:
nil is not of the expected type number This has been corrected.
Bug23585 - Audit dialog not showing any results
It was possible for the Audit Log query in AGWebView to fail to display results. This has been corrected.
Bug23526 - Fixed size deleted triple bitmap
Previously an error would occur if a triple with id greater than 34,359,738,367 was deleted. This has been fixed.
Bug23512 - Signal an error when attempting a freetext query and no freetext indices exist
Previously, AllegroGraph would simply return no results if a freetext query was executed and there were no freetext indices defined. Now, AllegroGraph will signal an error. This also modifies the behavior of list-freetext-indices for federated triple-stores. Previously, a federated store would return the intersection of its freetext indices. Now, it returns the union. This makes it easier to use the freetext indexing facility with them.
Bug23442 - Full filesystems can lead to truncated user files
In some cases a full filesystem could cause AllegroGraph user preference files to be truncated. This has been corrected.
Bug23441 - Retry tlog creation if filesystem is full
Previously, if the filesystem ran out of space while allocating a new tranasction log file during a checkpoint or commit, the instance would crash. Now the affected process will block, log a message to agraph.log, and periodically retry the operation until it succeeds. To avoid confusing hangs, failure to preallocate transaction log files on database creation will still result in an immediate error.
Bug23370 - N-Triple, N-Quad and Turtle parsers should handle UTF8 files with a BOM
AllegroGraph's parsers did not correctly parse UTF8 files that started with a byte order marker (BOM). This has been corrected.
Bug23270 - flush-in-memory-chunks deadlock
Previously, when performing duplicate triple deletion, it was possible for the AllegroGraph instance to become deadlocked and hang. This has been fixed.
Bug23361 - N-Triples parser could incorrectly import non 7-bit ASCII data
Unless a UTF-8 encoding was explicitly specified, the AllegroGraph N-Triples parser could fail to handle Unicode N-Triples data correctly. This has been corrected.
Bug23238 - Fix 'UPI has not been added to string dictionary' bug
Previously, certain queries involving large intermediate result sets could fail with a UPI has not been added to string dictionary error. This bug has been fixed.
Bug23233 - Fix bug in agraph-backup progress output.
Some garbled output showing a "Printer error" could be produced when running a backup operation using agraph-backup. This has been corrected.
Bug23231 - Prevent spurious output from agraph-replicate
Previously, agraph-replicate would emit spurious output when exiting. This output has been suppressed.
Bug23229 - agraph-replicate --stop would not stop replication.
Previously, agraph-replicate --stop would not actually cause replication to stop. This has been corrected. See Replication for information on agraph-replicate.
Bug22490 - Fix potential deadlock when creating/modifying freetext indexes.
With proper timing, it was possible hang the server when adding or updating a freetext index. This has been corrected.
Bug22489 - agraph-backup fails due to missing tlog file
Under certain circumstances agraph-backup could fail to back up a database due to not finding a transaction log file. This error has been corrected.
Bug22165 - commit-time duplicate suppression does not operate under lock
Previously, commit-time duplicate triple suppression did not properly consider triples that were added in a concurrent transaction, possibly resulting in duplicate triples being added. This has been corrected.
Bug19831 - xsd:booleans are now stored as encoded UPIs
Previously, xsd:booleans were stored as strings. They are now stored as encoded UPIs. This will make some queries slightly more efficient.
HTTP Client
Rfe13941 - Allow specification of chunk style when adding an index
The PUT command
PUT /repositories/[name]/indices/[type] now accepts a style query parameter which may be 0, 1, or 2. See Triple Indices for a discussion of index styles.
Rfe13750 - Log errors during writing the HTTP response
While using the REST API, which most clients do, serving a request usually has a computation phase that's followed by an output phase. If an error happens in the computation phase, the appropriate HTTP status code is returned. The tricky case is with chunked encoding (for example, streaming query results back) when some results were already written to the socket. Then upon encountering an error, it is not possible to change the HTTP status code and return a meaningful error message. For this reason, errors should normally be detected during the computation phase.
With this change, any errors (except those caused by the client end of the socket being closed) encountered during the output phase are logged to make debugging easier.
Rfe13711 - Add query PREFIX option to cancel queries when there are warnings
AllegroGraph has a new query PREFIX option: cancelQueryOnWarnings. If true, then a SPARQL query will cancel itself if it encounters any warnings during query planning or execution. Typical warnings include:
- unknown or out of scope variables in FILTER expressions
SELECT * { ?s ex:Predicate ?o . FILTER( ?missing > 2 ) } - constants that are not in the underlying store:
SELECT * { ?s ex:Predicate ex:thisResourceIsNotInTheStore . } - unknown variables in an ORDER BY clause:
SELECT * { ?s ?p ?o } ORDER BY ?age - cross products:
SELECT * { ?a ?b ?c . ?d ?e ?f }
If the option is set to no, then warnings will be logged but query execution will continue.
Bug23546 - Problems adding very long objects with JSON upload
An error could occur if triples with very long strings were imported using the REST API and the application/json content-type. This has been corrected. Other content-types were not affected by this bug.
Bug23324 - Some text index properties could not be viewed
If an AllegroGraph freetext index specified that it should only index literals with one or more particular datatypes, then that index would signal an error when its properties were read using the REST API. This has been corrected.
SPARQL
Rfe14076 - Optimize RDF serialization of CONSTRUCT and DESCRIBE queries
RDF serialization formats like RDF/XML or Turtle were spending more time than necessary trying to find namespace abbreviations. This has been improved.
Rfe13998 - agquery can now interact with SPARQL endpoints
AllegroGraph's command-line agquery tool can now send queries to SPARQL endpoints and retrieve the results. agquery is described in the Querying using agtool document.
Rfe13975 - Streamline internal AllegroGraph merge operations
Query operations that require merging of multiple result streams have been optimized. Examples include the inference of types and properties in the dynamic reasoner and computing distinct SPARQL results.
Rfe13519 - General low-level SPARQL query optimizations
Modified the low-level cursor operations in the SPARQL query engine so that they perform between 3 and 5% better for most queries.
Rfe13387 - Improve performance of SPARQL replace function
The SPARQL replace function is now significantly faster when called with a constant regular expression.
Rfe13327: Improve parsing of diskChunkRowCount and other query parameters
It is now possible to specify query parameters like diskChunkRowCount and other query parameters using sizes like 100m or 1g rather than needing to write out the numeric value as an integer.
Rfe12143 - Improve analysis of constant BIND and VALUES forms
AllegroGraph now does a better job of analyzing and using constant BIND values and single variable VALUES forms. When possible, these forms are pulled into the triple-patterns in the BGPs rather than being evaluated at run-time. For example, both of these queries will now run much more efficiently:
select * {
bind( 'test' as ?test )
?s :predicate ?test .
} and
select * {
values ?test { 'test' 'two' 'three' }
?s :predicate ?test .
} Rfe12131 - Remove support for the old SPARQL 1.0 query engine
AllegroGraph no longer supports the older and long deprecated SPARQL 1.0 query engine. The newer 1.1 engine is to be preferred in all cases. In particular, this means that you can no longer force the use of the 1.0 engine via the queryEngine PREFIX.
Bug23696 - Improve query plan for some property path queries in sub-queries
AllegroGraph could fail to optimize a simple alternation property path query when it appeared inside of a sub-query. This has been corrected.
Bug23653 - Remove spurious cross-product warning for (optional identity A)
AllegroGraph was signaling a cross-product warning for SPARQL queries that contained an OPTIONAL with no left hand side. For example:
select * {
optional { ?s a :Animal }
} has a query algebra like:
(left-join identity BGP) Though the warning is technically correct, the construct above is harmless and the warning is unnecessary.
Bug23632 - SPARQL parser has trouble with escaped backslashes at the end of a literal
The SPARQL parser would fail to correctly interpret literals with an escaped backslash as their final character, e.g., 'abc\'. This has been corrected.
Bug23616 - COUNT aggregation counts unbound values from unknown variables
If a COUNT aggregration expression included variables that were not projected from the underlying query, then AllegroGraph was treating them as if they were bound and returning a count. This has been corrected.
Bug23582 - SPARQL could lose results when a sub-query used limit and distinct
It was possible for solutions to be overwritten and lost when a sub-query used both LIMIT and DISTINCT. This has been corrected.
Bug23581 - Optional mis-analysis in sub-queries could drop solutions
It was possible for AllegroGraph to mis-analyze whether a variable in a SPARQL sub-query was optional or not and therefore to lose solutions. This has been corrected.
Bug23524 - Property path queries could lose results when there is circularity
Under some circumstances, a SPARQL query that used property paths could lose results that arose from circular paths through the triple-store. This has been corrected.
Bug23519 - SPARQL sub-query and GRAPH clauses could interact badly
It was possible for a query that used sub-query and GRAPH clauses to lose solutions. This has been corrected.
Bug23489 - Queries that started with a SERVICE clause could lose bindings
If a SPARQL query started with a SERVICE clause and contained a cross-product then it was possible for some bindings to be lost. This has been corrected.
Bug23452 - Some temporal magic properties could fail to produce results
Some of AllegroGraph's temporal magic properties could produce plans that introduced filters that could not succeed. This has been corrected.
Bug23448 - Fix problem with SPARQL t:pointBeforeDatetime and t:pointAfterDatetime
The two SPARQL magic properties t:pointBeforeDatetime and t:pointAfterDatetime could fail to find bindings if the dateTime supplied was a constant not present in the triple-store. This has been corrected. SPARQL magic properties are listed here in SPARQL Magic Properties.
Bug23443 - Remote SPARQL queries cannot handle really large results
Returning very large query result sets (i.e., Gigabytes) from remote SPARQL endpoints could lead to resource exhaustion. This has been corrected.
Bug23373 - Improve SPARQL efficiency for small ORDERed or DISTINCT result sets
SPARQL queries with small result sets that use ORDER BY or DISTINCT will now be slightly faster.
Bug23344 - SPIN magic properties don't work if they use other SPARQL functions
A query-time error would be signaled if a SPIN magic property used other SPARQL functions like CONCAT or SUBSTR in its definition. This has been corrected.
Bug23341 - remote-triple-stores handle limit and offset in SPARQL queries incorrectly
If a query specified an OFFSET as part of the query string, then any externally imposed OFFSET (i.e., from the HTTP query parameters) was ignored. This has been corrected. If a query specified a LIMIT as part of the query string, then the smaller of that limit and the externally imposed one would be used instead of just using the external one. This has been corrected. Note that this change means that specifying a limit in a query in AGWebView will have no effect as the limit is controlled via the dropbox selection box.
Bug23329 - Use SPARQL/JSON for SERVICE calls to avoid problems with embedded control characters
AllegroGraph now uses SPARQL/JSON rather than SPARQL/XML to receive results from SPARQL endpoints. This makes it possible to return results that contain control characters like ^Z. The format is also more concise and easier to parse which makes processing the transmissions approximately twice as fast.
Bug23321 - Nested sub-queries and BIND forms can fail to compute correct results
In rare cases, it was possible for a query involving nested SPARQL sub-queries and BIND expressions to fail. This has been corrected.
Bug23311 - Requesting application/nquads from a CONSTRUCT query leads to error
If an application requested application/nquads from a CONSTRUCT query via the HTTP interface, a string-table lookup error would result. This has been corrected. Although version 1.1 of SPARQL CONSTRUCT can only return triples, it still makes sense to ask for N-Quads. In this case, the N-Quads will simply have the graph place left off.
Bug22674 - DESCRIBE and CONSTRUCT queries on end points could fail
SPARQL CONSTRUCT and DESCRIBE queries against remote SPARQL end points could fail when trying to parse the returned results. This has been corrected.
Bug22672 - Queries where every FROM NAMED graph did not exist could signal an error
If every FROM NAMED graph in the dataset did not exist, then some queries that used the GRAPH clause in their body could signal an error rather than returning results. This has been corrected.
Bug22515 - Increase responsiveness of query canceling during result generation
AllegroGraph did not check for query canceling or timeouts during result post-processing (e.g., while sorting the final result or removing duplicate entries from a DISTINCT result set) or result generation. This has been corrected.
Bug22292 - tripleid magic property should handle fixed triple IDs
The SPARQL magic property for AllegroGraph tripleId-based reification would not function for fixed triple IDs. This has been corrected.
Bug22268 - Sub-queries that used Aggregation and limit could lose solutions
If a sub-query had a limit and used aggregation, then it was possible for some solutions to be lost during processing. This has been corrected.
AGWebView
Bug23690 - Fixed bug breaking the AGWebView View Statements dialog in IE.
A bug in AGWebView that prevented the View Statements dialog from fetching and displaying statements when brought up in Internet Explorer has been fixed. The symptom was a partially loaded page with the spinning "Loading..." icon remaining permanently on the page.
Bug23521 - More... button in AGWebView is broken for Prolog select queries
The button to show additional results in the AGWebView query window was broken for Prolog select queries. This has been corrected.
Bug23358 - AGWebView doesn't always display Unicode consistently
Under some conditions, AGWebView would display literals and resources containing Unicode using the \u escaping mechanism of the N-Triples syntax rather than just displaying the characters directly. This has been corrected.
Bug23315 - Running backup from within AGWebView does not work
The Backup this Store dialog in AGWebView was broken. This has been corrected.
Changes to the Lisp API
Rfe13941 - Allow specification of chunk style when adding an index
add-index now accepts a style keyword argument which can be 0 or nil (either meaning use the default chunk style for the flavor based on the Style2Indices configuration parameter), 1, or 2. See Triple Indices for a discussion of index styles.
Bug23600 - remote-triple-stores CONSTRUCT queries now correctly returns cursors
In the Lisp client, a SPARQL CONSTRUCT query should return a cursor that yields arrays of bindings but the remote-triple-store was instead yielding lists of bindings. This has been corrected.
Bug23597 - remote-triple-store CONSTRUCT queries return nil bindings
Many of the programmatic RDF formats (such as :lists or :arrays) for SPARQL CONSTRUCT queries on remote-triple-stores returned nil bindings rather than the actual values. This has been corrected. Note that this bug did not effect text based output formats like N-Triples or Turtle.
Bug23510 - remote-triple-stores did not support load-trig
Lisp remote-triple-stores did not support loading the TriG format. This has been corrected.
Bug23509 - Altering restriction-reasoning-enabled-p could lead to an internal error
Altering the value of restriction-reasoning-enabled-p using the setf method after the reasoner had been applied and prepared could lead to an error. This has been corrected.
Bug23404 - It was impossible to create an anonymous remote session
AllegroGraph would signal an error that This session is owned by another user when trying to create an anonymous remote session in the Lisp client. For example, the following would fail:
(open-triple-store "remote"
:triple-store-class 'remote-triple-store
:server "www.example.com"
:specification "<sp2b-5e4>[rdfs++]"
:port 9002) This has been corrected.
Bug23320 - The Lisp direct client now ensures that UPIs are interned when calling add-triple
The Lisp direct client was not checking that UPIs were interned which could lead to a UPI-not-in-string-table error when there were bugs in customer code. This has been corrected. The check does add a small cost to programmatic triple insertion. The while-not-validating-that-upis-are-in-string-table macro can be used to temporarily disable the checks once you are sure that the code is correct.
Bug22844 - The SPARQL datatype function failed to handle nD-geospatial literals
The SPARQL datatype function was unable to handle nD-geospatial literals. This has been corrected.
Bug22793 - Problem with remote-reasoning-triple-stores and SNA
On the Lisp client only, calling Social Networking (SNA) functions on a remote-reasoning-triple-store instance could lead to a stack overflow. This has been corrected.
Bug22791 - get-triple was broken for remote-reasoning-triple-stores
On the Lisp client only, calls to get-triple on a remote-reasoning-triple-store could fail to return results. This has been corrected.
Bug22587 - copy-triples doesn't support remote triple-stores
The copy-triples function did not work on remote-triple-stores. This has been corrected. Note that copy-triples on remote-triple-stores requires that
- the triples are all valid RDF
- no blank nodes are used.
Bug22586 - Improve copy-triples
The copy-triples function was not behaving correctly with respect to the source-db argument. This has been corrected.
Bug22447 - Fix cursor memory management
Under rare circumstamces, especially in a multi-threaded environment, cursors used in the Lisp client could end up being used in an unsafe way leading to various errors. With the AllegroGraph server, shared backends experiencing high load were at the risk of such failures. This bug has been fixed.
Also, in the Lisp client with a direct connection, any code that involved the creation of cursors (such as queries and calls to get-triples) had to be wrapped in a with-query-environment form. While that's still the recommended approach to take in production, with this change when there is no enclosing with-query-environment, cursors holding critical resources get finalizers which makes it more convenient to experiment at the REPL by reducing the risk of resource exhaustion.
Bug20852 - remote-triple-stores do not support SPARQL update
Lisp remote-triple-stores now support SPARQL update.
Prolog
Rfe13968 - Add :offset to Prolog select
AllegroGraph's Prolog based select query language now supports an :offset option. This can be used to skip the first offset results. For example,
(select (?class)
(:offset 10)
(q ?s !rdf:type ?class)) would skip the first 10 results and return the rest of the bindings for ?class.
Documentation
No significant changes.
Python Client
Rfe12828 - Support Python 3
AllegroGraph now supports both Python 2 and Python 3. The sources for Python 2 continue to be in the src2 directory whereas those for Python 3 are in the new src3 directory.
Java Client
No significant changes.
nD Geospatial
Rfe13978 - Validate arguments to ndfn:ordinateValue at parse time when possible
AllegroGraph now validates constant arguments to ndfn:ordinateValue at parse time. See the N-dimensional Geospatial Usage Guide and Example document.
AllegroGraph 5.1.2
Server Changes
Bug19296 - AllegroGraph can stop responding because all threads seem busy
Under rare circumstances it was possible for AllegroGraph to stop responding. Such a hang was accompanied by "All threads busy, pause" log messages. This has been corrected.
AllegroGraph 5.1.1
Because there are only a few changes in release 5.1.1, only headings with actual changes are listed.
Server Changes
Rfe13969 - Add HDFS input support to agload
agload now supports loading files from HDFS sources. For this to work, there must be a working "hdfs" program in your PATH and HDFS input files must be specified on the agload command line using the hdfs:// prefix. For example:
agload repo hdfs:///user/bruce/lubm-50.nt.gz agload is described in the Data Loading document. HDFS loading is described in the section Loading files from HDFS filesystems in that document.
Rfe11078 - Improve agload verbosity behavior
Previously, passing -v or --verbose to agload could actually decrease its level of verbosity. This has been fixed.
The new verbosity levels are:
0 (-v not supplied): Only report periodic load rate information. 1 (-v) : As above, plus print the job option summary before starting the operation. 2 (-v -v) : As above, plus print the name of every file that has been processed.w
agload is described in the Data Loading document.
AllegroGraph 5.1
Server Changes
Rfe13570 - Support the TriG format for serialization
AllegroGraph can now serialize data using the TriG format. For example, you can use the output format trig in agexport or an Accept type of application/trig from the REST API.
Rfe13501 - Implement HTTPNoProxy configuration directive
This change adds a new global configuration directive, HTTPNoProxy. HTTP requests to domains that match one of the suffixes specified with HTTPNoProxy are never proxied. HTTPNoProxy can be specified multiple times.
HTTPNoProxy localhost
HTTPNoProxy mydomain.com Requests to 'localhost' and '127.0.0.1' are never proxied. Top level directives are described in Server Configuration and Control.
Rfe12010 - Support the TriG format for data import
AllegroGraph now supports the TriG data format for loading data. See http://www.w3.org/TR/trig/ for details on the format. See Data Loading.
Rfe11730 - Update Turtle parser to conform to the specification
AllegroGraph's turtle parser now passes the W3C conformance tests (http://www.w3.org/TR/turtle/). The parser is also significantly faster.
Bug23077 - Rare thread safety issue in shared back-ends
Previously, when multiple requests for the same repository were being executed concurrently on the same shared back-end a very rare thread safety issue could cause the backend to die. This bug has been fixed.
Bug22944 - Freetext indexer could mishandle encoded literals
AllegroGraph's built-in freetext indexer was trying to index encoded literals (numbers, dates, times and dateTimes) and would sometimes misinterpret them leading to extremely slow indexing. This has been corrected by preventing the indexing of these literals. This means that freetext indices will not include values for encoded literals.
Bug20556: Double-fault of trie-full condition
Previously, when adding a word with many hits to a text index a "trie-full" error could occur. This has been fixed.
HTTP Client
Bug23064 - Post requests to the statements REST API could fail
It was possible for POST requests to the statements REST API to corrupt the stream of bytes being sent. This has been corrected.
SPARQL
Rfe13683 - Improve variable count tracking during query execution
AllegroGraph keeps better track of the estimated cardinality of bindings for variables during query execution which improves the execution engine's overall efficiency.
Rfe13601 - Improve handling of property path queries that use simple alternation
AllegroGraph can now apply its constraint machinery to simple property path queries that use only alternation. For example:
select ?o {
?s :a | :b | :c ?o .
filter( ?o > 10 )
}
This means that queries like the above can operate significantly faster. Rfe13599 - Improve query pattern constraint logic
AllegroGraph now does a better job in choosing how to constrain variables in SPARQL triple-patterns during execution.
Rfe13476 - Add more logging information to SPARQL SERVICE queries
AllegroGraph now includes more logging during the execution of SPARQL SERVICE queries.
Rfe13426 - Provide more control over federated SPARQL queries
The new query option serviceRequestSize provides control over the size of the POST requests used during SPARQL federated query. The default size is one megabyte. A smaller size may cause additional trips to the remote server. Some servers may reject requests that are too large. SPARQL query options are listed in the SPARQL Query Options of the SPARQL Reference.
Rfe13213 - Allow BIND to float during query analysis
AllegroGraph's SPARQL query analyzer now does a better job rewriting query patterns involving BIND. In particular, the BIND clause now floats around JOIN, GRAPH and FILTER clauses. This can help make these queries perform significantly better.
Rfe12269 - Improve query planning with respect to cross products
AllegroGraph's SPARQL query planner now does a better job of detecting cross products at plan time and modifying the query plan to avoid the cross product when this is possible.
Bug23125 - Improve query planning for some property path queries
SPARQL Property Path queries using the zero-or-more operator (i.e., *) could sometimes choose a sub-optimal plan because AllegroGraph was misestimating the size of the triple-patterns involved. This has been corrected.
Bug23095 - Correct internal error that occurred during some Property Path queries
Some Property Path queries could fail to run to completion because of an internal bookkeeping error. This has been corrected.
Bug23081 - Certain SPARQL queries could cause cursor data-structure leakage
Certain SPARQL queries could fail to correctly clean up all of the cursors that they used which could lead to backends becoming unresponsive if they were under heavy load. This has been corrected.
Bug23027 - SPARQL was not ordering reified triple IDs correctly
The SPARQL query engine was not correctly sorting URIs representing triple IDs. This has been corrected.
Bug23026 - Some SPARQL SERVICE calls could mangle their query
It was possible for a SPARQL SERVICE call to lose space characters and thereby send the wrong query to the remote end point. This has been corrected.
Bug23005 - RDF/XML serializer could emit an invalid namespace abbreviation
If there was a namespace mapping between the empty string and some URI, then the RDF/XML serializer would emit an invalid namespace abbreviation. This could occur, for example, when running a CONSTRUCT query using a PREFIX abbreivation like
PREFIX : <http://example.org#> This problem has been corrected.
Bug23004 - Sparql XML parser could fail to parse some literals
AllegroGraph's SPARQL XML parser is used when making federated queries against remote endpoints. This parser was incorrectly handling literals with ampersands (or other escaped characters) in them. This has been corrected.
Bug22999 - When embedded in a GRAPH clause freetext SPARQL queries could fail
Depending on the data and the query, a freetext match or matchExpression magic predicate could fail to find all results when it was embedded inside of a GRAPH clause. This has been corrected.
Bug22994: Fix bad interaction between SPARQL sub-query and federation
If a SPARQL query used a sub-query inside of a federated query, AllegroGraph could lose track of the variables involved and create inadvertant cross-products. This has been corrected.
Bug22966 - SERVICE clause in a sub-query could fail
Depending on the variables projected, a SPARQL query that used a SERVICE clause inside of a sub-query could fail during execution.
Bug22937 - GROUP BY returned an empty row when there are no aggregates projected
If a SPARQL aggregation query projects only variables (i.e., it does not use any aggregation functions) and the query has no results, then AllegroGraph would return a single row of unbound bindings rather than returning no results at all. This has been corrected. An example of the query in question would be
select ?anything {
?s <ex://missingPredicate> ?anything .
} group by ?anything Bug22915 - SPARQL STR function is incompatible with aggregation
Queries that used the SPARQL STR function on the result of an aggregation would fail. This has been corrected. An example of such a query would be:
select ( str( count(1) ) as ?count) {
?s :testPredicate ?o .
}
Bug22904 - Bad query bookkeeping could cause performance problems
Given the right data and the right query, the SPARQL query engine could be led to evaluate triple patterns twice: once with constraints and once without which would lead to severe performance problems. This has been corrected.
Bug22899 - Query options could not always be used with SPARQL SERVICE
When using the SERVICE clause, AllegroGraph's query PREFIX options were not always being sent to endpoints in a way that they could be read. This has been corrected.
Bug22874 - N-Dimensional magic property queries could fail
Depending on the data and the query, AllegroGraph could choose an incorrect plan ordering for an n-Dimensional SPARQL magic property. This would lead to the UPI not in string table error. This has been corrected.
Bug22845 - A query timeout of zero now means "no timeout"
Previously, a query timeout of zero was interpreted literally to mean that the query should terminate immediately. Now, a timeout of zero means that the query should run as if there is no timeout specified. This is especially helpful when a default timeout is included in the AllegroGraph configuration file.
Bug22751 - Error using one-or-more property paths in SERVICE clause
AllegroGraph was signaling an error if a query tried to use a one-or-more property path query inside of a SERVICE clause. This has been corrected.
Bug22680 - Correct SPARQL parsing problem for property paths in object lists
The SPARQL Parser was not correctly handling property paths embedded inside object lists like ?s <ex:predicate> [ <ex:Path>* ?o ]. This has been corrected.
Bug22593 - Use the correct count estimates during query execution
Depending on the query and the store, AllegroGraph could sometimes use the wrong variable count estimates which would lead to sub-optimial performance. This has been greatly improved.
Bug22388: Chunkprocessingmemory defaults to a larger value than memoryLimit
The chunkProcessingMemory size defaulted to a larger size than the memoryLimit which meant that queries could fail when using chunk processing. This has been corrected.
Bug22212 - Magic property for t:intervalStarts logic was wrong
The magic property for t:intervalStarts was incorrectly behaving the same as t:intervalCotemporal. This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
Rfe13593 - The compiler is needed in direct lisp clients that perform queries
This change will likely not affect any users since the compiler is normally in Lisp images, but it is possible to build an image without the compiler and without the compiler being available to load (though, again, it is unusual to do so). Such images serving as direct Lisp clients cannot perform queries.
Bug22973 - Fix Lisp direct client sigsegv when rolling back a closed store
Previously, in the Lisp client when using a direct connection to a triple store, calling rollback on a closed store would produce a SIGSEGV error. With this fix, an informative error condition is signaled.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
nD Geospatial
Bug22786, Bug22850: improve nD string writers and parsers
nD data encoding and decoding supports various human-readable formats (ISO6709 and ISO8601) when such a format can be applied unambiguously to a particular nD encoding. If a human-readable format applies, it will be preferred on decoding (output) and accepted on input (encoding). Previously some correct externalization formats were rejected on input or not used on output.
nD now correctly supports encodings with multiple ordinates of the same specialized type, i.e. lat, lon, alt, and time. Ordinates must of course all have different names. Use of such encodings will necessarily suppress human-readable externalizations since they are ambiguous.
Error detection during nD encoding/decoding is improved along with error message text.
AllegroGraph 5.0.1
Server Changes
Bug23016 - Incorrect triple-count and garbage triples
Under certain circumstances (particularly when delete-duplicate-triples is used), crash recovery or replication can result in an incorrect triple count and garbage triples being added to the database. This has been corrected. We recommend that users using replication and duplicate triple deletion recreate their replicas from a fresh backup. Replication is described in the Replication document.
HTTP Client
Bug23040 - Performing the same ASK query more than once under REST could error
Under certain circumstances, if an ASK query that used joins was evaluated more than once under the HTTP REST API, then AllegroGraph would signal an error. This has been corrected.
AllegroGraph 5.0
Server Changes
Rfe13506 - Minor query optimization for triple pattern filters
AllegroGraph now performs triple pattern filtering slightly more efficiently when the current set of solutions has only a single member.
Rfe13405 - Warn if transparent hugepages are enabled
Due to the potential for negative performace impact, AllegroGraph now logs a warning into agraph.log on startup if Transparent Hugepages are enabled. See this section of Performance Tuning for information on the transparent hugepage issue and its effect on performance.
Rfe13376 - Improve hashing performance
AllegroGraph's internal hashing functions have been further optimized leading to small general performance improvements.
Rfe13346 - configure-agraph should allow for specification of port
The configure-agraph script now prompts for the Port configuration parameter and accepts the --port command line argument. See Running the configuration script in Server Installation.
Rfe13265 - Update N-Triple and N-Quad parsers
AllegroGraph's N-Triple and N-Quad parsers now support the full W3C specifiation and pass all tests. The performance is also slightly improved. Note that this change means that AllegroGraph will no longer load invalid N-Triple and N-Quad source data unless the relax-syntax argument is used. See Data Loading, the Lisp API functions load-ntriples and load-nquads, and [PUT/POST /repositories/name]/statements in REST/HTTP interface.
Rfe13172: Allow setting size of strings hash table at database creation time
A new database configuration parameter StringTableSize is now available. It can be used to configure the number of hash table slots used for the string storage hash table and setting it properly can enhance performance. See Catalog Parameters that Improve Performance in Performance Tuning for more information on performance-enhancing options for details.
Rfe13146 - Improve suboptimal index scan performance
The efficiency of low-level cursor operations has been improved when an optimal index for the cursor is not available.
Rfe12872 - Remove ag3-export / ag4-import
The obsolete ag3-export and ag4-import utilities have been removed. Serializing data to nquads in AllegroGraph 3 and loading it into AllegroGraph 4 is the recommended alternative.
Rfe11580: Added --no-daemonize flag to agraph binary
The agraph binary is the actual program which starts AllegroGraph. It is not typically invoked directly, but rather using the agraph-control script, but it can be called directly. When it is and --no-daemonize is supplied, the service daemon will remain in the foreground. See agcontrol.
Bug22837 - Text index incorrectly indexing the string "nil"
Since version 4.13 the text indexer was incorrectly indexing the word "nil" for triples that were ineligible for text indexing. This resulted in unnecessary work being performed and space being consumed. This has been fixed. We recommend that you drop and recreate any existing text indexes.
Bug22835 - Fix numa info parsing failure
At startup, information about numa (non-uniform memory access) topology is collected and logged. Previously, parsing failed with an error if a cpumap file in /sys/ contained a comma separated list of numbers instead of just a single number. This bug has been fixed.
Bug22767 - Use agraph5 for the Lisp client fasl name
AllegroGraph 5.0 now uses agraph5.fasl instead of agraph4.fasl for the name of the Lisp fasl file.
Bug22754 - Concurrent duplicate triple deletion can result in inaccurate triple count
Under some circumstances duplicate triple deletion transactions which occur concurrently with other deletion transactions could result in an inaccurate triple count. This has been corrected.
Notes:
Though the triple count could become inaccurate, the integrity of the triples in the database remained intact.
The triple count on already-affected databases will remain inaccurate until they are recreated.
Bug22566 - Encoded ID validation was too weak
In some cases, a URI representing an invalid encoded ID could be accepted by AllegroGraph instead of signaling an error. This has been corrected.
Bug22564 - Data import could fail to determine the correct base URI
When importing a compressed RDF/XML file, AllegroGraph could fail to determine the correct base URI. This has been corrected.
Bug22559 - RDF/XML parser should validate the base URI if supplied
Previously, the RDF/XML parser failed to validate the base URI it was given which could lead to invalid data. This has been corrected.
Bug21867 - Ensure text index queries are called using strings
AllegroGraph sometimes called lower level text index query functions with invalid arguments. This has been corrected.
Bug21460 - Polygons data cannot be exported and imported
AllegroGraph now converts resources of the form
<http://franz.com/ns/allegrograph/4.14/index#_N> (where N is replaced by a whole number) into encoded UPIs rather than hashed ones. These URIs are used to serialize AllegroGraph polygon data which means that polygons can be efficiently exported and imported. Note that the Lisp API upi->value will return the numeric representation for the UPI rather than the string representation. Use part->string if the string representation is needed.
Bug20497, Rfe11047 - Fix fti too many open files error
Previously, when constructing a new text index for a database with existing triples, there was potential for the operation to fail with a "Too many open files" error. This has been fixed.
HTTP Client
Rfe13304 - Support relaxed load syntax for remote triple-stores
The HTTP REST API for loading statements now includes the relaxSyntax argument. When true, N-Triple and N-Quad data are not validated as stringently. For example, blank node names may include underscores. See PUT/POST statements in REST/HTTP interface.
Rfe13249 - Augment optimize-indices API to allow for single index optimization
The HTTP API's indices/optimize service now takes an additional index query parameter which can be specified multiple times. If omitted, then all of a store's indices will be optimized. If supplied, each parameter value should be the name of an index whose optimization is being requested. See [POST /repositories/name]/indices/optimize in REST/HTTP interface.
Rfe11547 - Support Sesame query timeout parameter in HTTP
AllegroGraph's SPARQL HTTP REST interface now supports an optional timeout parameter. If specified, it must be a number greater than zero which denotes the number of seconds to limit SPARQL query execution time. If unspecified, then there is no limit on the length of a time a query may run. See [GET/POST /repositories/name in REST/HTTP interface.
Rfe11221 - General type-mapping improvements
The interface for setting up predicate- and datatype-mappings has been simplified. The same HTTP REST API calls can be used to setup regular XSD datatype mappings and geospatial ones (including cartesian, spherical and n-Dimensional). In each case, the datatype must be specified using its URL. E.g., the 10x10 cartesian mapping with resolution 1 would use the URL:
<http://franz.com/ns/allegrograph/3.0/geospatial/cartesian/0.0/10.0/0.0/10.0/1.0> See Type mappings in REST/HTTP interface.
Bug22576 - POST|PUT statements service should not require eval permission
AllegroGraph would not allow client-side files to be imported into a store unless the user had eval permission. This has been corrected. Of course, eval permission is still required to import server-side files.
SPARQL
Rfe13510 - Improve performance of the SPARQL query REST API
AllegroGraph now automatically caches query information across REST API calls to improve overall query performance.
Rfe13323 - Provide API to retrieve only the ID part of an encoded ID
There is now a SPARQL function,
select * {
_:b ex:identifier ?id .
bind( <http://franz.com/ns/allegrograph/fn#encodedIdId>(?id) as ?value )
} Rfe13305 - Deprecate old-style SPARQL GEO syntax
Because AllegroGraph's magic properties support a richer range of geospatial features than the old-style SPARQL GEO syntax, the old syntax is being deprecated and will now signal a warning when it is used. See N-dimensional Geospatial Overview.
Rfe13297 - Reduce memory consumption for distinct aggregation in SPARQL queries
AllegroGraph now better manages memory when evaluating distinct aggregation queries like:
select ?dept (count(distinct ?student) as ?sc) where {
?dept a :Department .
?student :memberOf ?dept .
} group by ?dept
Rfe13289 - When logging, use query supplied namespaces for abbreviations
Previously, AllegroGraph would ignore SPARQL query supplied namespaces when displaying resources during query logging. It now uses these namespaces to improve the readability of the display.
Rfe13171 - Improve query checks for unknown variables
AllegroGraph now warns more consistently when a valid SPARQL expression uses variables in unreasonable ways. For example, this query is perfectly valid but the ORDER BY variables are not bound by the query so it is unlikely to be what was intended:
select ?a ?year ?label {
?a a :Car .
?a :modelYear ?year .
?a rdfs:label ?label .
} order by ?autoYear Rfe12470 - Apply multiple constraints to the same triple pattern
Previously, AllegroGraph could only apply a single logical FILTER expression to any given triple pattern. This meant that queries like:
select * {
?s ?p ?o .
filter( ?p = ex:pred1 && ?o > 10 )
}
would optimize the query plan with either the ?p constraint or the ?o
constraint but not both. This has now been improved so that queries
like the above will use both constraints simultaneously and thereby
run much more efficiently. Bug22803 - Fix bug in authorizationBasic query option
The authorizationBasic query option was not being parsed correctly which meant that queries that tried to use it would fail. This has been corrected.
Bug22802 - Support userinfo in SPARQL service request URIs
URIs used in SPARQL service requests did not allow the inclusion of userinfo information so that queries like this would fail:
select * {
service <http://user:[email protected]:7001/repositories/sp2b-1e4> {
?s ?p ?o
}
} limit 5 This has been corrected.
Bug22750 - Correct an error in SPARQL property path queries
In some circumstances, a SPARQL property path query could signal an error when trying to find paths between fixed endpoints. This has been corrected.
Bug22735 - SPARQL federated query could fail to recognize geospatial data
SPARQL federated query (via the SERVICE) clause could fail to transform returned results using AllegroGraph geospatial encoding. This would lead to query errors and has been corrected.
Bug22735 - SPARQL federated query could fail to recognize geospatial data
SPARQL federated query (via the SERVICE) clause could fail to transform returned results using AllegroGraph geospatial encoding. This would lead to query errors and has been corrected.
Bug22734 - Query options should be passed along to the HOST of any SERVICE clauses
Previously, any custom query options set using the PREFIX notation were not passed along to the remote server when using SPARQL Federated query. Now, these options are passed to the remote server which provides more control when that server is running AllegroGraph.
Bug22732 - Correct errors for SERVICE clauses that used blank node labels
SERVICE clauses that used blank node labels (or property paths that might be rewritten using blank node labels) could lead to an error when AllegroGraph tried to merge these bindings with the containing query. This has been corrected.
Bug22731 - Error when using property paths in a SERVICE query with distinct results
If a query asked for DISTINCT results and had a property path query inside of a SERVICE clause, then AllegroGraph would signal an error during query planning. This has been corrected.
Bug22730 - Some uses of UNION and VALUES could lead to an error
Some uses of UNION clauses and SPARQL in-line data via the VALUES clause could lead to a run-time query error. This has been corrected.
Bug22727 - EXISTS filters sometimes performed more computation than necessary
SPARQL EXISTS (and NOT EXISTS) filters were performing query computation even when the main result set was empty (and hence had nothing to filter). This has been corrected.
Bug22660 - Temporal magic properties introduce new variables
Temporal magic properties like
prefix m: <http://di.uoa.gr/~charnik/music_db#>
prefix time: <http://franz.com/ns/allegrograph/3.0/temporal/>
select * {
m:p_1939-08-19 time:pointBefore m:p_1980-12-04 .
} was returning bindings instead of empty rows. This has been corrected.
Bug22588 - Further improve handling of multiple range filters on the same variable
AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like
select * {
?s ex:test ?age .
filter( ( ?age >= 10 && ?age <= 20 ) ||
( ?age >= 60 && ?age <= 70 ) ||
( ?age >= 80 && ?age <= 90 ))
} will now execute as the combination of two range queries rather than a single scan. This change is similar to the change for bug22535 but more general.
Bug22569 - Improve canceling of SPARQL SNA generators
Previously, neither the administrator jobs list nor the query canceling API functioned properly if a query used an SNA generator that was implemented via a separate SPARQL query. This has been corrected. The SNA generator SPARQL queries will no longer appear in the administrator jobs list and they will now inherit any timeout from the query that they are running in service of. This means that canceling the parent query will cancel these child queries.
Bug22541 - Improve handling of invalid typed literals
Previously, the SPARQL parser would generate a general error and log a backtrace if a query included illegal XSD datatypes like "2014-Mar-13"^^xsd:date. These are now treated as SPARQL parse errors.
Bug22541 - Improve clause reordering for some SPARQL GRAPH queries
AllegroGraph's statistical BGP reordering could lead to sub-optimal query plans for certain queries that involved GRAPH queries and FILTERs on ranges. This has been corrected.
AGWebView
Rfe13530 - Rearrange some of AGWebView's settings
The "show long parts" and "show contexts" settings are global to the browser session but they were only available on the query viewer. These settings are now also accessible via the new Utilities | Settings menu. See the AGWebView document.
Rfe13504: Add a "large operations warning" to AGWebView
AGWebView's site settings now include a place to enter text that will appear before it executes an operation that potentially exposes a large amount of repository data. Note as well that the site settings are now available via the new Utilities | Settings menu.
Rfe13183 - Agwebview uses Unicode rather than \u escaping to display data
AGWebView now presents non-ASCII characters using Unicode instead of their \uXXXX escape sequences.
Rfe12523 - Improve data import dialogs in AGWebView
The data import dialogs in AGWebView now support AllegroGraph's relaxed syntax and continue on error options. Relaxed syntax allows the RDF importers to accept data that it not strictly correct but is still easily understandable. For example, a blank node label is not allowed to include the - (dash) character but many source files use it regardless. Normally, AllegroGraph will not allow these files to be imported but if relax syntax is turned on, it will. Continue on error allows AllegroGraph to continue loading data even if it encounters a bad triple or quad. Note, this option only affects line based formats like N-Triples or N-Quads. As ignoring errors is an all or nothing affair, this option should be used with caution: it will not be possible to determine how many triples were skipped or even if any errors were encountered. These new options correspond to the --error-strategy and the --relax-syntax options to agload.
Rfe11466 - Use double-clicks to navigate the file picker in AGWebView
AGWebView's file picker for selecting files on the server side used single-clicks rather than double clicks to change directories. This has been corrected.
Bug22800 - Fix regression in v4.14 that breaks proxying through frontend
There was a regression in v4.14 that partially broke the use of AGWebView over https. This is fixed. Additionally support is added for proxying requests to https sessions through an http frontend, or to http sessions through an https frontend.
Changes to the Lisp API
Rfe13249 - Augment optimize-indices API to allow for single index optimization
The Lisp API's optimize-indices function now takes an additional indices argument. This can be nil (the default) or :all to optimize all of the triple-store's indices, a single index name (like :psogi) to optimize one index or a list of index names to optimize each of them.
Rfe13184 - Check for uninterned UPIs when adding triples
Previously, AllegroGraph did not ensure that the UPIs in a triple being added were already interned in the triple-store's string-table. This led to the possiblity of adding triples that could not be re-serialized. The default is now to check UPIs unless the code doing the adding is wrapped in the new while-not-validating-that-upis-are-in-string-table macro.
Rfe6878 - open-triple-store can take a triple-store instance
The open-triple-store function can now be called using an existing triple-store instance. In this case, it will return a new instance of the same underlying store.
Bug22811 - Fix use of direct Lisp client when AllowHTTP is false.
Previously, the direct Lisp client would be unable to access an Allegrograph server if it was started with the config directive AllowHTTP set to false. This bug is now fixed.
Bug22772 - Include inferred-triple-p in the remote Lisp client
The inferred-triple-p function was inadvertently left out of the remote Lisp client. This has been corrected.
Bug22703 - Add relax-syntax argument to load-nquads|ntriples-from-string
The funtions load-nquads-from-string and load-ntriples-from-string were missing the relax-syntax argument. This has been corrected.
Prolog
No significant changes.
N-dimensional geospatial
Rfe13364 - Add automate-nd-datatype-mappings switch to agload
agload now supports AllegroGraph's automatic N-Dimensional geospatial type mappings. If the new --automate-nd-datatype-mappings option is specified on the command-line, then any geospatial datatypes in a source file will be automatically recognized and added to the triple-store. See Data Loading.
Rfe13350 - Add additional standard namespace abbreviations
The following namespace abbreviations have been added to AllegroGraph:
nd- <http://franz.com/ns/allegrograph/5.0/geo/nd#>ndfn- <http://franz.com/ns/allegrograph/5.0/geo/nd/fn#>keyword- <http://franz.com/ns/keyword#>
These are helpful in using the new N-Dimensional geospatial functionality.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Rfe12921 - Upgrade to sesame-2.7.11 and jena-2.11.1 jars
With this change, the Java Client has been upgraded to use Sesame 2.7.11 and Jena 2.11.1 jars. Previously, the Java Client used Sesame 2.6.8 and Jena 2.7.2 jars.
Rfe12920 - Tutorial for connection pooling using Sesame
Example24 has been added to the Sesame tutorial examples to create and manage connection pools using Sesame. Java Sesame tutorials in general are discussed in Java Sesame Tutorial. Example24 does not have a writeup and is in the file TutorialExamples.java discussed in Java Sesame Tutorial.
Bug22854 - TutorialExamples.java signals warnings in java client
With this change, Tutorial examples run with out any runtime warnings. Also some warnings are fixed in callimachus tests
Bug21911 - Implementation of Jena QueryExecution methods
Previously, the AGQueryExecution class had no implementation for following methods:
- execConstructTriples()
- execDescribeTriples()
- getQuery()
- setTimeout(long timeout),
- setTimeout(long arg0, TimeUnit arg1)
- setTimeout(long arg0, long arg1)
- setTimeout(long timeout1, TimeUnit timeUnit1, long timeout2, TimeUnit timeUnit2)
This has been corrected. An example (example24) for a Jena Graph Union has been added to the Jena examples. Jena tutorials in general are discussed in Java Jena Tutorial. Example24 does not have a writeup and is in the file JenaTutorialExamples.java.
AllegroGraph 4.14.1
Server Changes
Rfe13193 - Parameterize email notifications
This change builds on rfe13104 and makes it possible to specify for which audit events emails should be sent and to what address. The body of the email contains the triples added for the audit event in pretty printed Turtle format.
The AdminEmail and AdminEmailSMTPHost configuration directive were removed in favor of the new AuditEventsToEmail directive. (Note this is a backward-incompatible change. See the Auditing document.)
The following rule asks for emails to be sent to [email protected] via the gmail smtp host (defined with SMTPHost directive) when any of the expirePassword, addUser or deleteUser audit events trigger:
AuditEventsToEmail to="[email protected]", smtphost="gmail", \
events="expirePassword,addUser,deleteUser" If there is only one smtp host defined then it can be omitted:
AuditEventsToEmail to="[email protected]", \
events="expirePassword,addUser,deleteUser" Note that auditing must be enabled for this to work. See the Audit documentation for more information.
Rfe13201 - Restrict the range of backend ports
This change adds the BackendPorts global configuration directive to restrict the range of ports on the loopback interface to be used by shared back-ends. Note that the maximum number of shared back-ends is limited by the number of ports in the specified range even if the Backends directive is higher. Configuration options are described in Server Configuration and Control.
Bug22675 - Simple queries may not show up in the jobs list
This is a continuation of the work done for bug22536 and increases the number of queries that can be controlled in the jobs list.
Bug22666 - run-fti-merge doesn't fail fast
Previously, if the text index merger process encountered an error during merging, it would log the error and a backtrace to agraph.log and then continue operation. If the error condition persisted, this logging could continue indefinitely, resulting in agraph.log growing to excessive size. This has been fixed.
Bug22641 - Listing and cancelling queries
Previously, job management (available via /jobs REST service and also admin/job in AGWebView) had a bug which caused concurrent requests to fail with:
the assertion (endp (intersection
(db.agraph.spawn::receiver-for db.agraph.spawn::r)
(db.agraph.spawn::receiver-for db.agraph.spawn::receiver))) failed This bug has been fixed.
Bug22641 - Listing and cancelling queries
Previously, job management (available via /jobs REST service and also admin/job in AGWebView) had a bug which caused concurrent requests to fail with: the assertion (endp (intersection (db.agraph.spawn::receiver-for db.agraph.spawn::r) (db.agraph.spawn::receiver-for db.agraph.spawn::receiver))) failed This bug has been fixed.
Bug22636 - Accumulation of datafile mappings during unshuffling
For very large databases it was possible for index optimizers to exceed address space limitations and throw errors during large optimization operations. This has been fixed.
Bug22624 - Level 2 optimization should stop after single round
Previously, if a level 2 optimization was requested and, while it was running, additional triples were loaded into the database, the level 2 optimization would run again, redoing most of the work that it had done before. This has been fixed.
HTTP Client
Rfe12546 - Implement get-duplicate-triples.
Add support for retrieving the set of duplicate triples in a repository via the following REST/HTTP service call:
GET /repositories/[name]/statements/duplicates
See Deleting Duplicate Triples.
SPARQL
Bug22692 - Improve error logging for SPARQL update commands
AllegroGraph now prints additional information to the query log when errors are encountered during SPARQL update processing.
Bug22601 - SPARQL INSERT WHERE could fail to encode typed literals
When adding triples using the SPARQL DELETE/INSERT/WHERE update command, it was possible for typed literals to be saved as unencoded UPIs (stored in the string table) rather than as the more efficient encoded UPIs (stored directly with the triples). Depending on other settings, this could lead both to inefficent queries and missing results. The problem has been corrected.
Bug22588 - Further improve handling of multiple range filters on the same variable
AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like
select * {
?s ex:test ?age .
filter( ( ?age >= 10 && ?age <= 20 ) ||
( ?age >= 60 && ?age <= 70 ) ||
( ?age >= 80 && ?age <= 90 ))
} will now execute as the combination of two range queries rather than a single scan. This change is similar to the change for bug22535 but more general.
Bug22540 - Handle failures of WHERE parts of update queries
Previously, if the query in the WHERE part of an update query failed (it timed out, ran out of memory, or was cancelled), the update failed with a bus error. This bug has been fixed.
Bug22334 - SERVICE doesn't deal with encoded IDs
SPARQL Federated query was not working correctly with AllegroGraph encoded IDs. This has been corrected.
AGWebView
Rfe12546 - Add Export Duplicate Statements link in AG Webview
On the Repository Overview page there is now an option to have the browser download the set of duplicate triples in the repository. Use the adjacent dropdown box to choose the method by which duplicates are determined (SPO or SPOG). See Deleting Duplicate Triples.
Changes to the Lisp API
Rfe12546 - Implement get-duplicate-triples.
Add support for retrieving the set of duplicate triples in a repository with the following Lisp API get-duplicate-triples, which returns a cursor of all duplicate triples visible inside the current transaction. See also Deleting Duplicate Triples.
Prolog
No significant changes.
Documentation
New document Deleting Duplicate Triples discusses the issue of duplicate triples in a store.
Python Client
Rfe12546: Add getDuplicateStatements to AGRepositoryConnection
Adds a new method to the RepositoryConnection class called getDuplicateStatements, that returns the set of duplicate triples in the repository. The method signature is:
def getDuplicateStatements(mode) Duplicate triples are determined by the mode argument, which can be specified as either "spo" (compare as ntriples) or "spog" (compare as nquads). See also Deleting Duplicate Triples.
Java Client
Rfe12546 - Add getDuplicateStatements to AGRepositoryConnection
Adds a new method to the RepositoryConnection class called getDuplicateStatements, that returns the set of duplicate triples in the repository. The method signature is:
def getDuplicateStatements(mode) Duplicate triples are determined by the mode argument, which can be specified as either "spo" (compare as ntriples) or "spog" (compare as nquads). See also Deleting Duplicate Triples.
AllegroGraph 4.14
Server Changes
Rfe13144 - Add bulk mode switch to agmaterialize command line tool
agmaterialize now supports a --bulk command-line option. If specified, then materialization will be run in bulk mode. This mode provides approximately 30% faster materialization. Note that bulk mode is global to the store and means that transactions will not be durable (in particular, will not be logged). Therefore, the switch should generally be used only when the store is not being used for other operations. See the Materializer document for information on agmaterialize.
Rfe13116 - Audit events for AGWebView login, logout, timeout
With this change, if auditing is enabled, then logIn, logOut, logOutWithTimeout events are added to the audit log whenever a user logs in, logs out, or when a login session times out. See Auditing.
Rfe13108 - Allow restricting superuser access to triples data
This change adds a new global configuration directive: SuperUserCanAccessAllData. If it is enabled (the default), then users with the Superuser permission will have read/write access to all repositories (just like in previous AllegroGraph versions). If it is turned off, then the superuser needs to be granted access to repositories. This is most useful when auditing is enabled and any change to user permissions is logged, as superusers will be able to grant themselves any permissions they want and auditing must be enabled if the fact that a superuser granted him/herself permission is to be recorded.
Rfe13104 - Notify admin by email of user account status changes
This change adds three new global configuration directives: SMTPHost, AdminEmail, and AdminEmailSMTPHost. With SMTPHost one can associate login, password, port, etc with a server:
SMTPHost gmail \
server="smtp.gmail.com", ssl=true, starttls=false,\
from="[email protected]", login="[email protected]", \
password="somepassword" If AdminEmail is defined, then emails for user account status changes events are sent to it:
AdminEmail [email protected] If there are multiple SMTPHost definitions, then one can explicitly select one of them:
AdminEmailSMTPHost gmail See the documentation in Security Implementation for more information.
Rfe13094 - Add EvalAllowed configuration directive
This change introduces the EvalAllowed global configuration directive. If it is yes (the default), then the "Evaluate arbitrary code" permission bits are effective. If it is no, then arbitrary code evaluation is disabled for all users (including superusers), regardless of the value of a user's actual "Evaluate arbitrary code" permission. Configuration directives are described in Server Configuration and Control.
Rfe13091 - Password and account expiration, suspending, disabling
This change introduces a number of parameters to control passwords and accounts. All are described in Server Configuration and Control. The new options are:
PasswordExpiry: The time since the last password change after
which the password will be expired. One cannot login with an
expired password, it can only be used to change the password.
PasswordExpiryGrace: The time since password expiry after which
the account is disabled. Only the administrator can reenable
accounts. This option does not affect users with superuser
permission.
MaxFailedLogins: The number of failed logins in a row after which
the account is suspended.
AccountUnsuspendTimeout: The time after which suspended accounts
are unsuspended.
AccountExpiry: The time since the last authenticated activity of a
user after which the account is deleted. This option does not
affect users with superuser permission. Rfe13058 - Control password complexity
This change adds four new global configuration directives: PasswordMinLength, PasswordMinUppercaseChars, PasswordMinDigitChars, PasswordMinSpecialChars to allow the administrator to specify password complexity thresholds. All are described in Server Configuration and Control.
Rfe12941 - Allow use of more than 64GiB of query memory
Previously, the start address of the heap holding the query data structures has been such that when the heap grew beyond 64GiB it was likely to collide with shared libraries mapped there which made allocations fail. This change moves the heap so that collision is less likely and much larger heaps are possible.
Rfe12929 - Add changePassword audit event
With this change, when a user's password is changed and auditing is enabled, an audit event of type <http://franz.com/allegrograph/4.11/audit-log#changePassword> is logged.
Rfe12915 - Improve reporting of shared memory allocation errors
Previously, if allocation of shared memory used by processes serving a database instance failed, the error message was an opaque 'end-of-file'. With this change, the error message expicitly mentions the allocation failure.
Rfe12884 - Include failed login attempts in the audit log
With this change, when auditing is enabled, a failedLogin event is written to the audit log on authentication failures.
Rfe12848 - Removal of query temporary files
Sometimes queries write intermediate results to disk when they will not fit in memory, for instance with an ORDER BY query. If a query failed, some of its temporary files were not removed in a timely manner. With this change, temporary files are always deleted whether the query finishes or fails.
Bug22527 - Sorting UPI maps
Previously, on certain filesystems sorting of UPI maps could fail, which broke the repository instance. The only indication of this bug was the log message "Sort command failed". This bug has been fixed.
Bug22507 - Include id of deleted user in audit report
Previously, the audit report did not include the user id of the affected user associated with deleteUser events. This bug has been fixed.
Bug22466 - Setting geospatial type mappings did not register the subtype
Previously, setting a geospatial datatype or predicate mapping did not also add the geospatial subtype to the triple-store. This has been corrected.
Bug22465 - In the turtle format, QNames are a superset of XML QNames
AllegroGraph's turtle parser required QNames (like rdf:type) to be XML QNames but the turtle standard approved in February 2014 allows additional flexibility. In particular, the first character of the local part of a QName may start with a digit. Previously, a QName like rdfs:0 would have signaled a parse error. Now, it will be accepted.
Bug22450 - RDF/XML serializer could re-use node abbreviations
For efficiency, the RDF/XML and RDF-N3 serializers maintain a mapping from URI to prefix abbreviation (e.g., from
Bug22422 - Federated triple-stores cannot use geospatial magic properties
The check for whether or not a triple-store could use a geospatial magic property was too stringent. This has been corrected so that federated triple-stores can be queried with geospatial magic properties. Note that every store in the federation must agree on any geospatial predicate type mappings used in the query. If they do not, then an error will be signaled during query planning.
Bug22371 - Fix race condition when using free-text indices.
Previously, a race condition existed where a client A would fetch the list of free-text indices on a repository, but have the set of indices modified by a client B, thus rendering client A's list out of sync. This could result in unexpected errors in certain situations, such as when opening repositories. Now, attaching to free-text indices is synchronized so that this race condition is avoided.
Bug22363 - Fix log rotation
A bug was introduced in v4.13 where a SIGHUP sent to the Service Daemon would not cause the log file to be closed and reopened. This is now fixed.
Bug22345 - Use customer namespaces in remote clients
Similar to rfe12679 (under SPARQL below), AllegroGraph now uses customer namespaces when serializing output in RDF/XML for all clients. Previously, temporary namespaces were used.
Bug22319 - Materializer not inferring rdfs:subClassOf reflexivity
The materializer was not always correctly inferring that classes were rdfs:subClasses of themselves. This has been corrected. This change also improves the performance of AllegroGraph's OWL 2 RL materializer significantly.
Bug22300 - Fix error when system time is in the past
Previously, if the system time in the environment of deployment was before the build time of the AllegroGraph, then queries would fail with an error saying:
Attempt to do an array operation on nil which is not an array. This bug has been fixed.
Bug22297: ensure-not-lingering should latch
Previously, if a request to delinger a store could not be honored because it was still in use by another process, the request would be ignored. Now, the request is deferred until the store is no longer in use.
Bug22293 - Make agraph-backup support dynamic catalogs.
Previously, backup and restore operations would fail when operating on repositories in dynamic catalogs. Backup operations were unable to locate the main directory of the repository. Restore operations were unable to locate the main directory of dynamic catalogs.
Now, backup and restore operations support repositories created in dynamic catalogs. They follow the same rules as other repositories, most notably that when performing a restore operation, all dynamic catalogs must already have been created, or the restore will fail.
See Catalog definitions in Server Configuration for a description of dynamic catalogs.
Bug22259 - Ensure proper transaction log stream alignment on restart
Previously, under certain unusual circumstances, the transaction log stream could be left in a state that prevented a database from reopening properly. This has been fixed.
Bug22243 - Less fragmentation in the shared memory
With this change, the shared memory allocator is more aggressive about coalescing freed regions which reduces the tendency of shared memory to fragment.
Bug22229 - Geospatial type mappings could cause duplicate triples
If a triple-store registered a predicate and a datatype geospatial type mapping that applied to the same triple, then that triple would be added to the store twice. This has been corrected.
Bug22228 - Rare error when opening a triple store
Under extremely rare circumstances, opening a triple store could result in the following error:
The assertion (integerp db.agraph.storage::chunk-size) failed. This has been fixed.
Bug22226 - More efficient memory usage
With this fix, the query engine does a more thorough job of giving freed memory back to the operating system which enables more efficient operation under high concurrency workloads.
Bug22211 - Trix parser did not treat Return (ASCII 13) as whitespace
The TriX parser was not treating Return characters (ASCII 13) as whitespace which sometimes led to parse errors. This has been corrected.
Bug22208 - Concurrent queries in shared backends
When the system is under load, a shared backend may execute several queries concurently. Under rare circumstances, the queries involved could change each other's query options and produce incorrect results. This bug has been fixed.
Bug22198 - Listing freetext indices should not error for federated triple-stores
Previously, federate-triple-stores would signal an error when asked to list their freetext indices. They now return an empty list.
Bug22141 - Attempt to close instance during agraph-backup --supersede restore operations.
Previously, using agraph-backup to restore into an open triple-stores would fail. Now, when the --supersede option is specified, such restores can succeed. agraph-backup is described in Backup and Restore.
HTTP Client
Rfe12963 - SERVICE URIs with query strings
Previously, AllegroGraph dropped the query string in a SERVICE URI when making a request. In this example:
SERVICE <http://sparql.bioontology.org/ontologies/sparql?apikey=XXX> the 'apikey' was not sent to the remote web server. This has been fixed.
Rfe12952 - Add global configuration directive UseMainPortForSessions
Normally, HTTP requests to a dedicated session go directly to the process serving that particular session. This has low overhead, but it can make firewall management more difficult because more ports have to be open. With this change, the new UseMainPortForSessions directive may be enabled in the configuration file which makes all communication be proxied through the front-end. Previously, similar functionality existed only in the Java client controlled by the UseMainPortForSessions property.
Rfe12798: Implement repo-specific scripts
The existing scripting HTTP API (sitescripts) manages scripts that can be used with any repository. An HTTP API has been added for managing scripts that are local to a given repository, called "reposcripts". Reposcripts can only be loaded and run against the repository under which the script was loaded. See Scripting the server in HTTP Protocol for more information.
Bug22524 - HTTP protocol should not send partial responses
Previously, queries that used Chunk-at-a-Time (CaaT) processing over the HTTP REST interface would begin streaming results back as soon as the first chunk was processed. If a query timeout was in effect, this meant that AllegroGraph could issue a 200 (OK) response and then only send back only a partial response body. To prevent this, AllegroGraph now computes the entire query result before sending any HTTP response when there is a time out active.
Bug22476 - Unauthorized response for javascript eval
Previously, if the user had no eval permission, attempts to execute javascript failed with HTTP status 400 and a "MALFORMED PROGRAM" message. With this change, the coreect HTTP status 401 'unauthorized' is returned.
Bug22445 - Don't percent-decode path component of HTTP requests twice.
Previously, path components of HTTP requests sent to AllegroGraph were sometimes being percent-decoded twice. For example, the URI
http://localhost:10035/repositories/test%252frepo/statements could be seen by AG as either
http://localhost:10035/repositories/test%2frepo/statements or
http://localhost:10035/repositories/test/repo/statements depending on the URI being requested. This bug isn't likely to have been encountered unless you were attempting to include special characters in repository or catalog names. It is now fixed. Users with applications that make direct HTTP requests should check their request URIs to ensure that they do not double-percent-encode any path components, as these may stop working or have unexpected behavior. Percent-decoding of URI query arguments is not affected.
Bug22369 - Fix server-side javascript http-request property values.
Previously, the property values of javascript http-request objects (method, url, body, etc.) could raise errors due to returning invalid objects. This bug has been fixed, and return values should match what is described in the documentation.
Bug22329 - Fix handling of HTTP 'Expect: 100-Continue'
Previously, AllegroGraph did not send the '100 Continue' response to requests with 'Expect: 100-Continue' headers when the request was forwarded internally to another component. This happened, for instance, when a request to a shared backend or a dedicated session was forwarded via the frontend and also when a requested made directly to a dedicated session was forwarded to the frontend. With this change, '100 Continue' is sent as appropriate.
SPARQL
Rfe13029 - Use POST instead of GET for remote SPARQL queries
AllegroGraph now uses HTTP POST requests instead of HTTP GET requests when talking to remote SPARQL endpoints. This allows for larger query strings to be sent.
Rfe13028: Improve use of partial results when making SERVICE queries
AllegroGraph now makes more use of partially computed results when making federated SPARQL queries using the SERVICE clause. This results in more efficient queries against SPARQL endpoints.
Rfe12911 - Improve estimates for magic geospatial properties
AllegroGraph's size estimates for geospatial queries that use SPARQL magic properties are now more accurate. This improves query clause reordering and can greatly improve query performance.
Rfe12891 - Improve efficiency of some SPARQL geospatial queries
Some queries that use SPARQL magic properties to use AllegroGraph's geospatial engine will now operate more efficiently.
Rfe12822 - Add temporaryFileDiskSpace query option
Sometimes queries write intermediate results to disk when they will not fit in memory, for instance with an ORDER BY query. With a huge query it is possible for a such temporary files to fill the filesystem. In order to prevent this, the temporaryFileDiskSpace query option may be set. For example:
PREFIX franzOption_temporaryFileDiskSpace: <franz:10G> As with all query options, it is also possible to change the default value globally in the AllegroGraph configuration file. For example:
QueryOption temporaryFileDiskSpace=10G The default is to have no limit. Query options are described here in the SPARQL Reference; the AllegroGraph configuration file is described in Server Configuration.
Rfe12788 - Raise error for invalid use of SPARQL geospatial functions
AllegroGraph now signals an error at parse time when a geospatial function is used with a predicate that is not associated with a geospatial type-mapping.
Rfe12750: SPARQL Property Path cache improvements
SPARQL Property Path computations use a cache to improve the efficiency of transitive closure calculations. The data structures for this cache have been rewritten to be more memory and time efficient. Secondly, the heuristic that the query engine uses to determine whether or not the cache should be aggressively or lazily filled have been improved.
Rfe12284 - Signal an error if a SPARQL GEO pattern uses a constant object or graph place
The SPARQL GEO Clause uses its first pattern to link with AllegroGraph's geospatial system. The object place of this pattern must be left unbound. Previously, AllegroGraph would silently return no results when the place was bound. Now, it will signal an error. For example, the following query would have previously run but returned incorrect results but now it will signal an error.
prefix s: <http://franz.com/ns/allegrograph/3.0/geospatial/spherical/km/-180.0/180.0/-90.0/90.0/>
select distinct ?event ?coord ?date ?platform ?EventType {
GEO SUBTYPE s:1.0
BOUNDINGBOX( POINT(-084,34), POINT(-082,36)) {
# this is an error because the first clause is the one
# that connects the query with AllegroGraph's geospatial
# sub-system
?event a <http://UCore-SL-Taxonomy-OWL-DL#ObservationEvent> .
?event mo:hasGeospatialCoordinate ?coord .
}
WHERE { }
}
Rfe12240 - Improve bound variable analysis when using magic properties
The SPARQL 1.1 query planner now recognizes that variables set via assignment using the BIND clause can be treated as bound when determining whether or not a magic property can be used. This means that queries like
select ?event ?coord {
bind(44.30 as ?xMin) .
bind(33.40 as ?yMax) .
(?event ?coord) franzGeo:inBoundingBoxXY (swp:startLatLong ?xMin 44.50 33.30 ?yMax)
} can now be used.
Rfe11880 - Improve handling of some SPARQL GRAPH clauses
AllegroGraph's analysis of when and how it can execute SPARQL GRAPH clauses has been improved. Depending the query, this enhancement can speed execution by a factor of 10 or more.
Rfe10132 - Improve performance of find unique values for a column queries
SPARQL queries that ask for the DISTINCT values of a particular column are now sometimes significantly faster. Previously, they performed linearly in the size of the triple-store; now, they perform linearly in the size of the number of values returned. In particular, for stores with a small number of predicates or graphs (relative to the size of the store), queries like these will be much, much faster:
SELECT DISTINCT ?p { ?s ?p ?o }
SELECT DISTINCT ?g { graph ?g { ?s ?p ?o } }
SELECT DISTINCT ?class { ?s a ?class } Note that aggregate queries still require a full-scan of the store, so a query like:
SELECT ?p (count(1) as ?pCount) {
?s ?p ?o .
} will still be expensive.
Bug22529 - Improve use of intermediate information during query
AllegroGraph's set based SPARQL 1.1 query engine sometimes used an intermediate result set when it would have been more efficient to compute the solutions directly. This has been corrected.
Bug22535 - Improve handling of multiple range filters on the same variable
AllegroGraph now handles queries with multiple range filters more efficiently. For example, a query like
select * {
?s ex:test ?age .
filter( ( ?age >= 10 && ?age <= 20 ) ||
( ?age >= 60 && ?age <= 70 ))
} will now execute as the combination of two range queries rather than a single scan.
Bug22485 - Improve pattern size estimation logic
AllegroGraph's estimation of the size of triple-patterns in a BGP has been improved. This in turn improves the automatic BGP re-ordering and leads to better query plans when using the statistical clauseReorderer.
Bug22430 - Cross products with unbound variables could lose results
Depending on the query algebra and the order in which clauses were evaluated, it was possible for AllegroGraph to lose some solutions with unbound variables when computing a cross product. This has been corrected.
Bug22391 - Make SPARQL rand() results more independent
Due to an unintended sharing of random number generator state, SPARQL rand() calls in queries running on the same backend can be produce the same stream of pseudo random numbers.
Bug22389 - The chunkProcessingMemory query option could not be specified in a query
The chunkProcessingMemory could only be set using the AllegroGraph config file rather than being able to also specify it in using the PREFIX notation in a SPARQL query. This has been corrected.
Bug22480 - Range queries could cause other filters to be ignored
If a query had multiple FILTERs that applied to a single pattern and one of these FILTERs created a range query, then it was possible for the other FILTERs to be ignored. This has been corrected.
Bug22372 - SPARQL JSON results format was using literal-typed instead of typed-literal
The SPARQL JSON results serializer was emitting literals that had an associated datatype as
"name" : {"type":"literal-typed", "datatype":" D ", "value":" S "} rather than
"name" : {"type":"typed-literal", "datatype":" D ", "value":" S "} This has been corrected.
Bug22348: EXISTS and NOT EXISTS FILTERs could fail
If an EXISTS or NOT EXISTS FILTER used variables from the outer query in a FILTER (rather than a BGP), then too many solutions could be returned. E.g., a query like:
PREFIX : <http://example.com/>
SELECT ?x ?n WHERE {
?x :p ?n
FILTER NOT EXISTS {
?x :q ?m .
FILTER(?n = ?m)
}
} uses the bindings for ?n in the NOT EXISTS FILTER and ?n is not one of the variables that the NOT EXISTS FILTER binds itself. Previously, AllegroGraph will leave ?n unbound and therefore return too many results. This has been corrected.
Bug22336 - Allow the use of magic properties in more situations
The SPARQL 1.1 query planner ensures that a magic property can only be used when all of its required variables are bound. Previously, however, the test for bound variables was overly strict and did not, for example, take BIND or VALUES into account. This has been corrected. For example, this query would previously have been disallowed:
prefix sna: <http://franz.com/ns/allegrograph/4.11/sna/>
prefix pred: <http://www.testData.com/predicate/>
prefix : <http://ex#>
select * {
values ?start { <http://www.testData.com/graph/10444> }
?graph sna:nodalNeighbors (:graphSubClass ?start) .
} because AllegroGraph would not have seen that ?start is bound by the VALUES clause. Now this query will function as expected.
Bug22347 - Query options were sometimes ignored in SPARQL update commands
Previously, AllegroGraph PREFIX options were ignored during the WHERE clause of a SPARQL DELETE/INSERT/WHERE command. This has been corrected.
Bug22321 - Use other sets more in many more cases
The set-based SPARQL query engine now makes better use of partially computed result-sets during query execution. This can greatly accelerate many queries.
Bug22272 - Correct unbound variable handling in MINUS
In rare circumstances variables that were unbound in the right-hand side of a SPARQL MINUS clause could be incorrectly matched with bound variables on the left hand side. This has been corrected.
Bug22270 - Correct BGP estimation for predicates not in the store
If a query clause had either a fixed subject or object and a predicate that was not in the triple-store, then AllegroGraph's statistical estimator could return a positive number rather than zero. This has been corrected.
Bug22253 - Combining reorderDuringExecution and Property Path queries could lose filters
Under rare circumstances, the query engine could lose track of FILTERS in queries that used SPARQL Property Paths and the AllegroGraph reorderDuringExecution option. This has been corrected.
Bug22246 - Catch argument count mistakes in query functions
SPARQL queries that contained functions being called with an incorrect number of arguments would silently fail. These errors are now detected and reported at query time. For example, a query like:
prefix gf: <http://franz.com/ns/allegrograph/3.0/geospatial/fn/>
select * {
bind( gf:cartesianDistance(1, 2, 3, 4) as ?b )
}
would have previously returned a single unbound result because the
`cartesianDistance` function should only be given two arguments. Now,
AllegroGraph will signal an error. Bug21941 - Magic properties could lose track of FILTERs
In rare circumstances a query that used magic properties and FILTER on some of the variables in the properties could incorrectly filter out too many solutions. This has been corrected.
AGWebView
Rfe13159 - Enhance AGWebView query GUI
AGWebView's SPARQL query user interface now supports query log viewing, query plan inspection and reports additional details about query execution such as query time and warnings. The HTTP REST API for query has a new parameter, returnQueryMetadata. If supplied and true, then the HTTP response will include a JSON payload containing information about the query in an HTTP header named x-query-info. The revised query page is described here.
Rfe13103 - Reposcripts added to AGWebView
Access to the new Reposcripts API has been added to AGWebView (see the Rfe12798 release notes entry, which has links to other documentation).
Rfe13068 - Login timeout for AGWebView
The new global configuration directive LoginTimeout controls the expiry of inactive AGWebView login sessions. See the description of that option in Server Configuration and Control for the details.
Rfe12925 - Only run AGWebView queries when the Execute button is pushed
Previously, AGWebView executed recent and saved queries as soon as they were selected. Now AGWebView only executes queries when the execute button is clicked.
Rfe12579 - Allow users to change their own passwords
This change introduces the PasswordChangeAllowed global configuration directive. If it is yes (the default), then users can change their own passwords via AGWebView. This is a change in default behavior (there was no facility to allow users to change their own passwords in earlier releases). If you set the value of the option to no, the old behavior will be restored. Configuration directives are described in Server Configuration and Control. User passwords are described in Security.
Rfe12275 - Add query cancellation interface to AGWebView's query pane
In addition to the existing job canceling screen, AGWebView now provides query cancellation directly from the SPARQL query interface. The revised query page is described here.
Bug22536 - Simple queries may not show up in the jobs list
Previously, simple queries like
select { ?s ?p ?o . filter( fn(?o) ) } could fail to appear on the query jobs list in AGWebView. AllegroGraph now checks for query timeout more often which means that even these queries will appear and be cancelable.
Bug22260 - Fix AGWebView audit log date range
Previously, when the start date or the end date was specified in the audit log query (at /#audit), then the query failed with a type error. This bug has been fixed.
Bug20820 - AGWebView's multi-core data import could not specify a graph
When using multi-core loading, AGWebView was incorrectly formatting the graph it sent to AllegroGraph's parallel data import tool. This meant that it was impossible to specify a graph. This has been corrected.
Changes to the Lisp API
Rfe12862 - Extend Lisp geospatial functions with subject, object and graph arguments
Add s, o and g arguments to the Lisp API geospatial functions:
- get-triples-haversine-miles
- get-triples-haversine-km
- get-triples-geospatial-bounding-box
- get-triples-geospatial-radius
The new arguments act as additional filters on the triples returned by the functions. g can only be used when the use-g argument is nil and o can only be used when the use-g argument is non-nil.
Rfe12830 - Warn when a query has constants that are not in the store
In the Lisp API, the fourth return value of run-sparql now contains information about any warning that occurred during query planning and execution. This includes a warning when a query contains a constant that is not in the store. For example:
select * { ?s <ex://#notHere> ?o } would generate a warning regarding <ex://#notHere>. Additional warnings will be added in the future.
Bug22322 - remote-triple-store does not honor default-prefixes argument to run-sparql
The default-prefixes argument to the Lisp run-sparql function was being ignored for remote-triple-stores. This has been corrected.
Bug22225 - Memory corruption in the Lisp direct client
Previously, if an unclosed database object became garbage, memory corruption could occur which lead to unpredictable errors such as segmentation faults under extremely rare circumstances.
Bug22167 - Fix flaws in delete-duplicate-triples.
delete-duplicate-triples is documented as preserving the triple with the lowest possible triple-id from a group of duplicate triples. Flaws in the algorithm used to delete duplicate triples made it possible that the triple preserved would not necessarily have the lowest possible triple-id, and could depend on the order in which the duplicates were sorted. This change corrects that problem.
Prolog
Bug22179 - Allow safe Prolog query requests in shared backends
Previously, AllegroGraph only allowed Prolog queries when a user had eval privileges for the store. Now, eval privileges are required only for Prolog queries that make use of functors that escape to Lisp.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Rfe12952 - Update Java client for handling of useMainPortForSessions
When the server configuration option UseMainPortForSessions is enabled, all requests to sessions are multiplexed through the frontend port. In this case, the Java system property useMainPortForSession is redundant and its value is not consulted. This means that one can turn on this feature on the client if it's not enabled in the server config, but cannot turn it off.
Bug22392 - Hang with useMainPortForSessions
Previously, when the Java client's useMainPortForSessions feature was enabled, a request returning a large response could hang. This bug has been fixed.
Bug22323 - Fix UseMainPortForSessions when using SSL
Previously, when the com.franz.agraph.http.useMainPortForSessions property was set to true, the Java client arranged for all communication with a session be funneled through the front-end. This fix makes this feature work with SSL connections. Also, with this change it is required that SSLPort is set in the configuration file, otherwise using SSL connections to dedicated sessions with fail.
AllegroGraph 4.13.2
The only change was to replace the SSL library with one that avoided the heartbleed bug.
AllegroGraph 4.13.1
Server Changes
Bug22255 - Concurrent full index optimization requests
Previously, if index optimization was requested with a full index optimization already in progress, the database instance would log an error and hang. This bug has been fixed.
AllegroGraph 4.13
Server Changes
Rfe12645 - Add a command line interface to AllegroGraph's materializer
AllegroGraph now includes a command line tool to materialize the inferred triples in a triple-store. The tool is named agmaterialize. It can currently be run only on the same machine where the AllegroGraph server is running. The command line interface is described in the Materializer documentation.
Rfe12642: Improve handling of datafile free space information
Previously, datafile free space information was maintained in a fixed sized data structure which could run out of room after a certain level of datafile fragmentation was reached. This has been corrected.
Rfe12590 - Shared backend scheduling improvements
Previously, when all shared backends were busy, further requests were held until a backend became available. Such pending requests were scheduled for execution in basically random order with latency up to 1s.
With this change, pending requests are scheduled rather than essentially random. The scheduling is basically FIFO (first in, first out, meaning the oldest request is processed first), but FIFO may be violated for efficiency reasons. (Strict FIFO behavior is undesirable, because it greatly reduces AllegroGraph's ability to schedule requests optimally under stress.)
Also with this change, scheduling latency is now minuscule and shared backends are now allowed to execute slightly more queries in parallel if all backends are busy, which allows greater efficiency in most cases.
Rfe9523 - Logging improvements
This change adds a centralized logging facility to the AllegroGraph daemon. The new logger guarantees no interleaving of messages and is able to deal with full filesystem conditions gracefully. Lisp direct clients have now the option of sending log messages to an AllegroGraph server with the with-log-routed-to-server macro. See the documentation for more. Previously, log messages were sent to the AllegroGraph server belonging to the first direct connection opened in the client.
Bug22220 - Race condition while determining last written transaction log id
Previously, a rare error could be encountered when starting replication or while switching roles, indicating that replication failed due to a missing transaction log that had not yet been written to disk. This problem has been fixed. Now, only transaction logs on disk are used to start replication.
Bug22184 - Instance shutdown and replication
In rare circumstances, when the instance belonging to a replicated repository was shut down, writing the transaction log could fail. This bug has been fixed.
Bug22156 - date-string-to-upi doesn't handle zero time zones consistently
The date-string-to-upi functor was not treating a Zulu time zone specified as "Z" consistently with the rest of AllegroGraph's date and time parsers. This has been corrected.
Bug22111 - Database names may not contain special characters
Because it is impossible to reliably use databases (also called repositories) in session specifications if their names contained characters that are used in those specifications, the list of characters which must not be used in names has been expanded. The following characters are now disallowed in database (repository) names:
\ / ~ : Space $ { } < > * + [ ] | Previously, only the following four characters were disallowed in database (repository) names:
\ ~ / : Note that this change only effects new databases (repositories); existing databases (repositories) with any of the newly invalid characters in their names will continue to be accessible. It may, however, result in difficulties when attempting to use them in more complex session specifications.
Bug22100 - More strict, faster parsing of some xsd datetimes
The parsers for xsd:date, xsd:dateTime and xsd:time used to be lenient and accept things like:
20131118 # missing separator
2013-11-18T12:27 # missing seconds and many other convenient but non-standard formats. The new parser only accepts what the XML Schema specification mandates which also allows it to be faster. See XSD Date and Time datatypes for more information on these datatypes.
Bug22086 - Materializer deletes triples with the same subject, predicate and object
The initial version of AllegroGraph's materializer deleted all triples with the same subject, predicate and object. This limitation has now been lifted: the materializer now only deletes triples with matching subject, predicate, object and graph.
Bug22032 - Control whether or not external references are followed during RDF/XML import
Previously, AllegroGraph ignored external references in RDF/XML source files. The Lisp API now includes a new keyword argument to load-rdf/xml:
- :resolve-external-references : if true, the external references will be followed. The default continues to be
nil.
agload also has a new command-line switch. Supplying either of -x or --external-references will cause agload to follow external references. See Data Loading.
Bug21953 - Allow import of triples to reasoning-triple-store
A bug was introduced in v4.12 that prevented import of triples via load-* routines (e.g. load-rdf/xml) on reasoning-triple-stores, when the source argument was passed a URI. This problem has been fixed. This change also provides a better error message when attempting to import statements to a federated-triple-store, which is not allowed.
Bug21656 - Cleaner daemon shutdown
When the AllegroGraph daemon was stopped, despite the claim of doing a 'clean shutdown' in agraph.log, the shutdown of the instances was not clean and the corresponding repositories would potentially require crash recovery the next time they were opened. Also, such a shutdown lost any index optimization work performed since the most recent checkpoint.
With this change, instance processes perform a checkpoint if necessary which makes shutdown slower, but startup faster and it makes sure the index optimization results persist.
Bug20773 - Support federations of RDFS++ stores
Previously, queries against a federation of RDFS++ stores would cause an error during query planning. This has been corrected.
HTTP Client
Rfe12763 - Update HTTP API for the materializer
The REST API for materializeEntailed now includes an inferredGraph argument that can be used to specify which graph newly entailed triples are added to (or deleted from). The materializer is described in Materializer.
SPARQL
Rfe12767 - Improved the efficiency of single group aggregate queries
The code generated for single group aggregate queries is now slightly more efficient.
Rfe12762 - Optimizing level 2 index optimization
With this change, level 2 index optimization (the default) no longer tries to optimize access for triples added since optimization was started. This allows index optimization to finish even if triples are added continuously at a high rate. The new code is faster on average, has the same worst case disk usage (about two times the original size), but it has a higher average disk usage.
Rfe12760 - Improve efficiency of some SPARQL Property Path queries
AllegroGraph now builds some property path data structures as needed rather than computing all of them up front. Depending on the data, this can greatly enhance the efficiency of some queries.
Rfe12757 - Improve efficiency of some property path queries
In some circumstances, AllegroGraph could choose the less efficient of two strategies when evaluating a property path query. AllegroGraph now ensures that it makes the better choice.
Rfe12717 - Handle RDFterm-equality of strings consistently
Previously, AllegroGraph implemented sameTerm() using a strict interpretation of RDF literal equality when comparing plain literals and typed literal strings but implemented = using a more relaxed interpretation. These tests are now done consistently using the relaxed interpretation. In particular,
'hello' is RDFterm-equal to 'hello'^^xsd:string This is in accordance with both common usage and the RDF 1.1 specification.
Rfe12699 - Improve efficiency of SPARQL equality tests on resources
The efficiency of equivalency tests in SPARQL filter expressions has been improved.
Rfe12681 - Optimize interprocess communication
The performance of message passing between different AllegroGraph server processes has been slightly improved.
Rfe12679 - Use query local namespace abbreviations in result serialization
AllegroGraph now uses the namespaces defined by a SPARQL query's prefixes when serializing output in RDF/XML. Previously, temporary namespaces were used instead.
Rfe12670 - Allow /SPARQL access to AllegroGraph as a SPARQL endpoint
Previously, access to an AllegroGraph triple-store as a SPARQL endpoint was via sending a query to
SERVER:PORT/respositories/NAME?query=... This REST API has been augmented so that sending a query to the triple-store's URL with a trailing /sparql also works. E.g.,
SERVER:PORT/respositories/NAME/sparql?query=... This is not required by the SPARQL standard but is in conformance with many other existing endpoints.
Rfe12657 - More efficient handling for the VALUES clause
AllegrGraph now rearranges queries so that VALUES clauses are evaluated first when possible. This improves query efficiency as VALUES clauses are generally small.
Rfe12656 - Improve execution of some nested joins
AllegroGraph now evaluates nested joins more efficiently.
Rfe12498 - Improve full-scan warning heuristics
AllegroGraph warns when a SPARQL query executes a full-scan. Heretofore, this meant that a query like
select * { ?s ?p ?o } limit 100 could have produced a full-scan warning even though it is benign. AllegroGraph now saves its warnings for when they may actually indicate a problem with the query.
Rfe11981 - New query option chunkProcessingMemory
Previously when chunk-at-a-time processing was enabled (see query option chunkProcessingAllowed), the query option chunkProcessingSize controlled memory consumption by limiting the number of intermediate results stored in memory. The total memory consumption depended on the number of variables and the complexity of the query, so it was rather difficult to know how much a query will actually use. With this change, chunkProcessingSize is deprecated and chunkProcessingMemory can be specified instead. The unit of chunkProcessingMemory is a byte but the usual abbreviations are also accepted ("8G" (the default), "2M").
Rfe11515 - Fully support the SPARQL 1.1 HTTP Protocol
AllegroGraph now fully supports the five different HTTP request types specified in the SPARQL 1.1 HTTP Protocol. These are:
- Query
- via GET with query parameters
- via POST with urlencoded parameters
- via POST with a plain query string (using the content-type of application/sparql-query).
- Update
- via POST with urlencoded parameters
- via POST with a plain update request string (using the content-type of application/sparql-update).
In addition, AllegroGraph continues to support SPARQL update using POST where the update command is encoded in the query portion of the HTTP request.
Bug22350 - Distinct queries could produce duplicate results in rare cases
Under rare circumstances it was possible for the query engine to produce duplicate results when using chunk-at-a-time (CaaT) processing. This has been corrected.
Bug22214 - Encoded IDs failed to compare properly in SPARQL queries
Inequality comparisons between encoded IDs were always returning true. This has been corrected.
Bug22204 - decimal to decimal cast loses precision
When in a SPARQL query an xsd:decimal was cast to xsd:decimal it lost precision as it was converted to a single float internally. This bug has been fixed.
Bug22203 - Fix decimal parsing in SPARQL queries
In SPARQL query strings, decimals in the (-1,0) interval were parsed with the wrong sign. For example, -0.9 was parsed as 0.9. This bug has been fixed.
Bug22153 - Some wild card property path queries could generate incorrect paths
In rare circumstances, the data structures used to cache property path information could be invalidated leading to too many path results. This has been corrected.
Bug22147 - Add support for ObjectId's in the _id field.
The SPARQL MongoDB magic predicate now works with documents containing ObjectId's in the _id field.
Bug22142 - Expressions on aggregates fail for empty group
If a query wrapped an aggregate with an expression like
select (xsd:float(avg(?age)) as ?avgAge) ... and the result set contained an empty group (i.e., a group with no members), then AllegroGraph would signal error. This has been corrected.
Bug22134 - SPARQL aggregates on expressions with errors could compute incorrect results
Other than COUNT, a SPARQL aggregate on an expression should return unbound if that expression evaluates to a type error during computation. AllegroGraph was incorrectly returning the aggregate of any bindings produced after the last type error. This has been corrected.
As an example, if AllegroGraph needed to compute the MIN( ?p * 2 ) and ?p was bound to 4, 7, 15, "hello", 6, and 1, then AllegroGraph would previously have returned 2 (the minimum of 12 and 2). Now, it will return no binding for that expressions.
Bug22123: Empty construct template will cause an error
If a SPARQL CONSTRUCT had an empty template like
CONSTRUCT {} WHERE { ?s a ?o } then AllegroGraph would signal an error. This has been corrected.
Bug22094 - Correct sparql< and sparql> for date and dateTime comparisons
Because there is no defined operator mapping for inequality comparisons between xsd:date and xsd:dateTime, comparing them should throw a SPARQL type-error. AllegroGraph was incorrectly returning nil instead. This has been corrected. See XSD Date and Time datatypes for more information on these datatypes.
Bug22074 - DELETE templates with unknown variables caused an error
AllegroGraph would signal an error if a SPARQL DELETE/WHERE command's DELETE clause contained variables that were not bound by the WHERE clause. This has been corrected. An example of such a command would be:
DELETE { graph ?unknownGraph { ?s ?p ?o } }
WHERE { ?s ?p ?o }
Bug21561 - Remote-triple-stores do not support the table results format
Remote-triple-stores did not support returning tables from SPARQL queries. This has been corrected.
AGWebView
Bug22161 - Adding security filters in Firefox
Previously, with Firefox, clicking the 'add' button in the security filters table of the user management page (Admin/users in the menu) of AGWebView did not submit the form. This bug has been fixed.
Changes to the Lisp API
Rfe12632: Improvements in Lisp API to AllegroGraph materializer
Materialization previously occurred only for triples in the default-graph of the triple-store. It now occurs for all of the graphs in the store. Materialization on subsets of the graphs can be accomplished by creating a graph-filtered triple-store and calling materialize-entailed-triples on it.
Materialization previously placed inferred triples into a special graph that could not be utilized by the rest of AllegroGraph (the special graph behaved similarly to but not exactly like the usual AllegroGraph default-graph). It now places inferred triples into a graph that can be specified in the call to materialize-triples using the materialized-graph argument. It defaults to http://franz.com/ns/allegrograph/4.12/materializer/defaultGraph.
Two new arguments were added to delete-materialized-triples. These are:
db- which controls the triple-store that is modified.materialized-graph- which controls the triples that are deleted. It defaults to http://franz.com/ns/allegrograph/4.12/materializer/defaultGraph.
Materialization now logs its start and end to the AllegroGraph log file. The overall efficiently of the materializer has also been improved.
Bug22132 - calling run-sparql with the :with-variable argument with a timezoned upi
When the Lisp function run-sparql was used in conjunction with a :with-variable argument that bound a variable to an encoded xsd:dateTime or xsd:time, then in some cases comparisons involving that variable failed to match equivalent representations of the same xsd:dateTime or xsd:time with a different timezone. This bug has been fixed.
Prolog
Bug22171: Select0 should return UPIs when it can
Previously, the Prolog select0 query macro could return future-parts rather than UPIs. For example, this query would return a future-part.
(select0 (?a) (= ?a !rdf:type)) This could cause problems in the unlikely event that namespace abbreviations changed when running the same query multiple times. select0 now returns UPIs so that the above behavior is not possible. Note that if a future-part represents a triple that is not currently in the triple-store, then a query can return UPIs that cannot be printed (because the UPI will not be in the string dictionary).
Bug22157 - Temporal functors could signal an error when used with all bound values
Previously, AllegroGraph used real numbers to represent times, dates and dateTimes. Now, the internal representation includes time-zone information which means that comparing two values can no longer use simple inequalities. The Prolog temporal functors like point-before-datetime had not been updated to reflect these changes. This oversight has been corrected. This change means that the Prolog functors will use the same XSD logic used by the SPARQL date and time functions.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
AllegroGraph 4.12.2
Server Changes
Rfe12734 - More aggressive level 2 index optimization
In some cases, level 2 index optimization (the default) left a small part of the index unoptimized. With this change, the entire index is optimized in all cases.
Rfe12726 - Index optimization increasing database size on disk
Space in some garbage on-disk data structures is now reclaimed in a more timely manner which makes the increase in database size during index optimization less severe.
Rfe12725 - Faster level 2 index optimization
Level 2 index optimization (which is the default level) has been made faster.
Rfe12636 - Make config-file argument to agraph-backup backup-settings optional.
Previously, the agraph-backup backup-settings command required a config-file argument. This argument has been eliminated in favor of an optional --config option that can be used in place of the --port argument. If both --port and --config are supplied, then the port argument takes precedence. Also, agraph-backup could fail in the past if there were discrepencies between a --port argument specified on the command line, and a Port directive contained in a config file. Now, if the --port argument is specified, it will take precedence over any Port directives read from a config file. See Backup and Restore.
Rfe12635 - Major revision to agraph-backup operating modes.
The various backup and restore modes of agraph-backup have been overhauled.
Previously, backup mode created an archive file containing only repository data. Now, it creates an archive directory that contains both repository data and settings. This archive can be restored via restore or restore-all. Previously, restore mode operated on an archive file. Now, it can also operate on an archive directory created by backup or backup-all. In either case, it will restore a single repository (data and settings) from the archive.
The layout of archive directories created by backup-all are slightly changed. The restore-all mode will now fail if repositories being restored to already exist. Use the --supersede option to force an overwrite of any such repositories.
Previously, the backup-settings mode created an archive containing both repository and system settings. It also supported the --ext-backup-command option. Now, it only archives system settings and the --ext-backup-command option is no longer supported. Adds a restore-settings option that restores system settings, overwriting any existing settings. It operates on any valid archive directry (though ones created by backup will likely not have any system settings data).
Please refer to the updated Backup and Restore documentation for full details.
Rfe12621 - xsd:time improvements
Zoneless xsd:times are no longer converted to zero timezone when added, and timezone offsets roundtrip to and from the database. Also, with this change xsd:times and xsd:dateTimes are no longer comparable. This puts xsd:time on equal footing with xsd:dateTime. See the date and time documentation for more information.
Rfe12580 - Don't create repo if client/server uids don't match
Previously, if the uid of the direct Lisp client and the AllegroGraph server differed, then a create-triple-store call would fail with a `server-pid-and-client-pid-must-match-error` indicating that the creation failed. However, the repository would actually be created, possibly overwriting an existing repository. Now, the uid check is performed earlier in the create repository process, so that the repository is not created when the uids do not match.
Rfe12456 - Time zone roundtrip for xsd:dateTime and xsd:time
AllegroGraph previously represented xsd:times and xsd:dateTimes with Zulu (i.e. Greenwich Mean) time internally and converted data to that time zone as it was imported. On output, any xsd:time or xsd:dateTimes would be displayed in Zulu time and would lose the original time zone information. For example,
"2014-02-23T08:18:59+05:00"^^xsd:dateTime would become
"2014-02-23T03:18:59Z"^^xsd:dateTime AllegroGraph now keeps the original data's time zone information. Note that the additional computation involved with coercing time information during query means that queries involving xsd:dateTimes or xsd:times may run slightly (a few percentage points) more slowly. There is also a new query option, presentationTimeZone, that controls the display of time zones in the output of SPARQL queries. For example, adding
PREFIX franzOption_presentationTimeZone: <franz:-04:00> will cause all xsd:times and xsd:dateTimes to be printed at the -04:00 time zone. See the documentation for more information. Since the database format has been changed, the usual upgrade process involving a backup/restore cycle (see Backup and Restore) is required. Please note the following:
1: There are no changes for xsd:dates. They still support what the XML Schema describes as 'recoverable timezone'.
2: "2014-02-23T08:18:59+08:00"^^xsd:dateTime is DISTINCT from "2014-02-23T00:18:59Z"^^xsd:dateTime.
3: Joins can be made over dates that represent the same point on the same timeline even if they have different timezones. Which (equivalent) representation is returned by the join is unspecified.
See the date and time documentation for more information.
Rfe11303 - Fix partial repo creation on agraph-backup restore operations
When performing an agraph-backup restore operation, if you passed a directory name as the archive argument, the restore would fail, but not before a repository was created. Further, the repository would remain in 'restore' mode, so it could not be opened, and its existence would prevent a subsequent valid agraph-backup restore operation from succeeding. agraph-backup now first checks that the argument filename names an existing file which is not a directory before creating any repositories. Also, error messages when agraph-backup** is passed invalid archive file arguments have been improved. See Backup and Restore.
Rfe11044 - Add -p short form of --port to agraph-backup.
With agraph-backup, users may use either -p or --port to specify the port on which an AllegroGraph server is running. See Backup and Restore.
Rfe10629 - Add --supersede argument to agraph-backup
agraph-backup now accepts a --supersede option, which (if specfied) allows overwriting of existing archives (for the backup command) and directories (for the backup-all command), and databases (for the restore and restore-all commands). See Backup and Restore.
Bug22099 - Ensure UUIDs are unique during agraph-backup restores.
Previously, when restoring archive backups to an AllegroGraph server, it was trivially possible to create multiple repositories with the same uuid. Now, agraph-backup will error if a restore operation will cause a uuid conflict. See Backup and Restore.
Bug22090 - Timezone minutes in xsd types
Specifying timezone minutes is no longer optional in xsd:dateTimes and xsd:times, so the following examples are invalid:
"1978-01-03T00:00:00+05"^^xsd:dateTime INVALID
Correct form: "1978-01-03T00:00:00+05:00"^^xsd:dateTime
"00:00:00+05"^^xsd:time INVALID
Correct for: "00:00:00+05:00"^^xsd:time Previously, only xsd:dates checked for the presence of timezone minutes. See the documentation for more information.
Bug22083 - xsd:date and xsd:dateTime years
AllegroGraph now correctly parses both positive and negative dates with more than four digit years. It also signals an error if the date is too large to be encoded into a UPI. Previously, dates like these were silently truncated. See the date and time documentation for more information.
Bug22073 - ORDER BY and non-encoded decimals
When non-encoded decimals were being ordered, they were first converted to IEEE 54 double floats (which have a precision of 15-17 significant digits) which potentially caused them to be sorted incorrectly. This bug has been fixed.
Bug22072 - Validate xsd:decimals
AllegroGraph now signals an error if an xsd:decimal's exponent in scientific notation is outside of the range [-63, 63] because these values cannot be encoded into a UPI. Previously, xsd:decimals like these were silently truncated.
Bug22068 - Reasoning and constant triple patterns
With reasoning enabled, some queries involving triple patterns without variables failed with 'Bus error'. This has been corrected.
Bug22066 - Printing of NaN, INF and -INF
Serialization of xsd single and double float NaN, INF and -INF was broken (for instance, INF became "#.inEinity-double"). This bug has been fixed.
Bug22057 and Bug22054- agraph-backup backup-settings works as documented.
The agraph-backup backup-settings command was incorrectly backing up data as well as settings rather than just backing up settings. Now it just backs up settings. Also, the backup-settings command now can work by finding the settings location from the config file (so the server need not be running). See Backup and Restore where a note describes all the agraph-backup revisions.
Bug22045 - 14 hour datetime comparison logic
For two xsd:dateTimes, one with and one without a timezone, all comparisons must return false if they differ by less than 14 hours. Similar logic applies to xsd:dates and xsd:times. Previously, support for this 14 hour comparison logic was spotty. It worked for some combinations (such as comparisons in a FILTER with a literal) but didn't work for others. With this fix, comparisons for all three types follow the XML Schema specification. See the date and time documentation for more information.
Bug22044 - Fix xsd:date xsd:dateTime comparisons
AllegroGraph previously compared xsd:dates and xsd:dateTimes by converting the date to a dateTime at its first instance. Because dates are actually 24-hour intervals, this logic is suspect so AllegroGraph now signals a type error if this comparison is made (i.e., FILTERs will not longer match and BIND would leave its variable unbound). The previous behavior can still be had by using an explicit cast from date to dateTime. See the date and time documentation for more information.
Bug22041 - Comparisons to decimal zero
Due to a bug in the encoding, decimal 0 appeared to be greater than all decimals less than 1.0. The usual upgrade process involving a backup/restore cycle (see Backup and Restore) is required. All misencoded decimal 0s will be converted to the correct value. If any such corrections are made during the upgrade, then indices are resorted and background index optimization is begun. The index optimization may continue even after the upgrade process has completed. The resulting indices are then as optimized as if all the triples were just imported into the database. This means if the backup was of a fully optimized repository, then after the upgrade running level 2 (the default) index optimization is necessary to get back to that state.
Bug22039 - agload character encoding fixes
Previously, the --encoding argument to agload was ignored. This has been corrected. See Data Loading for information on agload.
Bug22034 - part->concise on encoded dates and decimals
For xsd:dates, xsd:dateTimes, xsd:times and xsd:decimals, the Lisp functions part->concise and part->terse returned the stringified internal representation (like "172839/500" for decimals, and "(
(part->concise (value->upi "2013-01-03T10:34:00-05:00" :date-time))
=> "2013-01-03T10:34:00-05:00" See XSD Date and Time datatypes.
HTTP Client
No significant changes.
SPARQL
Rfe12624 - Standardize some low-level SPARQL equality tests
The efficiency and conformance of low-level tests used to compute RDFterm-equality and SPARQL ordering has been improved. One non-backwards compatible change is that AllegroGraph previously allowed ordered comparisons between plain literals that looked like numbers and typed numeric literals (e.g., "6.21" and "6.23"^^xsd:integer). This comparison now correctly throws a SPARQL type error.
Bug22092 - Fix for SPARQL queries that use the STR() function with a constant
If a SPARQL query contained a FILTER that used the STR() function applied to a constant, then the query could signal an error at planning time. For example, a FILTER like
FILTER( STR( <http:://example.com> ) = ?var ) would not succeed. This has been corrected.
Bug22078 - SPARQL insert did not always encode typed literals
Triples added to a triple-store using SPARQL inserts that used computed values did not always have their typed literals encoded into UPIs. For example, an insert like:
insert { <ex:a> <ex:b> ?c . } where { bind( '1'^^xsd:byte as ?c ) } would store the literal in string form rather than encoding it directly. This has been corrected.
Bug22077 - Improve efficiency of certain SPARQL sub-queries
Bound variable information could be lost while processing certain SPARQL sub-query forms which would then result in less efficient query execution. This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
Bug21608 - Include ag-call in remote Lisp client.
Previously ag-call was omitted from the remote Lisp client. This has been corrected.
Prolog
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
No significant changes.
AllegroGraph 4.12
Server Changes
Rfe12576 - Improve RDF/XML parser efficiency
Improve the lower-level efficiency of AllegroGraph's RDF/XML parser so that it is between 5 and 10% faster.
Rfe12522 - Global configuration directive for query options
A new global configuration directive named QueryOption was added. It can be used multiple times. Each specification is equivalent to a query prefix option. The global configuration directive "QueryOption NAME=VALUE" is the same as the query prefix option "PREFIX franzOption_NAME:
Rfe12429 - More flexible parsing for InstanceTimeout
With this change, InstanceTimeout configuration parameter values in the configuration file can be specified as times rather than only as a number of seconds. For example, you can use 12s, 2m or 3h to specify 12, 120 or 10,800 seconds. The configuration file is described in Server Configuration.
Rfe11010 - Require the i index for freetext queries
AllegroGraph's native freetext index requires the presence of the triple-id (i) index in order to operate efficiently. A common problem was to try to use freetext queries when there was no i index. This change automatically adds an i index whenever a freetext index is created and also signals an error if a freetext query is attempted when there is no such index.
Bug22024 - Serialization of decimals
xsd:decimals with zeroes right before the decimal point were serialized with the decimal point misplaced. This bug has been fixed. The database contents were not affected.
Bug21978 - serialize-* functions in Lisp client default to UTF-8.
Use UTF-8 as the external-format when a filename is passed to one of the serialize-* functions in the Lisp client instead of the default external format, which can vary depending on the system's locale.
Bug21976 - Complex GRAPH queries could produce incorrect graph bindings
Depending on the dataset, queries with deeply nested structure that used EXISTS or NOT EXISTS FILTERS with embedded GRAPH clauses could generate incorrect bindings for the graph variable. This has been corrected.
Bug21975 - Increase AllegroGraph Load speed
Low-level efficiency enhancements improve AllegroGraph load speed by roughly 10%.
Bug21966 - Possible "UPI not in string table" problem with remote-triple-stores
It was a possible for a remote-triple-store to lose track of the strings cached on the client after a rollback operation. This could lead to a "UPI not in string table" error. This has been corrected.
Bug21956 - Improve memory usage for zero-length property path queries
A zero-length property path query connects all subjects and objects on the right and left hand sides of the path to themselves. If the store is large and the path produces many matches on one or both sides, AllegroGraph was using more memory than necessary. This has been greatly improved though some queries may experience a slight performance degradation.
Bug21954 - Problem generating soundexes for phrases in Callimachus
If a store was operating in Callimachus mode and a phrase was soundexed that had words in it with no soundex (e.g., a number) then AllegroGraph would generate garbage soundex data. This has been corrected
Bug21949 - Improve bookkeeping of already processed variables during query execution
AllegroGraph was not updating the list of bound variables when evaluating a BIND clause during query execution. This did not change the final results of a query but did mean that AllegroGraph would not always evaluate the query in the most efficient manner. This has been corrected.
Bug21937 - Zoneless xsd:dateTime roundtrip
Adding a triple with an xsd:dateTime without a timezone (e.g. "2013-07-30T23:59:59") was stored as if it had a timezone of zero. With this bug fix, zoneless dateTimes are stored and serialized without timezone. This is in agreement with the XML schema specification and similar to how xsd:dates behave.
Bug21934 - String table mutex not reinitialized after recovery
A mutex used in the string table data structure was not being reinitialized after database restart. Under certain circumstances this could result in a database hanging after restart. This problem has been corrected.
Bug21920 - Soundex should ignore punctuation
AllegroGraph's soundex function (which is used by Callimachus) was not ignoring punctuation properly. This has been corrected.
Bug21915 - Make upi->value on xsd:dates return integers
Due to an undocumented change in v4.11, upi->value on a upi encoded as :date returned a cons of the integer representing the instant and a flag indicating whether the timezone was present. With this change only the integer part is returned (like before v4.11). The flag is returned as the second value by upi->values.
Bug21914 - xsd:dates on the first day of the month
Adding a triple with an xsd:date with a positive timezone that falls on the first day of a month resulted in error "illegal date 0". This bug has been fixed.
Bug21909 - agraph-backup could fail to generate unique UUID on restore
When using the --newuuid option, it was possible for agraph-backup to create multiple triple-stores with the same UUID. This has been corrected. See Backup and Restore for information on agraph-backup.
Bug21903 - Free text indexing with chunk-at-a-time processing
Depending on the execution plan, a query that used both fti:match and chunk-at-a-time processing (see franzOption_chunkProcessingAllowed) could produce too few results. This bug has been fixed.
Bug21899 - Shared memory server fails to clean up
Under low memory conditions, the shared memory server can fail to allocate blocks of shared memory on behalf of a database instance. Under certain circumstances the shared memory server would fail to clean up from partial allocations, thereby exacerbating the low memory condition. This problem has been fixed.
Bug21873 - Link to docs in rpm install
When AllegroGraph was installed from an rpm package, the documentation links in AGWebView were broken. This bug has been fixed.
Bug21872 - agload will write output to agload.log when in --debug mode
Along with the other effects of using the --debug command line argument to agload, all output will be logged to agload.log. See Data Loading.
Bug21869 - Fix problem with registering cartesian predicate mappings
There was a problem which prevented the shortened form of cartesian predicate mappings from working. This has been corrected.
Bug21853 - Connection Pool session port connection failure
Under certain circumstances, the AllegroGraph Java client would throw an exception like the following:
com.franz.agraph.http.exception.AGHttpException:
Possible session port connection failure Setting the SessionPorts configuration parameter is discussed here in Server Installation and also in Server Configuration. But this exception could occur even with a proper SessionPorts setting. This problem has been resolved.
Bug21846 - Long running federated queries could fail
The query canceling interface introduced in AllegroGraph 4.11 could lead to query failures when using federated triple-stores. This has been corrected. Note that it is not currently possible to cancel queries running on a federated store. This will be improved upon in a future release of AllegroGraph.
Bug21816 - Fix session multiplexing file descriptor leak
When requests to sessions are multiplexed through the server's main port, the frontend didn't close the http socket used to communicate with the session process. This bug has been fixed. Note that session multiplexing can be enabled for the java client by setting the system property:
com.franz.agraph.http.useMainPortForSessions=true
Bug21784 - Safer auto-close for garbage db's in the Lisp direct client
The db's are now marked for closure when it is noticed they are no longer used and closed at a later specific time.
Bug18813 - Reasoning and subclasses of owl:TransitiveProperty
With this fix the reasoner is able to infer that a property whose type is a subclass of owl:TransitiveProperty is a transitive property. That is,
:p rdfs:subClassOf owl:TransitiveProperty .
:q rdf:type :p .
:a :q :b .
:b :q :c . entails
:a :q :c .
HTTP Client
No significant changes.
SPARQL
Rfe12537 - Improve zero length property path clause estimation
This change improves the heuristics AllegroGraph uses to estimate the number of rows that a zero length property path query can generate which improves the way AllegroGraph processes a BGP.
Rfe12520 - The SNA members and size magic properties work with RDF lists
You can now use the http://franz.com/ns/allegrograph/4.11/sna/members and http://franz.com/ns/allegrograph/4.11/sna/size magic Properties to query RDF lists stored in your triple-store. For example, if
prefix sna: <http://franz.com/ns/allegrograph/4.11/sna/>
select ?item {
<ex:list> <ex:pointsTo> ?list .
?item sna:members ?list .
} Would return "1", "2" and "3". See SPARQL Magic Properties for SNA.
Rfe12506 - Improve handling of some queries that use GRAPH clauses
SPARQL is defined such that a query like
GRAPH ?g { A } should behave as if A was evaluated with ?g bound to each graph in the dataset in turn and then the results of all of these evaluations were unioned together. While correct, this implementation can be very inefficient. AllegroGraph uses various heuristics to determine if a given GRAPH clause can be evaluated more directly and does so when it can. This change improves these heuristics so that more queries will be able to use the more efficient implementation.
Rfe12481 - Improve SPARQL DELETE efficiency
This change greatly increases the efficiency of the DELETE portion of a SPARQL DELETE... INSERT... WHERE... command. The actual increase is dependent on the data and the delete template but it can be more than 40 times faster.
Rfe12442 - Efficiency improvements for SPARQL aggregation
Optimize the code used for SPARQL aggregate queries in general and for count aggregates in particular.
Rfe12400 - Minor property-path query optimization
Improve the underlying efficiency of some property-path queries.
Rfe12398 - Clarify semantics of the identity clause reorderer
AllegroGraph has two clause reorderers which try to preserve the ordering of triple-patterns in a BGP (basic graph pattern). Previously, these were named:
:identity - try to preserve ordering but still prefer to augment the current set of bindings rather than start a new set.
:strict-identity - use the given ordering directly without changing it.
To avoid future confusion, these options are being renamed such that the first method is called :weak-identity and the second is called :identity.
The default query clause reorderer continues to be statistical so queries that do not specify a reorderer will behave exactly as they did before this change.
Rfe12392 - Optimize certain complex join combinations
The query optimizer now converts a query with an algebra like:
(:join (:join A :bgp1) :bgp2) into the equivalent:
(:join A :bgp1-and-bgp2) This BGP (basic graph pattern) merger improves AllegroGraph's ability to reorder queries for efficient execution. AllegroGraph also converts the expression:
(:join (:join :bgp1 A) :bgp2) into
(:join :bgp1-and-bgp2 A) unless the two BGPs do not overlap (in which case the transform would produce a cross-product).
Rfe12385 - Improve BGP reordering in some cases
In some cases, the SPARQL query planner now does a better job reordering the triple-patterns in a BGP (basic graph pattern).
Rfe12380 - Improve efficiency of some DISTINCT or REDUCED queries
AllegroGraph now collects fewer intermediate results for some DISTINCT or REDUCED queries.
Rfe12377 - Add intervalContainsTime magic property
Added an additional temporal magic property that can be used to determine if a time is contained within an interval. See SPARQL Magic Properties.
Rfe12372 - Make BGP reordering more intuitive
Previously, AllegroGraph would switch the execution of patterns in a BGP when the BGP contained only two patterns and these patterns had the same statistical clause estimates. E.g., in a query like:
?a ?b ?c .
?c ?d ?e . AllegroGraph would execute the second pattern first. AllegroGraph now executes the first pattern first in the case of ties which is more intuitive.
Rfe12368 - Use constraints to limit bindings for GRAPH queries
Improve AllegroGraph's use of FILTERs in limiting the graphs considered in GRAPH clauses with variable graphs. For example,
GRAPH ?g { A }
FILTER( ?g = <ex:test> ) will now be evaluated more efficiently since AllegroGraph will only consider the graph <ex:test> rather than every graph in the dataset.
Rfe12352 - Provide a sort order for SNA path and group gensyms
Social Network Analysis (SNA) Magic Properties can create node identifiers to keep track of paths and groups. These act like blank nodes and so had no sort order defined for them. This was inconvenient because it made it impossible to, for example, list all of the nodes of a path together. This change adds a sort order to these node identifiers so that they are easier to use in queries.
Rfe12192 - Optimize some wild-card property-path queries
AllegroGraph now computes property-path queries that use the * or + operators more efficiently.
Rfe12032 - Improve property-path expansion
AllegroGraph now expands more property-path queries into their equivalent BGPs (basic graph patterns). This can improve the performance of the query optimizer. For example, AllegroGraph previously expanded
?s :a/:b ?o .
into ?s :a :b0 . :b0 :b ?o .
but did not expand paths with non-simple operators like wild-cards or alternation. It now expands these as well so that a path like
?s :a/:b/(:c|:d)/:e/:f ?o becomes
?s :a _:b0 .
_:b0 :b _:b1 .
_:b1 (:c|:d) _:b2 .
_:b2 :e _:b3 .
_:b3 :f _?o .
Rfe11948 - Add PREFIX option for enabling reasoning in a SPARQL query
AllegroGraph's SPARQL 1.1 engine now supports the inferenceMode query option to control the kind of inference used by an individual query. The mode is selected using the query option PREFIX notation. For example, the following PREFIX would cause a query to be evaluated using AllegroGraph's RDFS++ reasoner:
PREFIX franzOption_inferenceMode: <franz:rdfs++> The following modes can be supplied:
rdfs++- use RDFS++ reasoning,false- use no inference,restriction- use RDFS++ and the restriction reasoner.
Rfe11625 - Improve algebraic optimizations of joins and left-joins
Improve the logic used in the algebraic transform of
(:join (:left-join A B) C) into (:left-join (:join A B) C) Which is the textual equivalent of moving the optional clause down in a query like
A
optional { B }
C so that it becomes
A
C
optional { B } which can be evaluated more efficiently unless A and C share no bindings.
Rfe11146 - Handle simple OPTIONAL clauses more efficiently
AllegroGraph now handles OPTIONAL clauses that contain only a single triple-pattern more efficiently. This means that queries with clauses that look like
?s :predicateA ?o .
OPTIONAL { ?s :predicateB ?b }
OPTIONAL { ?s :predicateC ?c } will be processed more efficiently.
Rfe11068 - Improve literal equality constraint handling
AllegroGraph now handles filter expressions involving literal equality more efficiently. For example, in a query like
select * {
?a :predicate ?b .
filter( ?b = 'literal' )
} AllegroGraph previously would scan all triples with the given predicate and apply the filter afterwards. Now AllegroGraph determines the set of possible matches for 'literal' and queries for them directly.
In addition, AllegroGraph now performs a similar operation if the filter is a literal equality on an expression using the SPARQL str function. For example:
filter( str(?variable) = 'searching' ) As with regular literal equality constraints, such an expression is converted into the equivalent tests for specific values.
Bug22030 - ORDER BY expressions with non-query variables could cause errors
If a SPARQL query used an ORDER BY expression that contained variables that were not found anywhere in the query, then an error could result. An example of this would be a query like:
select * {
?s a ?o .
} order by desc( str( ?notFoundInQuery )) This issue has been corrected.
Bug22021: Correct SPARQL TSV printing problem for encoded UPIs
When encoded UPIs were serialized from a SPARQL query using the SPARQL TSV (tab sepated values) output format, they were printing without the enclosing quotes. This has been corrected.
Bug22016 - Correct SPARQL 1.1 bug with IN FILTERs
It was possible for the query engine to produce incorrect results when using FILTER with either the SPARQL IN operator or multiple equality filters on the same variable. This has been corrected.
Bug22015 - Fix possible error when using the SPOGI cache and SPARQL 1.1
In rare circumstance it was possible for a SPARQL 1.1 query to signal an error when the SPOGI cache was enabled. This has been corrected.
Bug21986 - Using distinct aggregation with GROUP_CONCAT and a separator could fail
If a SPARQL query used the GROUP_CONCAT aggregator along with both the DISTINCT and SEPARATOR modifiers, then the solution bindings for that variable could be left empty. This has been corrected.
Bug21980 - Too many results in DISTINCT and REDUCED queries
Under extremely rare circumstances, queries involving DISTINCT or REDUCED could return too many results. This bug has been fixed.
Bug21974 - Assertion failure involving 'n-valid-partials'
When chunk-at-a-time processing was enabled (see query prefix franzOption_chunkProcessingAllowed, described here in the SPARQL Reference), the code keeping track of the number of intermediate results used to run into an assertion failure under very rare circumstances (when two consecutive intermediate results were the same). This bug has been fixed.
Bug21962 - Improve estimates for certain query patterns
AllegroGraph was mis-estimating the sizes of query patterns that resulted from the inlining of filters that used the IN expression. For example,
?s ?predicate ?object .
FILTER( ?predicate IN ( ex:a, ex:b, ex:c ) This could result in sub-optimal BGP re-ordering. This has been improved.
Bug21961 - Incorrect results with EXISTS and NOT EXISTS
Some queries involving EXISTS or NOT EXISTS filter expressions returned too few results. This bug has been fixed.
Bug21955 - Sorting on unbound variables could lead to a string dictionary error
If a query used an ORDER BY expression involving a computation and some of the variables in the expression were unbound, then the expression could signal a UPI not in string dictionary error. This has been corrected.
Bug21948 - Improve bookkeeping of bound variables during sub-query
When executing a sub-query with a limit, AllegroGraph failed to update the list of variables known to be bound. This prevented some queries from executing as efficiently as possible. This has been corrected.
Bug21947 - Improve efficiency of VALUES in some cases
AllegroGraph now does a better job of using the information in a VALUES clause to restrict alternate solutions. This can greatly increase the speed of some queries.
Bug21941 - Queries with magic properties could fail in a remote-triple-store
Queries that used magic properties could cause an error when run from a remote-triple-store. This has been corrected.
Bug21940 - Using magic properties with filters could lose solutions
If the circumstances were just right, a query that used a magic property that produced multiple bindings and that used a filter on those bindings could fail to return all solutions. This has been corrected.
Bug21933 - Externally introduced variables could fail in nested sub-queries
If a query used externally introduced variables (e.g., via the Lisp APIS's with-variables argument or via the variable bindings in an HTTP REST request), and these variables were used in FILTER clauses in a nested sub-query, then the query could fail to find any matches. This has been corrected.
Bug21921 - BGP triple filters with Chunk-at-a-Time processing
In rare cases, a filtering triple-pattern could fail to remove some solutions when using Chunk-at-a-Time query processing. This has been fixed.
Bug21902 - Fix incompatibility between magic predicates and the identity reorderer
If the identity BGP clause reorderer was being used in a query with magic predicates, then the magic predicates were being sent to the end of the BGP. This has been corrected.
Bug21901 - Minor improvements to some range query clause orderings
Improve the statistical estimates used to reorder BGP patterns that use range queries.
Bug21892 - Query engine selection failed to take OFFSET into account
AllegroGraph's decision on whether to use the single-set or the chunk-at-a-time query execution engine relied only on the LIMIT of the query rather than the LIMIT and the OFFSET. This has been corrected.
Bug21880 - Using with-variables in SPARQL describe caused an error
Using SPARQL describe with an external variable binding would lead to an error at query execution time. For example, this query would fail:
describe ?x (where ?x was bound externally). This has been corrected.
Bug21865 - Comparisons of xsd:dateTimes with fractional seconds
Comparisons (including ORDER BY) of xsd:dateTimes with fractional seconds were broken. This bug has been fixed.
Bug21862 - Some FILTERs and BGP orderings could lead to empty bindings
In rare cases, AllegroGraph could fail in its bookkeeping of which variables were bound while it executed a query. This could lead to the introduction of spurious rows with null bindings. This has been corrected.
Bug21858 - Distinct, SERVICE and sub-query could lead to planning failure
If a query used a sub-query in a SERVICE clause and asked for unique results, then AllegroGraph would signal an error during query planning. This has been corrected.
Bug21852 - Improve bound variable tracking
AllegroGraph should assume that any variable supplied via external settings (e.g., via the with-variables parameter to run-sparql) is bound. Failure to do so could cause some queries to fail. This has been corrected.
Bug21828 - VALUES could fail to correctly handle OPTIONAL bindings
Using the VALUES clause could incorrectly drop solutions whose rows contained unbound values. This has been corrected.
Bug21815 - Fix property-path memory leak
Some queries involving property-path expressions could run out of memory, because the intermediate result set was not freed as soon as it was consumed. This bug has been fixed.
Bug21803 - Exclusive ranges of encoded non-integer types
Previously, filter expressions on all non-integer types (floats, decimals, etc, excluding xsd:date and xsd:dateTime) involving exclusive ranges ('<' and '> ') behaved as if an inclusive range ('<=', '> =') had been specified. This bug has been fixed.
Rfe12591: Improve efficiency of some Basic Graph Pattern evaluations
AllegroGraph now makes better use of intermediate result sets when evaluating basic graph patterns (BGPs) in nested groups of joins and left-joins.
AGWebView
Bug21997 - Agwebview and sessions
Due to a thread safety issue, AGWebView's 'Create Session' or any subsequent action in that session could produce garbage results ranging from 'Not found' to garbled output. This bug has been fixed.
Changes to the Lisp API
Rfe12562 - upi->value round-trip
For dates and date-times a complete representation is a (<universal-time> . <timezone-information>) cons. While this would make a good candidate for the return value of upi->value, to maintain backward compatibility by default upi->value must return a single number representing the universal time.
However, this means that during the upi->value/value->upi round-trip the timezone information is lost.
This change adds a new keyword argument :complete? (defaults to nil). With :complete? t, upi->value returns the complete representation and the roundtrip works in the sense of:
(upi= (value->upi (upi->value upi :complete? t) (upi-type-code upi))
upi) Currently, for UPIs other than dates and date-times, :complete? has no effect.
Rfe12557 - run-sparql can now use a UPI or future-part for the default base
Previously the Lisp function run-sparql only allowed a string to specify the default-base. It can now take a string or a UPI or future-part.
Rfe12508 - normalizing with-variables should convert $var to ?var
The with-variables parameter in the Lisp API to run-sparql allows for variable names to be identified as strings or symbols. The name can be a plain string like "person" or a string whose first character is #\? or #\$ like "?person" or "$person". All three of these representations refer to the same query variable: ?person. Prior to this change, AllegroGraph could misinterpret variable names that used $ as the first character. This has been corrected.
Bug21953 - Support server-side loading via the Lisp client.
When using the Lisp client and connected to a remote-triple-store, passing a URI or URI-like string to one of the bulk-load functions (e.g. load-ntriples) will cause the resource to be loaded directly by the server instead of being delivered via the client.
Bug21891 - db.agraph.log:log-output captured in dumplisp
If dumplisp (a function used to save a running image) was used after opening a triple store, the stream used for logging would be captured. Upon restore of the dumped image, subsequent logging attempts (performed by internal AllegroGraph code) could result in confusing errors or file corruption. This has been fixed.
Bug21860 - Make Lisp clients find bundled liblzo2.so.2
Previously, Lisp clients resorted to using the liblzo library located in the acl directory ("sys:") or somewhere the OS dynamic library loader can find it. With this change the bundled liblzo2.so.2 is loaded.
Bug21841 - db-get-triple leaks sockets
The Lisp function db-get-triple didn't close the socket it used for communication with a remote triple store which resulted in leaking sockets in CLOSE_WAIT state. This bug has been fixed.
Bug21848 - load-turtle could use the wrong triple-store
The load-turtle command could fail to respect the value of the db parameter and instead load triples into the store currently bound to db. This has been corrected.
Bug21810 - Polygon-vertexes did not work with future-parts
If called with a future-part rather than a UPI, the polygon-vertexes function would return no results. This has been corrected.
Bug21611 - Load patches in Lisp remote client
With this fix the Lisp remote client loads patches and enables pretty printing of parts and triples just as the direct client does.
Bug21070 - Fix corruption hazard in direct Lisp client.
Prevent possible database corruption that could occur when interrupting a direct Lisp client app that is modifying the transaction log.
Bug21045 - create-freetext-index function failed when called with UPIs
The create-freetext-index function (in the Lisp API) would fail if called with UPIs (rather than, e.g., future-parts). This has been corrected.
Prolog
Rfe12222 - AllegroGraph Prolog unifies UPIs and future-parts
When AllegroGraph is loaded, the semantics of Prolog unification is extended such that:
Non-eq UPIs that are equivalent (byte-for-byte the same) will unify.
Non-eq future-parts that resolve to the same UPI will unify.
Similarly, a UPI and a future-part that resolves to that UPI will unify with each other.
This enhancement does not change the capabilities of Prolog queries in AllegroGraph, but allows simpler, less-verbose coding that looks a little more declarative and less procedural. For example, the following
(select (?manager ?employee)
(member ?employee (!acme:joe !acme:jill !acme:jim !acme:jerry))
(q ?employee !acme:hasManager ?manager)
(member ?manager (!acme:bill !acme:bob !acme:betty !acme:benjamin))) would not work as intended before this change because ?manager would be unified with a UPI, and a UPI would not unify the future-parts in the last clause. The last clause would have had to be coded as something like
(member ?manager1 (!acme:bill !acme:bob !acme:betty !acme:benjamin))
(part= ?manager manager1) Using Prolog is AllegroGraph is documented here in the Lisp Reference.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Bug21953. Fix generation of some POST method requests.
AGHTTPClient would write query parameters to the body of POST requests when the content-type was not application/x-www-form-urlencoded. This can result in http servers not finding the query parameters and processing the request incorrectly. This bug corrects the problem by writing POST request query parameters to the URL, unless the request has no body and the content type is application/x-www-form/urlencoded. AGRepositoryConnection.load() (see Javadocs) now works for importing server-side statements.
AllegroGraph 4.11
Server Changes
Rfe12309 - Improve algebraic query manipulation
AllegroGraph now notices patterns of the form
(:join (:join A (:bgp B)) (:bgp C)) and transforms them into the equivalent
(:join A (:bgp B C)) (I.e., it merges the two BGPs). This transform brings more triple-patterns together which lets AllegroGraph order the query operations better.
Rfe12305 - Relative paths in the configuration file
Relative file or directory paths in the configuration file were always interpreted relative to the directory containing the configuration file. In this rfe, a BaseDir configuration directive was added that allows another path to be specified for this purpose. See the documentation in Server Configuration for more.
Rfe12280 - Simplify geospatial type-mapping API
Geospatial subtype mappings can now be set up with the regular datatype and predicate mappings rather than needing to use the specialized functions. The old functions still work but are now unnecessary. They will be removed in some future release.
Rfe12260 - Automatically map temporal predicate data to xsd:dateTimes
Triple-stores created in this version of AllegroGraph will include the three AllegroGraph temporal predicates in the list of automatically mapped predicates. I.e., the objects in triples with these predicates:
- <http://franz.com/ns/allegrograph/3.0/temporal/endpoint>
- <http://franz.com/ns/allegrograph/3.0/temporal/startpoint>
- <http://franz.com/ns/allegrograph/3.0/temporal/time>
will be converted into encoded xsd:dateTime literals. This encoding makes range queries against them faster and makes it easier to use AllegroGraph's temporal functionality.
Rfe12259 - Optimize certain combinations of JOINs and FILTERs
The query engine now recognizes certain patterns of JOINs and FILTERs and rewrites them to be more efficient. For example, a pattern like
(:join A (:filter B f)) can often be better evaluated as
(:filter (:join A B) f)
Rfe12182 - More restrictive umask
With this change, users who aren't the owner of the file, or a member of the group of the file (that is, "other" users in chmod manpage terms) have no read, write or execute permission on files created by AllegroGraph.
On existing installations, it's recommended, but in no way enforced, to change permissions the same way with 'chmod -R o-rwx <directories>' where <directories> is the list of directories holding AllegroGraph data (LogDir, SettingsDirectory, catalog directories, etc).
Rfe12175 - Audit log
When the 'Auditing' global configuration directive is set to 'yes' (it defaults to 'no'), events that have implications on security or performance will be logged to the new system database in the new system catalog. Note that regardless of permissions only superuser can read or write the system catalog.
See Auditing for more information.
Rfe12151 - Remove support for loading N-Triples into a federated triple-store
AllegroGraph 3 introduced some techniques that optimized loading of N-Triples into a federated-triple-store. These have been superseded in AllegroGraph 4 because agload is significantly faster and easy to use.
Rfe12147 - Better manage memory especially for large solutions
AllegroGraph now manages large query result sets (with or without DISTINCT or ORDER BY) more efficiently.
Rfe12138 - Agload: reduced memory footprint and other enhancements
Reduced memory consumption of agload processes. Particularly in the case where there are many input sources.
Agload now analyzes input sources on the fly. This means you will not necessarily encounter errors about invalid input sources prior to modification of a triple-store. Use --dry-run command line argument if you wish to pre-analyze an agload job.
Rfe12125 - Improve performance of unordered queries with small limits
AllegroGraph now uses chunk-at-a-time query processing for queries with small LIMITs that do not use ORDER BY more frequently. In particular, the chunkProcessingAllowed query option can now take three values:
- yes - use chunk-at-a-time mode whenever possible,
- no - always use single-set mode, or
- possibly - choose the execution mode that makes the most sense for the current query.
The default value for chunkProcessingAllowed has been changed from no to possibly.
Rfe12111 - Improve memory checking efficiency
Improve the efficiency of AllegroGraph's memory tracking code slightly.
Rfe12089 - Improve cursor resource management
Improve AllegroGraph's low-level cursor memory management.
Rfe12063 - Allow in-memory-triple-stores to use blank nodes
The in-memory-triple-store implementation for the remote-lisp-client was missing several methods required for full blank node support. This has been corrected.
Rfe11973 - Improve usage of UPI-map
Improved the way AllegroGraph manages the use of secondary indices (UPI-maps).
Rfe11955 - :commit argument added to delete-duplicate-triples
The :commit argument was added to delete-duplicate-triples to allow for the deleting a very large number of duplicates.
Rfe11886 - Use statistics more often when reordering query clauses
The query engine now uses store statistics in more places when working to reorder the clauses in a query's basic graph patterns (BGPs). This should produce better plans more often with less need for customer query tweaking.
Rfe11585 - Some queries unnecessarily accumulated partial results
AllegroGraph was sometimes accumulating results unnecessarily. For example, the query
select (count(*) as ?total) where {?s ?p ?o} was accumulating the actual results even though only the summary was necessary.
Rfe11511 - Encode xsd:decimals
xsd:decimals used to be stored as strings. With this change, they are automatically encoded directly in the UPI which potentially brings significant savings in storage space and computation in queries. The most significant 20 digits are preserved by the encoding.
Note that this change only affects the encoding of new triples. To convert old triples and take advantage of the more efficient storage, the database contents must be reloaded from a serialized format. We recommend exporting in nquad format and then reimporting it with agload. Restoring from a backup is not sufficient.
Rfe11197 - Serialize triple IDs as resources rather than literals
Previously, a triple ID would have been serialized as a typed literal like
"13"^^<http://franz.com/ns/allegrograph/4.0/triple-id> This format prevented the use of triple IDs in the subject of a triple so we have changed the serialization to a URI like
<http://franz.com/ns/allegrograph/4.0/tripleId#13> Any triple IDs already in a triple-store will remain the same (though they will print differently). This change does mean, however, than any files that contain triple IDs in the old format will need to be converted in order to be re-loaded (note, however, that the IDs assigned to triples are not part of any serialization format which means that they cannot reliably round-trip in any case).
Rfe10292 - Handle filename limit for unix domain sockets
Linux has a limit of 107 characters for filenames of UNIX domain sockets. With this change, when bumping into that limit, a meaningful error message is printed and the workaround of setting the UnixDomainSocketDirectory or the SettingsDirectory configuration directive to a shorter path is suggested.
Bug21793 - Fix race in inter process communication
Under extremely rare circumstances, multiple threads could end up listening on the same communication channel which can lead to errors and deadlocks. This was duly detected by an assertion that appeared in the log as:
the assertion (endp (intersection
(db.agraph.spawn::receiver-for
db.agraph.spawn::r)
(db.agraph.spawn::receiver-for
db.agraph.spawn::receiver)))
failed. This bug has been fixed.
Bug21782 - Potential double close of cursors
An inadvertent double close of cursors was discovered that could potentially result in improper sharing of a data structure. This has been corrected.
Bug21779 - Fix low-level cursor resourcing bug
Under heavy load, the secondary index (UPI-map) creation function could leak a cursor leading to potential resource problems. This has been corrected.
Bug21761 - Prolog q- functor could be treated as q
If a reasoning triple-store can use secondary indices (UPI-maps) then it was possible for AllegroGraph to treat some q- functors as q functors in a given Prolog select query. This has been corrected.
Bug21745 - Fix crash related to aggregate functions
Under rare circumstances, queries involving aggregate functions may cause the backend to crash. This bug has been fixed.
Bug21744 - Some Prolog select queries could return duplicate results
If using secondary indices (i.e., UPI-maps), it was possible for Prolog select queries that returned raw UPIs to overwrite one solution with another. This problem could only occur with UPI only queries and never with queries that returned strings. The problem is now fixed.
Bug21740 - Rare hangs under high load
The message broker (called the "Hub" process) may get into a state where forwarding a message blocks, preventing all other messages from being handled. This bug has been fixed.
Bug21735 - Corrupted distinct results
Under rare circumstances, queries involving DISTINCT could produce results with the bindings assigned to the wrong variables. This bug has been fixed.
Bug21702 - XSD casting in a BIND could convert to xsd:integer incorrectly
Using an XSD cast in a BIND could loss the datatype and switch it to xsd:integer. E.g.,
BIND(xs:long(?test) AS ?testLong) would bind ?testLong to a value whose datatype was xsd:integer rather than xsd:long. This has been corrected.
Bug21693 - Avoid forking too many blank node servers in agload.
Improves agload performance under default conditions when loading ntriples or nquads files which are small and numerous.
Bug21691 - Reasoning over subproperties of symmetric properties
AllegroGraph's RDFS++ reasoner partially failed to infer triples from sub-properties of a symmetric property. This has been corrected.
Bug21680 - Some BIND forms could be evaluated incorrectly.
A BIND form at the start of a BGP could fail to get values from earlier in the query evaluation.
For example, this form
?a a ?b .
{ BIND( ?b as ?otherB
?otherB a ?c .
} could lose the binding for ?otherB. This has been corrected.
Bug21670 - FILTER and EXISTS could interact incorrectly.
Depending on the shape of the rest of the query, it was possible for a FILTER clause to be applied to the inside of an EXISTS or NOT EXISTS filter rather than being applied to the main body of the query. This has been corrected.
Bug21663 - Redo of new index creation confuses instance process
If a database with a newly-created index has been shut down uncleanly, restart of the database may crash. This has been corrected.
Bug21651 - Prolog ?? and reasoning
On a reasoning triple store, the prolog ?? construct in the predictate position used to fail with a type error. This bug has been fixed.
Bug21642 - Agload buffer overflow
Due to a buffer overflow, agload could corrupt memory and potentially crash. This has been fixed.
Bug21640 - Race condition when reaping abandoned sessions
Previously, if concurrent processes performed the operation to reap abandoned sessions, shared memory state could be corrupted, resulting in hangs or crashes. This has been fixed. (This bug was actually fixed in version 4.10.)
Bug21625 - Large result sets could produce bogus triples with INSERT and CONSTRUCT
In cases where there were more than 100,000 results, the machinery that prevents INSERT and CONSTRUCT queries from producing duplicate results could create invalid answers. This has been corrected.
Bug21596 - Serialize-rdf/xml could fail on some predicates
AllegroGraph's RDF/XML serializer would fail if a predicate ended with something other than a letter, a digit, an underscore or a dash. This conformed to an older version of the Extensible Markup Language but not to the most recent version. This has been corrected.
Bug21586 - Possible divide by zero error in index optimizer
Under some circumstances an index optimizer could attempt to perform a division by zero when logging statistics to agraph.log. This would result in a non-fatal error message being logged instead of the statistics. This has been corrected.
Bug21580 - Improve external variable binding bookkeeping
The values for externally supplied variables could be lost for some complex interactions of EXISTS (or NOT EXISTS) filters and UNIONS.
This has been corrected.
Bug21570 - Fix 'No session ports available' error message
When trying to start a dedicated session and no more free ports were available, AllegroGraph failed with an internal error (calling @UUID on NIL) instead of producing an informative error message. This bug has been fixed.
Also, the http error code was changed from 400 ("Bad request") to 503 ("Service unavailable") to indicate a temporary condition.
Bug21553 - Fix MemoryCheckWhen without MemoryReleaseThreshold
If the memoryReleaseThreshold configuration directive was not specified but memoryCheckWhen was, then query execution failed with type errors trying to compare memory footprint to NIL. With this fix, AllegroGraph refuses to start in these circumstances.
HTTP Client
Rfe12136 - Optimize front-end http server
The timeout for reading headers was lowered to 5s which makes the front-end http server make better use of its available resources and more resilent when clients with http keep-alive don't close the connection in a timely manner.
Also, a warning is logged if the number of idle http worker threads falls below 2, because it can easily cause a performance drop.
SPARQL
Rfe12296 - Support AllegroGraph direct reification in SPARQL
AllegroGraph now supports direct reification in SPARQL queries using the <http://franz.com/ns/allegrograph/4.0/tripleId> magic property. This property allows you to bind the ID of a triple to a SPARQL variable which can then be used in reification. For example, the following query first adds two triples and then adds four additional triples that describe the first two:
prefix : <eh://>
prefix franz: <http://franz.com/ns/allegrograph/4.0/>
insert data {
:gary :likes :dogs .
:gary :likes :cats .
} ;
insert {
?id :mentioned ?now .
:gary :believes ?id .
} where {
?id franz:tripleId (:gary :likes ?what) .
bind( now() as ?now )
} See the documentation for additional details on direct reification and its advantages and limitations.
Rfe12246 - Enhancements to SPARQL freetext queries
The fti:match and fti:matchExpression predicates can now optionally retrieve the object of the matching triples and select the freetext index to use. (These enhancements are currently available only for AllegroGraph's native freetext indices. Support for Solr and MongoDB will be forthcoming).
The new features rely on SPARQL magic properties to provide a syntactic shortcut for list patterns. As an example, this query retrieves the subject (into ?subject) and text (into ?text) for the freetext query that matches "nepal":
select ?subject ?text {
(?subject ?text) fti:match 'nepal' .
} This query retrieves only subjects and uses only the freetext index named comments.
select ?item {
?item fti:match ('upgrade' 'comments')
} More examples and information can be found in the documentation.
Rfe12243 - Support AllegroGraph Social Networking Analysis in SPARQL
AllegroGraph's SPARQL engine now supports a set of magic properties to enable Social Networking Analysis including centrality measures, path finding and cliques. See SPARQL Magic Properties for more details.
Rfe12231 - Improve SPARQL geospatial integration
AllegroGraph now supports several magic properties to make SPARQL / geospatial integration easier. For example, this query finds all of the points within a circle centered at 10, 2:
select ?who ?where {
(?who ?where) geo:inCircleXY ( ex:location 10 2) .
} The following magic properties are provided:
geo:inBoundingBoxXY
geo:inBoundingBox
geo:inCircle
geo:inCircleKilometers
geo:inCircleMiles (where geo is the usual Geospatial namespace abbreviation for <http://franz.com/ns/allegrograph/3.0/geospatial/>). See SPARQL Magic Properties for additional details on using these properties.
Rfe12171 - Improve variable GRAPH clause handling in some situations
SPARQL specifies that a graph clause like
GRAPH ?g { ... } must be implemented as if it iterated over each graph in the dataset and took the union of the resulting bindings. There are sometimes more efficient ways to get the same results and AllegroGraph uses these when it can. AllegroGraph was not, however, using information about any constraints on the variable ?g that were provided by other parts of the query. For example, a query like:
?s :p ?g .
GRAPH ?g { ... } might produce a single binding for ?g but AllegroGraph was still iterating over all of the named graphs of the dataset. This has been optimized so that AllegroGraph will use the additional information when it can determine that the number of bindings for ?g are smaller than the number of graphs in the dataset.
Rfe12057 - Manage disk and memory resources better for CONSTRUCT queries
When there are many results, SPARQL CONSTRUCT queries buffer their results to disk to conserve memory. The cutoff value for when to buffer to disk was unnecessarily small which meant that CONSTRUCT went to disk more often then necessary. This has been corrected.
Rfe12035 - Improve handling of some SPARQL queries with a GRAPH clause
By definition, SPARQL must handle a GRAPH clause as if the query engine evaluated the clause once for each graph in the dataset. However, it is often possible to evaluate the clause more efficiently.
This change improves the heuristics AllegroGraph uses to determine when the more efficient approach will achieve the correct results which means that some queries with a GRAPH clause will be executed more quickly.
Rfe11319 - Support SPARQL federated queries (the SERVICE clause)
AllegroGraph now supports SPARQL federated query using the SERVICE clause. The initial implementation is functional but not highly optimized.
Rfe8063 - Support AllegroGraph temporal reasoning in SPARQL
AllegroGraph's SPARQL engine now supports a set of magic properties to enable temporal reasoning. See SPARQL Magic Properties for more details.
Bug21757 - SPARQL geospatial queries could fail
Depending on the set up of a triple-store's geospatial subtype mappings, it was possible for a SPARQL query using geospatial extensions to fail to find results. This has been corrected.
Bug21742 - SPARQL parser failed on quad patterns with collections
The SPARQL parser could lose track of triple-patterns that arose from abbreviations using either nested blank nodes or collections. This has been corrected.
Bug21725 - Correct potential error when SPARQL Update touches too many rows
If a SPARQL DELETE ... INSERT ... WHERE request changes many rows, then AllegroGraph caches the results of the WHERE expression to disk to reduce memory pressure. In some cases, this could result in a error when the cached results were re-read. This only occurred for Update requests that touched more than 100,000 rows. This has been corrected.
Bug21718 - Unused variable optimization could produce incorrect results
AllegroGraph analyzes SPARQL queries that use DISTINCT in order to remove the collection of bindings for variables that are not needed to produce the final result. In some cases, it was possible for this analysis to cause the execution of a code path that added solutions incorrectly. This has been corrected.
Bug21573 - SPARQL queries could intern strings unnecessarily
SPARQL queries that used the FROM or FROM NAMED clause always tried to intern the graphs into the triple-store's string table. This was both unnecessary and could cause an error if the triple-store was read-only or a federation.
This has been corrected.
Bug21011, Bug21571 - handle nested SPARQL BGP syntactic sugar correctly
Some SPARQL triple-patterns that involved nested brackets and object lists were being parsed incorrectly. This has been corrected. Some examples of queries that work correctly now but that failed without this change include:
PREFIX : <ex:>
SELECT * {
:x :y [:z (:b [:d (:e :f),(:g :h)])]
}
PREFIX : <ex:>
SELECT * {
?psi :a :b, [:c :d], [:e :f] .
} and
PREFIX : <ex:>
SELECT * {
?psi :a :b, [:c [:d :e]] .
} Improve Pattern estimation where there is no :p index
Improved query pattern estimators for triple-stores with no predicate index (e.g., :posgi, :psogi, etc.). This will help reorder query clauses more intelligently.
AGWebView
Rfe12325 - Add audit log query interface to AGWebView
There is now an audit menu item in the admin menu. This provides access to the audit log query page which lets you filter audit log events by user, class and date.
Changes to the Lisp API
Bug21708 - Fix some low-level Lisp API geospatial functions
Some low-level Lisp API functions had not been enhanced to account for AllegroGraph's geospatial encoding. This has been corrected.
Bug21689 - Lisp client printer variable leakage
On a remote store, the run-sparql function failed with an 'Illegal sharp character' error if print-length was set to a small value. This bug and more generally the leakage of printer variables into run-sparql has been fixed.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Rfe12118 - AGRepository with connection pooling
With this change, an AGRepository can now be configured to use a connection pool via the setConnPool method, enabling Sesame apps to transparently benefit from connection pooling. See the javadoc for setConnPool for details.
Rfe10513 - Configurable blankNodesPerRequest
With this change, the Java client supports configuring the number of blank nodes automatically fetched per request to the server by the AGRepository's AGValueFactory. The default remains at 100, but it can be configured by setting the property:
com.franz.agraph.repository.blankNodesPerRequest
Applications can also configure it directly, see the javadoc for AGValueFactory#setBlankNodesPerRequest for details.
Rfe9869 - Support SPARQL/Update from Jena
With this change the Jena adapter supports an execUpdate method in AGQueryExecution for executing SPARQL Updates. Jena Tutorial example13 has been augmented to demonstrate its use.
AllegroGraph 4.10
Server Changes
Rfe12004 - Ag4 plugins for TBC 4.0.0
With this change, the AG plugins have been updated to work with TBC 4.0.0. For installation details, see Top Braid Composer Plugin.
Rfe11986 - Improve efficiency of some property-path queries
AllegroGraph now evaluates property path queries whose initial or final endpoints come from a small set of values more efficiently. For example, a query like
prefix gnO:<http://www.geonames.org/ontology#>
select ?id ?par ?parname {
bind (URI('http://sws.geonames.org/104515/') as ?id)
?id rdf:type gnO:Feature .
?id gnO:parentFeature+ ?par .
optional {
?par gnO:name ?parname
}
} will be evaluated much more quickly than before because the subject of the path query is fixed.
Rfe11937 - Agload: remove blank node cache size guessing code
In agload, blank node cache size guessing code did not work very well. It was removed and the blank node cache size defaults to 1,000,000 which works reasonably well on small and large memory machines.
This change will only be noticed when loading ntriple or nquad files with blank nodes.
Rfe11839 - Improve eager evaluation of impossible query constraints
AllegroGraph tries to evaluate FILTER constraints as early as possible. In particular, a constraint may become impossible or a tautology before the query even runs if external variable bindings are supplied. AllegroGraph was already handling tautologies eagerly but was not doing the same for impossible constraints. This has now been remedied.
Rfe11827 - Improved memory footprint of joins with unbound variables
Previously, queries involving joins had a worst case memory footprint of O(2^N) where N is the number of join variables. This is now O(1), but the time complexity is O(N). In practice, this means that the client gets a timeout instead of a memory exhaustion warning.
Rfe11233 - Simplify property path queries when possible
Property path queries that do not use the zero or more or one or more operators and that do not have complex alternation can be expanded directly into simpler forms. This lets them be executed using the regular query machinery which can often be faster than the specialized path query machinery.
Rfe9699 - Dynamically grow string table files
Prior to this change string tables files were large sparse files. The apparent size of these files frequently led to user confusion. In addition, full filesystem errors were not handled gracefully when modifying the sparse files. Now string table files start at a minimum size and grow as needed. If growth of a string table fails due to a full filesystem error, a complaint will be logged to agraph.log and the operation will be retried periodically until it succeeds.
Bug21592 - Improve cursor resource management
Some cursors were failing to close all of their file handles which could lead to errors. This has been corrected.
Bug21583 - see Bug21575 below.
Bug21582 - Complex nested unions could fail to return some bindings
Given the right data, a complex query involving multiple nested unions could misapply an optimization meant for one part of the union to a different part. This could lead to some variables in the query being unbound.
This has been corrected.
Bug21575, Bug21583 - property path optimizations and better bookkeeping
Improve the way that AllegroGraph implements some property path queries.
Correct a problem with property path queries and over-zealous resource management.
Improve the bookkeeping of unused variables when using external bindings.
Bug21570 - Fix 'No session ports available' error message
When trying to start a dedicated session and no more free ports were available, AllegroGraph failed with an internal error (calling @UUID on NIL) instead of producing an informative error message. This bug has been fixed.
Bug21550 - Complex FILTER EXISTS expressions could cause parsing errors
Complex FILTERs that used EXISTS (or NOT EXISTS) could signal errors during parsing. This has been corrected.
Bug21549 - Nested sub-queries could fail to project variables correctly
A query with nested sub-queries that used * to return all variables in some of the sub-queries could incorrectly reason about variable scope which could cause queries to fail. This has been corrected.
Bug21534 - Garbled error message on memoryCheckWhen parsing failure
When the memoryCheckWhen configuration directive had an unsyntactic value ("memoryCheckWhen junk:1"), the error message was wrong. This bug has been fixed.
Bug21506 - Correct possible error when mixing UNION and GRAPH
Under some circumstances, a query with both UNION and GRAPH clauses could signal an error during execution. This has been corrected.
Bug21505 - Unused variables from a sub-query could confuse AllegroGraph
AllegroGraph removes query variables that are not used if a query has either the DISTINCT or REDUCED modifiers. The unused variables logic was not correctly handling the case of a sub-query whose projected variables were not used elsewhere. This could lead to an error during query processing. This has been corrected.
Bug21498 - Do not signal error when all ORDER BY expressions are redundant
AllegroGraph removes ORDER BY expressions that do not depend on the bindings of a row because they cannot have any bearing on the outcome. For example, the query:
SELECT * { ?s a ?o } ORDER BY ?pineapple is equivalent to the query
SELECT * { ?s a ?o } Unfortunately, AllegroGraph was incorrectly signaling an error when all of the expressions in an ORDER BY clause were irrelevant and therefore removed.
This has been corrected.
Bug21497 - Filters could be misapplied in some situations
AllegroGraph attempts to move FILTERS closer to where they apply in order to increase query speed. If a FILTER was applicable on both sides of a UNION and also applied to single triple-patterns then it was possible for AllegroGraph to think that the FILTER only needed to be used on the left-hand side of the UNION. For example, a query like this could fail to apply the FILTER to ?c on the right-hand side:
select * {
{ ?a ex:b ?c . } union { ?a ex:c ?c . }
filter (?c*2 = 2)
} This has been corrected.
Bug21496 - Improve memory management for CONSTRUCT queries
AllegroGraph now tracks memory more accurately and manages it more efficiently for CONSTRUCT queries.
Bug21493 - Bind is too aggressive in assuming single values
AllegroGraph was over-optimizing assignment (BIND) such that assignments to the same variable name in different pieces of a UNION could fail to work correctly. For example, a query like
SELECT * {
{ BIND( :person1 as ?person ) }
UNION
{ BIND( :person2 as ?person ) }
} would return only one binding for ?person rather than two.
This has been corrected.
Bug21486 - Correct handling of some IN LINE data syntax
AllegroGraph was not correctly reading all of the supplied values when using the abbreviated simple-variable syntax. E.g., a query like
select ?person (concat(?first, ' ', ?last) as ?name) {
values ?person { ex:person1 ex:person2 ex:person10 }
?person ex:first-name ?first .
?person ex:last-name ?last .
} was processing only the first supplied value. This has been corrected.
Bug21476 - Some blank nodes could fail to print in a 32-bit Lisp client
It was possible for blank-nodes to raise an error when AllegroGraph tried to print them in N-Triples format. This has been corrected.
Bug21470 - Corrected Turtle parser relative URI normalization
AllegroGraph was incorrectly applying sections 6.2.2 and 6.2.3 of RFC3986 when it merged a base URI with a relative one. This has been corrected.
Bug21458 - Standardize AllegroGraph exit code
AllegroGraph's license check can be resolved in three ways:
Normal start up - a license is present and correct.
Free start up - no license is present so the free limit is used.
Limited start up - only partial license information is present so AllegroGraph runs in free mode.
In all of these cases, the exit code returned will be zero (0).
If the license code is invalid or if there are any other problems with server start up, then AllegroGraph will fail to start and return a non-zero exit code.
Bug21454 - URI parsing was too strict
AllegroGraph was being stricter than necessary when parsing BASE URIs such as foo:bar:/baz. This has been corrected.
Bug21379 - Reasoning triple-store confusion about unspecified graph fixed
A reasoning triple-store failed to distinguish between a query where the graph was left off (i.e., left as a wild card) and a query where the graph was explicitly set to the default-graph of the current triple-store. This could cause it to infer incorrect triples.
This has been corrected.
Bug21365 - Fix lookup of user access rights when authorizing access to repositories in the root catalog.
AllegroGraph would deny authorization requests on repositories in the root catalog under some conditions, despite specific user access right allowing such (read/write access on catalog "/", repository ""), or global rights (read/write access on catalog "", repository "*"). This has been corrected.
Bug21353 - Correctly handle aggregation on empty datasets
AllegroGraph was failing the Data Access Working Group (DAWG) "agg empty group" test because it was not returning a row for a query asking for aggregation on an empty dataset. This has been corrected.
Bug21158 - Catch errors during filter analysis
AllegroGraph was signaling an error during query planning if the eager analysis of a FILTER expression resulted in a error. The correct behavior is to treat the FILTER as an impossible constraint.
This is now handled correctly.
HTTP Client
Rfe11609 - Support text/table content-type in the HTTP protocol
AllegroGraph will now return a simple ASCII table if an HTTP request uses the content-type text/table. For example, a query might return a table like:
-----------------------------------
| person | name | location |
===================================
| mo:Bob | "Bob" | --- |
| mo:Robert | "Robert" | 23 |
-----------------------------------
Bug20622 - "Mode" was not optional for DELETE /statements/duplicates
The mode parameter, which is used to indicate which components of each triple must be equivalent to count as duplicates of each other, is now optional and defaults to "spog" for the DELETE /repositories/[name]/statements/duplicates HTTP protocol request.
SPARQL
Rfe11920 - Signal error when aggregate expressions are used in the wrong place
AllegroGraph did not signal a useful error when aggregate expressions were used incorrectly in a SPARQL query. E.g.,
SELECT * {
?a ex:foo ?value .
BIND( SUM(?value) AS ?sumValue )
} This has been corrected.
Rfe11494 - Update semantics of SPARQL property path queries
AllegroGraph's property path queries now match the updated semantics of the SPARQL 1.1 specification. In particular, wild-card operators like * and + no longer count their matches. AllegroGraph still supports the now deprecated counting operations, e.g.,
{2,4} but no longer counts the unique paths found. I.e., you can use these non-standard operators to find if there is a path but not to count the number of paths.
Note that AllegroGraph now passes all of the Data Access Working Group (DAWG) tests with the exception of
(pp37) Nested (*)* This test uses the path
((:p)*)* and AllegroGraph incorrectly returns some duplicate results.
Bug21599 - Variable GRAPH queries that used the graph could lose results
A SPARQL GRAPH query that used a variable graph and included the graph variable in the body of the query could under some circumstances lose results. For example, a query like
SELECT * {
GRAPH ?g {
?g a <FoodGroup> .
?g <covers> ?r .
}
} could possibly have returned too few results depending on the data in the store.
This has been corrected.
Bug21581 - Improve bookkeeping when using property paths
If AllegroGraph can determine that a variable in a SPARQL query is not used, then it will enable certain optimizations. For example, if a query is returning DISTINCT results and a variable is not used in any joins nor returned by the query, then its values do not matter.
This logic was not correctly rewriting property paths to account for the unused varaibles.
This has been corrected.
Bug21562 - Insert & remove-graph-uri(s) should not override WITH
If insert-graph-uri(s) or remove-graph-uri(s) were passed as query parameters to a SPARQL DELETE...INSERT...WHERE statement, then AllegroGraph was using these parameters even if the statement used WITH to specify a GRAPH. This is now corrected. SPARQL Update statements that use WITH will not be overridden via insert or remove graph URI(s).
Bug21551 - Deletes via SPARQL Update did not correctly handle encoded UPIs
SPARQL DELETE DATA and DELETE...INSERT...WHERE commands could fail to remove some triples due to AllegroGraph's internal representation of typed literals. The failure would occur only if the literal was specified directly (i.e., when the literal was a variable, AllegroGraph behaved correctly).
Bug21548 - Nested unions with fixed patterns could miss results
If a SPARQL query contained nested unions and these unions contained fixed triple-patterns (i.e., patterns with no variables or patterns where all of the variables are bound via some external binding), then some results from these unions could be lost. This has been corrected.
Bug21546 - Correct off-by-one error in some SPARQL constraints
Depending on the data and the query, some SPARQL constraint analysis could lead to a range error. This has been corrected.
Bug21515 - Improve SPARQL parser speed
Compliance changes in the SPARQL parser of AllegroGraph v4.9 reduced the speed of parsing significantly for very large queries and update requests.
This has been corrected so that parse speed is now equivalent to what it was in AllegroGraph v4.8.
Bug21514 - Fix possible error caused by nested sub-query
The data structures used for SPARQL sub-queries were sometimes being cleaned up too early which could lead to a bus error. This has been corrected.
Bug21507 - Fix problem with property path query clause estimation
AllegroGraph was incorrectly estimating the size of clauses involving property paths in SPARQL queries which could lead to poor plans. This has been corrected.
Bug21478 - Fix spurious parsing error when mixing sub-query and aggregation
AllegroGraph's SPARQL parser could become confused if a query mixed sub-query and aggregation and the sub-query used * to refer to all variables.
This has been corrected.
Bug21469 - Relative URIs were not being correctly normalized by the SPARQL parser
Section 5.2 of RFC3986 specifies how "." and ".." path components of relative URIs should be normalized when merging with a BASE URI. The original SPARQL parser implementation skipped this step in the interests of efficiency. This has now been corrected.
Bug21464 - Some combinations of nested UNION and OPTIONAL could fail
The SPARQL 1.1 engine could fail to correctly keep track of solution bindings for queries with certain patterns of UNIONs and OPTIONALs. This has been corrected. #### Bug21501 - Queries with cross-products and empty RHS could emit incorrect solutions
If a query had a cross-product and the right-hand side of the cross-product was empty, then it was possible for AllegroGraph to not discard the solutions on the left-hand side. This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
Bug21535 - Db registry memory leak in the Lisp client
If a triple store object isn't closed properly and becomes garbage, the finalizer is supposed to close it. Fixed a bug that rendered this mechanism ineffective, causing leakage of memory.
Bug21512 - With-variables not handled consistently with sub-queries
In some cases, external bindings imposed from with-variables could be lost when using sub-queries. This has been corrected.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Rfe11527 - Support sessions that use the main server port
With this change, the Java client can now be configured to have sessions use the main server port rather than a dedicated port. By default, a dedicated port will be used. Setting the system property com.franz.agraph.http.useMainPortForSessions=true will configure the client to use the main port for sessions.
Rfe8384 - Limiting the duration of a query
With this change, the Java client now supports limiting the duration of a query. When Sesame applications use the AGQuery#setMaxQueryTime method, the AG server will timeout the query appropriately during evaluation and clean up server resources, and the client library will throw a QueryInterruptedException back to the client application.
AllegroGraph 4.9
Server Changes
Rfe11922 - Add materiazer. See Materializer for details.
Rfe11879 - Improve error message on invalid license
AllegroGraph now prints information to the console when there are problems with the license settings in the configuration file.
For example, if the configuration file is missing the LicenseExpires parameter, then AllegroGraph will print:
AllegroGraph Server Edition 4.9, built on October 01, 2012 10:04:33 GMT-0700
Copyright (c) 2005-2012 Franz Inc. All Rights Reserved.
current-time: Monday, October 01, 2012 10:43:09 AM
Daemonizing...
Server started: LicenseExpires parameter missing. Running with triple-limit of 5,000,000.
Rfe11845 - Optimize blank node printing
Blank nodes now print slightly faster and more efficiently than they did before.
Rfe11827 - Reduce memory footprint a bit
Result arrays are now freed right when they become unused. Previously, garbage accumulated until a certain limit was reached. This should have a small positive effect on high water-mark memory footprint.
Rfe11784 - Release memory back to the operating system after query failure
Prior to this change, AllegroGraph would continue to claim memory used for queries even after the query would complete. AllegroGraph will now return unused memory back to the OS when a query memory failure occurs.
Rfe11762 - Improve the efficiency of queries with unbound results
Queries that use UNION or OPTIONAL can have unbound columns in the result set. This change improves the way AllegroGraph does joins when there are unbound columns in a result set.
Rfe11756 - Improve the chunk processing behavior of ORDER BY and LIMIT queries
The chunk-at-a-time processor automatically reduces the size of the chunks being processed based on the limit (it makes no sense to process a chunk of 500,000 rows if you are only interested in the first one!). In the case of ORDER BY queries, however, it makes sense to keep the larger limit since all of the results must be examined. This is now the case.
Rfe11728 - Improve handling of large DISTINCT or ORDERed results
This enhancement improves the efficiency of the on-disk merging operations so that the size of ORDERed or DISTINCT result sets is limited only by disk space and not by available RAM.
Bug21531 - Non-simple EXISTS FILTERs could cause query failure
AllegroGraph was incorrectly processing non-simple graph patterns in an EXISTS or NOT EXISTS FILTER. This meant that queries like
SELECT * {
...
FILTER( EXISTS { ... } ||
EXISTS { ... FILTER( ... ) } )
} could cause an error rather than being correctly processing. This has been corrected.
Bug21415 - Queries could return invalid solutions
It was possible for solutions with no bindings for some of their variables to escape from the right-hand side of a join. This would cause too many solutions to be returned from the query.
This has been corrected.
Bug21410 - Duplicate suppression not handling deleted triples correctly
AllegroGraph was handling deleted triples improperly when duplicate suppression was enabled. This has been corrected.
Bug21407 - An error in a SELECT expression prevented processing of subsequent ones
An error in a SELECT expression would prevent processing of the rest the expressions in the query for the current solution. For example, depending on the data, in a query like:
SELECT
?s
?str
(STRAFTER(?str,"b"@cy) AS ?abcy) # could cause a type error
(STRAFTER(?str,"ab") AS ?aab)
WHERE {
?s :str ?str
} The evaluation of STRAFTER() with a second argument of "b"@cy could signal a type-error. AllegroGraph was stopping further computation at that point which meant that the binding for ?aab could be left empty when it should not have been.
This has been corrected: now, each expression is computed separately.
Bug21404 - Config file EOL format
Previously, non-unix newlines (CR+LF, LF+CR) in the license section of the config file could prevent the server from starting up. This has been fixed and any mix of Unix, Mac and DOS style newlines is accepted.
Bug21398 - Variables in EXISTS filter could be incorrectly bound
It was possible for variables that appeared only in an EXISTS (or NOT EXISTS) filter to be incorrectly bound during query evaluation which could lead to incorrect results.
This has been corrected.
Bug21390 - N-Triples serializer does not handle embedded angle brackets
Even though valid URIs are not allowed to contain embedded angle brackets, it is still possible to add them to a triple-store. For example, the valid N-Triples file:
<ex://h> <ex://p> <ex://invalid\u003E> would add a triple whose object is ex://invalid>.
AllegroGraph was serializing such URIs without performing the necessary \u escaping which meant that the data exported could not be re-read by AllegroGraph's parsers.
This has been corrected.
Note that this was only a problem for data with invalid URIs.
Bug21377 - Updates with both DELETE and INSERT
Due to a race condition in a shared backends, updates that involved both DELETE and INSERT could produce bogus results. This bug has been fixed.
Bug21376 - Wrong results with ORDER BY
Due to a rare race, in shared backends, queries with ORDER BY could produce bogus results. This has been fixed.
Bug21367 - Expressions in ORDER BY could cause query failure
Queries with expressions in an ORDER BY could cause query failure during string look up. This bug was introduced in AllegroGraph v4.8 and is now corrected.
Bug21349 - Eager evaluation of boolean FILTERS could lead to failures
If a query used FILTERS consisting of boolean combinations of simpler filters and also used external variable bindings such that some of the FILTER clauses could be evaluated immediately, then it was possible for the FILTER evaluator to signal an error.
This has been corrected.
Note that bug21342 corrected this problem for the simpler case of negation. This bug fixes the more general case.
Bug21345 - Possible problem with BIND inside of an OPTIONAL
If a query included a BIND clause that stood by itself and there was a join, it was possible to lose the value of the binding. In particular, a query like
SELECT * { ?a ?b ?c . OPTIONAL { bind('hello' as ?x } } could leave ?x unbound. This has been corrected.
Bug21344 - Bind could fail to use external variable bindings
If a query used the BIND form and the value of the expression depended on variables set via an external binding (e.g., via the with-variables argument in the Lisp API or via HTTP parameters in the HTTP interface), then it was possible for the BIND expression to ignore those values. This has been corrected.
Bug21342 - Eager evaluation in FILTERs can lead to problems using variables
If a query uses a FILTER and also variable bindings such that an expression in the FILTER could be evaluated before actually running the query and if the expression was used in other boolean combinations (e.g., with not, and, or or), then it was possible for AllegroGraph to signal an error. For example, if the query was
select * { ?a ?b ?c . FILTER( ! bound( ?d )) } and if ?d was bound using a variable binding, then AllegroGraph would signal an error instead of correctly realizing that the FILTER could never succeed.
This has been corrected.
Bug21341 - Non-simple IF statements could cause an error
It was possible for non-simple IF statements in an expression to lose track of their variables which led to an error at query planning time.
This has been corrected.
Bug21326 - Geospatial query failure
Fixed a bug that, under rare circumstances, caused geospatial queries to fail with bus error.
Bug21325 - Lost results in sub-queries and some joins
When an internal buffer holding intermediate results was grown in a sub-query or in some joins, results could have been lost. This bug has been fixed.
Bug21272 - No results with BIND
Sometimes, as in the query below, instead of no results, a single result was returned when BIND was involved.
select * {
bind('o' as ?c)
?a ?b ?c .
?d ?e ?f .
filter( ?a < ?d && ?d < ?a )
} This bug has been fixed.
Bug21270 - Sub-queries with ORDER BY can fail to bind results
Depending on the data and what other queries have been evaluated, a query that used an ORDER BY clause in a sub-query could fail to correctly bind the result variables. For example, a query like:
select ?age {
{ select ?age {
?who <http://people.example/age> ?age .
} order by ?age
}
} could return results but leave the bindings for ?age unbound.
This has been corrected.
Bug21260 - XPath Constructor Functions fail on strings to numbers
The XPath Constructor Functions should convert typed string literals into the requested numeric type when the value of the string can be so coerced. For example:
xs:int('2'^^xsd:string) ==> 2 AllegroGraph was raising a type error in this case. This has been corrected.
Bug21256 - Parser problem with some queries that mixed aggregation and sub-query
AllegroGraph was incorrectly signaling a parser error on some queries that mixed aggregation and sub-query. For example,
select (max(?a) as ?foo) {
select distinct ?a ?b {
?a a ?c .
?c a ?b .
}
} This has been corrected.
Bug21245 - Allegrograph's turtle parser performs URI normalization incorrectly
Percent encoded URIs should not be unescaped when loading data in the Turtle format. AllegroGraph was incorrectly doing so. This has been corrected.
Bug21243 - Non-QNameable URIs cause turtle serializer error
If a triple contained a URI that could not be expressed as a QName, then the turtle serializer would signal an error that it could not split the URI rather than just emitting the entire URI and continuing. For example, a triple containing the URI <<ttp://example/> would cause this error:
Cannot split <http://example/> into XML QName. This has been corrected.
Bug21226 - agload max blank node cache size not obeyed in all cases
Agload could over-allocate memory in a certain case. This has been fixed.
Bug21215 - Improve cursor resource usage of Prolog SELECT queries
AllegroGraph previously closed some of the cursors created while running a Prolog query lazily. It now closes them when the query completes which improves overall resource utilization.
Bug20906 - Remove duplicate triples fails on a store with zero triples
An error would be raised if duplicate deletion was attempted on a store with zero triples.
This has been corrected.
HTTP Client
Rfe11755 - Add support for insert-graph-uri, remove-graph-uri, etc.
Version 6 of the Sesame HTTP Protocol includes several graph parameters that can be used to control the behavior of SPARQL Update statements. These are:
remove-graph-uri - the graph from which triples are deleted in a DELETE/INSERT/WHERE clause.
insert-graph-uri - the graph into which triples are added in a DELETE/INSERT/WHERE clause
using-graph-uri and using-named-graph-uri - these specify the dataset (similar to using-graph-uri and named-graph-uri in the HTTP protocol and to the from and from-named keyword arguments in the Lisp API's
run-sparqlcommand.
These four parameters are now supported by AllegroGraph
Rfe11743 - Add baseURI to the SPARQL HTTP Protocol
The SPARQL HTTP protocol now accepts baseURI as an HTTP query request parameter. This can be used to shorten the rest of a SPARQL query.
Bug21408 - Retrieving JSON statements over HTTP
Fixed a bug that caused the statement list to be truncated when an 'application/json' encoded response was requested via the HTTP protocol.
Bug21280 - Query failures cause spurious errors
AllegroGraph 4.8 modified the internal API used to signal query failures (e.g., due to timeout or memory limits). Unfortunately, the code in the HTTP layer of AllegroGraph was not modified to accommodate the change which led to spurious bus errors when a query failed. This has been corrected.
SPARQL
Rfe11862 - Match default dataset behavior for SPARQL Update and Query
SPARQL Update was using a default dataset that contained only triples with unnamed graphs (the default graph in AllegroGraph terminology) in the FROM portion and put triples with graphs into each of their graphs in the FROM NAMED. This was different from the default dataset for SPARQL Query which put all of the triples into the FROM portion and treated the FROM NAMED portion similarly.
AllegroGraph now uses the same default dataset behavior for both Query and Update. This dataset is as described for query above. Note that this means that an Update command like
DELETE DATA { <eh://a> <eh://b> <eh://c> } will delete the triples
{ <eh://a> <eh://b> <eh://c> } from every graph in which it appears whereas the previous behavior would have been to delete it only from the default graph.
Note that triples added during an Update command which do not specify a graph will be added to the triple-store's default graph.
Rfe11817 - SPARQL parser should signal an error on badly formed requests
Previously, the SPARQL parser would silently accept nonsense queries like:
<http://www.foo.com> <http://www.bar.com> 'banana' . because it would silently ignore tokens that came before the SPARQL prologue (i.e., before PREFIX and BASE) and convert such a query into an empty UPDATE request (i.e., a non-operation).
These queries will now signal a SPARQL parse error. Because the parser is now more strict, some queries that previously parsed may now signal an error. For example, this invalid query was previously accepted and now will not be:
# WRONG: bad prefix because it ends with a period (".")
PREFIX ex: <http://example.com> .
SELECT * { ?s ex:testsFor ?o . }
Rfe11793 - Improve handling of numeric and date literal equality filters
Improve the SPARQL 1.1 query planner's handling of numeric and date literal equality filters. In particular, AllegroGraph will now optimize queries with FILTER clauses like:
?s ex:p1 ?o .
FILTER( ?o = 3 ) which would previously have required a scan of all triples with predicate ex:p1.
Rfe11787 - Support SPARQL 1.1 UUID and STRUUID functions
Adds support for the SPARQL 1.1 UUID() and STRUUID() functions.
Rfe11779 - Support SPARQL 1.1 change to inline data syntax
SPARQL 1.1 recently switched from a syntax for inline data that used BINDINGS to one that uses VALUES. The standard also made it possible to use inline data in more places. AllegroGraph now fully supports this feature and passes all of the Data Access Working Group (DAWG) tests for it.
Note that the BINDINGS syntax is still supported in this version of AllegroGraph. It will be deprecated and removed in some future release.
Rfe11757 - Provide select-N strategy for ordered results with small limit
Typically, a SPARQL query with an ORDER BY clause must accumulate all of the results before sorting them. If, however, the query has a small LIMIT, then it can be more efficient to retain the top LIMIT results and only sort those.
Rfe11555 - Manage memory used to ensure unique results from CONSTRUCT and UPDATE
The data structures AllegroGraph uses to ensure that results of a SPARQL CONSTRUCT or UPDATE are unique have been modified so that result sets can be processed correctly regardless of size (up to the amount of available disk space).
Bug21401 - SPARQL Update must intern strings from variable bindings
If variable bindings are used in a SPARQL Update query and these bindings are part of the template being used to insert new triples into the store, then the strings from the variable values must be added to the triple-store's string table. This was not being done which could lead to a UPI not in string table error.
This has been corrected.
Bug21391 - Queries against in-memory-triple-stores could return incorrect results
The Lisp SPARQL API allows for the creation of in-memory-triple-stores from CONSTRUCT and DESCRIBE queries. These suffered from a bug such that queries that specified more than two of subject, predicate, object and graph fields could return incorrect triples. As examples, a query like:
ASK { ex:foo ex:bar ?o } would work correctly whereas a query like
ASK { ex:foo ex:bar ex:baz } could return yes when it should have returned no.
This has been corrected.
Bug21372 - Possible error when requesting sparql-xml output for Prolog select
An error could be signaled if the default query engine was sparql-1.0 and a results format of sparql-xml was requested for a Prolog select query.
This has been corrected.
Bug21371 - IF confusion with free variables
SPARQL queries with free variables could produce incorrect results when using IF. For example, the following query
SELECT ?result {
BIND( IF(!bound(?dataType), "first", "second") as ?result)
} would return second instead of first. This occurred because the query engine was not correctly analyzing the free variable ?dataType (this variable is free because the query provides not binding for it).
This has been corrected.
Bug21361 - Improve SPARQL BASE URI handling
AllegroGraph was not always correctly merging base and relative URIs. For example, a query like
BASE <http://localhost:8080/>
SELECT * { ?a ?b </> } was being incorrectly run as if it was
SELECT * { ?a ?b <http://localhost:8080//> } This and other similar issues have been corrected.
Bug21357 - SPARQL blank-node parsing was too strict
The SPARQL parser was not allowing the full range of blank node names to be used. Thus, queries like:
SELECT * {
?a ?b _:123 .
_:123 ?c _:fv-0 .
} would fail to parse correctly. This has been corrected.
Bug21354 - Update STRAFTER() and STRBEFORE() to match changes in the standard
Recent changes to the SPARQL 1.1 standard changed the output of STRAFTER() and STRBEFORE() when no match is found. Previously, they were specified to return an empty literal with language or datatype that matched the first argument. Now they should produce an empty literal with neither language nor datatype. AllegroGraph now matches the standard.
Bug21294 - Sparql XML results for ASK queries missing head clause
The SPARQL XML generated by AllegroGraph was missing the <head/> clause. Since this clause is generally empty, only systems that validate the results against an external DTD were effected by the missing clause.
This has been corrected.
Bug21277 - Improve cursor resource usage of SPARQL 1.1 queries
AllegroGraph previously closed some of the cursors created while running SPARQL 1.1 queries lazily. It now closes them when the query completes which improves overall resource utilization.
Bug21266 - Queries with only a geospatial clause could fail to gather solutions
If a SPARQL query contained only a geospatial clause, then the SPARQL 1.1 engine could fail to add the solutions it found to the answer set. This has been corrected.
Bug21249 - Incorrect return value of COPY GRAPH <X> TO GRAPH <X>
A SPARQL Update statement like
COPY DEFAULT TO DEFAULT should have no effect on the triple-store and should succeed. AllegroGraph implemented the behavior correctly but was returning failure instead of success. This has been corrected.
Bug21239 - Concurrent queries in shared backends
With the sparql-1.1 engine, queries running in the same shared backend concurrently had unexpected interactions that lead to various errors. This has been fixed.
Bug21237 - Computations in aggregate expressions could fail
If a complex expression was used inside of a SPARQL aggregation function, the value could be left unbound. A work-around was to express
(min(2 * ?x) as ?smallestDoubled) as
(min(?x) as ?smallest) (2*?x as ?smallestDoubled) This is now corrected and the work-around is no longer necessary.
Bug21233 - SPIN functions could fail when used with constants
SPARQL queries that used SPIN functions could fail if the arguments to the functions were constants not already in a triple-store's string-table. This has been corrected.
Bug21222 - Failing SPARQL UPDATE HTTP request should error
Previously SPARQL UPDATE requests that failed would return false in SPARQL/XML format. In order to be better conform with other SPARQL endpoints, AllegroGraph now returns a 412 status code (precondition failed) instead.
Bug21163 - Add new default-dataset-behavior: rdf
SPARQL leaves things up to the implementation when no explicit dataset is specified (i.e., when no FROM or FROM NAMED clauses are part of the query). AllegroGraph now provides three choices for dataset behavior:
:all- All triples will be in the FROM portion of the dataset. Triples with a non-default graph will be in that graph. Note that this means that triples with non-default graph slots will appear in the dataset twice: once in a named graph (named by their graph slot) and again in the default graph of the dataset.:default- Only triples whose graph is the default graph of the triple-store will be in the default graph of the dataset. No triples will be in the named graph portion of the dataset.:rdf- Only triples whose graph is the default graph of the triple-store will be in the default graph of the dataset. Triples with a non-default graph slots will be in the named graph portion of the dataset (each non-default graph triple will be in a graph named by the graph of that triple).
AllegroGraph's standard behavior continues to be :all. You can change this on a per-query basis using the defaultDatasetBehavior query option:
PREFIX franzOption_defaultDatasetBehavior: <franz:rdf>
AGWebView
Rfe11910 - Add HTTP handlers for materialization
Added the materializeEntailed service.
A PUT request will materialize entailed triples based on the with, without, useTypeSubproperty, and commit query parameters.
A DELETE request will remove the materialized triples.
Rfe11856 - WebView - new UI Beta for Graph
In WebView next to the Documentation menu is a link to "WebView Beta" a new experimental UI for Query and Graph. A link at the top will returns to the classic WebView.
A link at the top will open the new Graph View UI.
Also Node queries have a View menu in the results to open the node in Graph view.
Administration features of WebView are not yet included within the new UI.
Rfe11676 - Webview: in Scripts editor, use CodeMirror JavaScript
The WebView Scripts editor now supports syntax highlighting and indentation by including the CodeMirror JavaScript mode.
rfe11596 - WebView: new UI Beta for Query
In WebView next to the Documentation menu is a link to WebView Beta a new experimental User Interface (UI) for Query. The new UI provides a modern style and easier navigation. A link at the top will returns to the classic WebView.
All existing features of WebView Query are implemented:
- select repo
- query sparql
- query prolog
- free text search (pattern or expression)
- namespaces
- save and delete any of the above to user or repo areas
- recent queries
- example queries
- click node in results goes to new query screen
- download query results
- sessions (and functors)
Administration features of WebView are not yet included within the new UI.
Rfe11582 - Webview: upgrade Query Editor to CodeMirror v2.32
The query editor for SPARQL and Prolog, implemented by CodeMirror, has been upgraded from v2.24 to v2.32.
Bug21250 - Webview: remove limit from download
The download link for Query results no longer uses the limit parameter in the query UI.
Changes to the Lisp API
Rfe11454 - get-triples-list (Lisp API) returns cursor only on request
The Lisp get-triples-list function previously returned the cursor it created as a second value. Now, however, it returns a cursor if either one was passed in initially (using the cursor keyword argument) or if the new keyword argument return-cursor is true.
Bug21381 - (Lisp API) SPARQL introspection functions required engine argument
The functions get-allowed-results-formats and get-allowed-rdf-formats returned nil if no query engine was provided. This has been corrected.
Bug21378 - (Lisp API) sparql-1.1 engine does not implement destination-db
The Lisp API allows SPARQL Update commands to specify the triple-store that they modify. This feature was not included in the sparql-1.1 engine. This has been corrected.
Documentation
No significant changes.
Python Client
Rfe11911 - Add materializer calls to python client
Added materializeEntailed and deleteMaterialized calls to RepositoryConnection.
Java Client
Rfe11906 - Java support for the materializer
With this change, the Java client now supports materialization, allowing inferences to be generated and added to a repository in a transactional fashion. See the Javadoc for AGMaterializer#newInstance() to acquire a configurable materializer, and AGRepositoryConnection#materialize to run the materializer.
Rfe11771 - Upgrade embedded JavaScript implementation
Fixes several small bugs in the way uploaded JavaScript scripts are interpreted. The most visible one was that the !== operator was acting like the < operator.
Rfe11241 - Support more query/update protocol params
With this change, the Java client now sends HTTP protocol params for update, baseURI, timeout, remove-graph-uri, insert-graph-uri, using-graph-uri, using-named-graph-uri, and logQuery.
AllegroGraph 4.8
Server Changes
Rfe11711 - Improve cursor resource usage of Geospatial Prolog queries
AllegroGraph previously closed cursors created while running a Prolog geospatial query lazily. It now closes them when the query completes which improves overall resource utilization.
Rfe11653 - Rm requests can be processed out of order
The order in which file management requests were processed caused unexpected errors under heavy load. The bug is now fixed.
Rfe11649 - Manage memory for ORDER BY and DISTINCT queries
Previously AllegroGraph needed to accumulate all results in memory for queries that used DISTINCT or ORDER BY. This could greatly increase the overall memory footprint. AllegroGraph now handles large result sets in available memory.
Rfe11631 - Improve query optimizer when inference is enabled
One part of query optimization is reordering the triple-patterns in each BGP. This reordering is based on the statistics collected by the store and these statistics do not take inference into account. This meant that the plan produced could be quite bad.
This has been corrected by modifying the estimation code to take RDFS++ into account.
Rfe11416 - Provide greater control of query memory usage
Added a memoryLimit query option that controls how much memory a query can use. If a query needs to allocate more memory than this value, it will fail.
Bug21235 - Resource manager sync-files effectively ignores msync errors
If the filesystem on which a database resided filled up during a checkpoint, this fact was logged but the checkpoint process would continue running as if the error didn't happen, which would result in database corruption. This bug has been fixed. Now when a full filesystem is detected during a checkpoint, AllegroGraph stops operating in order to protect committed data.
Bug21224 - Crashes while processing queries with DISTINCT
Under rare circumstances, the internal data structures used while processing DISTINCT queries could get corrupted leading to segmentation faults. This bug has been fixed.
Bug21214 - Correct scope of BINDings introduced on the LHS of union
The binding of a variable on the left-hand side of a UNION should not influence its value on the right-hand side. For example, the following query should produce two rows with ?a bound to 1 in one of them and left unbound in the other.
select ?a {
{ bind(1 as ?a) } union { bind(2 as ?b) }
} AllegroGraph was retaining the binding on the LHS across to the RHS of the union and so was answering with two rows each of which had ?a bound to 1. This has been corrected.
Bug21187 - Stack exhaustion with shared cursor
With lots of deleted triples a query could run out of stack when creating cursors. This has been fixed.
Bug21182 - Unbound values in distinct aggregrates could lose results
If a variable binding used in a distinct aggregate was unbound, it could cause other aggregrates in the same row to fail to update properly. For example, a query like:
SELECT (count( distinct ?taste) as ?tasteCount ) (count(?who) as ?whoCount) {
?who <http://people.example/age> ?age .
OPTIONAL { ?who <http://people.example/taste> ?taste . }
} could lose data for the ?whoCount bindings if the ?taste binding from the OPTIONAL was unbound. This has been corrected.
Bug21145 - agload: message corrected for insufficient buffer size
The error message for insufficient ntriples/nquads buffer size was reporting buffer as size -1. This message has been corrected and now reports actual buffer size in order to abate confusion about the condition.
Note: This condition is usually triggered when an ntriple of greater than 4MB is encountered while using the default dispatch strategy.
Bug21136 - Free BGP patterns can lead to very large memory allocation requests
Queries with free patterns (like ?s ?p ?o) in a BGP on the right-hand side of a query could lead the query size estimator to believe that it should reserve a very large space to save the putative result set. For example, a query like
SELECT ?s WHERE {
{ ?s ?p ?o }
MINUS
{ ?other ?p ?o . ?other ?someProp ?s }
} would lead the query engine to try and allocate 1024-Petabytes. This has been corrected.
Bug21124 - Zero length path between non-matching constants
AllegroGraph was incorrectly finding a path between two different constants which led to queries like the following to return a single (empty) result when they should have returned no results at all.
select * {
ex:one ex:connectsTo{0} ex:two .
} This has been corrected. Other zero-length path queries were behaving correctly.
Bug20853 - Updated the soundex algorithm
AllegroGraph's freetext indexing word filter was using an older variant of the Soundex algorithm that did not match the latest version (cf. http://www.archives.gov/research/census/soundex.html). The algorithm has now been updated.
Freetext indices created with the soundex word-filter should be dropped and re-built.
HTTP Client
Rfe11681 - HTTP: Allow statements queries using the POST method
Add a [store]/statements/query URL that corresponds to /statements, except that it isn't limited to the GET method, and thus can be used to do complex queries without building up overlong URLs.
Rfe11680 - HTTP: Add /unique/[column] service
Add a [store]/unique/[column] URL that allows clients to fetch the set of unique parts that occur in a column in the database, optionally restricting other columns.
SPARQL
Rfe11682 - Improve performance of SPARQL OPTIONAL queries
If a query has the form A OPTIONAL B (where A and B share variables so that it is a join) and the A part has no rows, then it is not necessary to evaluate the B part at all.
The SPARQL 1.1 query engine now recognizes this situation and skips the evaluation of the right-hand side.
Rfe11591 - Modified the Lisp run-sparql API to better handle query failure (e.g., for when a query runs out of time or other system resources).
Previously run-sparql would return (as multiple values):
- the results of the query (or nil if the results were streamed)
- the verb of the query (e.g., :select or :construct)
- the list of variables
Now run-sparql returns four values. In normal execution, the first three values are the same and the fourth value will be nil. When a query fails, the results will be:
- a keyword indicating why the query failed (e.g., :timeout)
- the verb
- the variables
- a property list of additional information (the properties will vary depending on the failure).
Code that requests a cursor should be prepared to handle a condition of type db.agraph.query.base:cancel-query-condition.
Rfe11581 - handle repeated Prologues in SPARQL UPDATE queries
SPARQL 1.1 UPDATE allows the Prologue (BASE and PREFIX definitions) to be repeated between each UPDATE command. Previously, AllegroGraph only allowed BASE and PREFIX to be used at the beginning of a SPARQL statement. This is now corrected.
Rfe11559 - Add logQuery parameter to the HTTP SPARQL protocol
SPARQL queries can now use the logQuery parameter in their HTTP requests. The parameter can be set to one of:
- false - no logging is reported
- true - log information is sent back in the body of the response (using text/plain).
- log - log information is added to the AllegroGraph log file.
The log output can be cryptic; please contact Franz Inc. for support in interpreting the messages.
Bug21220: SPARQL Update LOAD should require only write permission (not eval)
A user would need to have evaluate permission on a triple-store in order to use SPARQL Update's LOAD command to load data from a file. This has been changed so that having write permission is sufficient.
Bug21203 - SPARQL Update insert data fails to treat blank nodes correctly
The INSERT DATA and MODIFY SPARQL Update commands failed to use the small blank node identifier in the INSERT template. This caused multiple blank nodes to be created when there should have been only one.
Bug21192 - Upi not in string table error when mixing BIND, SPARQL Update and FTI
SPARQL Update queries that produced new strings (e.g., using BIND) on a triple-store with free-text indexing turned would lead to UPI not in string table errors. Note that in this case a triple would have been added to the store but its strings would not be added to the string table which could cause subsequent queries to fail until the errant triple was removed.
This has been corrected.
Bug21168 - Support SPARQL 1.1 COUNT(DISTINCT *)
The SPARQL 1.1 query engine was not correctly handling the COUNT(DISTINCT *) form. This has been corrected.
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Documentation
No significant changes.
Python Client
Rfe11697 - Added RepositoryConnection.prepareUpdate
Added RepositoryConnection.prepareUpdate for SPARQL 1.1 Update requests. The call to evaluate on the returned UpdateQuery object returns True or False.
Java Client
Rfe11748 - Upgrade to sesame 2.6.8
With this change, the Java client has been upgraded to use openrdf sesame 2.6.8 jars.
Rfe11243 - Support X.509 authentication
With this change, the Java client now supports X.509 server and client authentication. For set up details, see the javadoc for AGServer.
Bug21225 - With no requestEntity, post using setRequestBody
Previously, AGHTTPClient post method used setQueryString to pass params when there was no requestEntity provided. This burdened the server when a param became very large. Now params are passed using setRequestBody when no requestEntity has been provided.
AllegroGraph 4.7
Server Changes
Rfe11587 - Change InstanceTimeout defaults
The configure-agraph script now supports instance timeouts in interactive mode and via the --instance-timeout command line option. It defaults to one week.
Also, the default value the InstanceTimeout configuration directive (effective when there is no global or catalog level InstanceTimeout) has been changed from 0 (= disabled) to one hour.
Rfe11536 - Transitive inference queries could be slow
The RDFS++ reasoning for transitive queries could be slower than necessary for some datasets. The code has been reimplemented to be significantly faster.
Rfe11285 - Improve query constraint analysis
AllegroGraph now does a better job of applying filter constraints throughout a query. For example, previous versions of AllegroGraph would not have transformed either of the two BGPs (on either side of UNION) into range queries (which are more efficient).
SELECT * {
?a ex:d ?c .
{ ?a ex:b ?c . } UNION { ?a ex:c ?c . }
FILTER (?c > 2)
} Now it will.
Bug21114 - Instances lingering too long
Instances with non-zero InstanceTimeout only timed out when there was activity involving the service daemon. This has been corrected. Instances with an InstanceTimeout of zero (the default) are not affected.
Bug21094 - Queries against federations could signal UPI-not-in-string-table errors
It was possible for queries against federated triple-stores to lead to UPI-not-in-string-table errors. This has been corrected.
Bug21089 - agload division by 0
Progress reporting failed with division by 0 if the operation completed within 1ms.
Bug21077 - Expressions can be analyzed too eagerly
It was possible for constant expressions to be analyzed to their final values too quickly which could lead to an error if the expressions were used in a BIND. For example, this query would fail:
select * { bind(str('hello') as ?x } rather than producing a single binding of ?x to the string hello. This has been corrected.
Bug21069 - A warm-standby client now exits non-zero if error occurs during commit replay
Replication secondary (warm-standby client) now exits non-zero if error occurs during commit replay. This is to ensure transactional behavior in the replica.
Bug21059 - Improve efficiency of all predicates cache when RDFS++ is enabled
The query engine keeps track of the predicates in the triple-store to optimize queries which contain clauses where the predicate position is a variable. If RDFS++ was enabled and the cache was not already fresh, then AllegroGraph would try to find all of the store's predicates with reasoning enabled. Since new predicates cannot be inferred, this could be hugely inefficient. This has been corrected.
Bug21057 - Fix log rotation causing crashes
When the SIGHUP handler closes a log stream, attempts to log could fail with a 'stream closed' error potentially crashing the server. This bug has been fixed.
Bug21040 - Turtle data import should use UTF-8 encoding
The Turtle specification mandates that Turtle files are encoded using UTF-8. AllegroGraph was not enforcing this in all cases which could mangle Unicode data on import. This has been corrected.
Bug21022 - Group expressions with unbound elements could be dropped
AllegroGraph could lose rows in a aggregation query if the GROUP BY expression resulted in unbound columns. This has been corrected.
Bug21008 - Correct XSD datatype for certain division operations
When performing division of two xsd:integers or two xsd:decimals, AllegroGraph was emitting the result as an xsd:float rather than an xsd:decimal. This has been corrected.
Bug21007 - Error creating a session with non-default graph for inferred triples
The syntax for starting a reasoning session that uses a different graph for its inferred triples is
<store>[rdfs++#<graph>] AllegroGraph was failing to parse this specification correctly and signaled an error instead. This has now been corrected.
HTTP Client
Rfe11601 - HTTP and WebView: MongoDB interface config
Added resources to HTTP protocol for MongoDB parameters:
GET /repositories/[name]/mongoParameters
POST /repositories/[name]/mongoParameters See Mongo Interface, The HTTP Protocol, GET mongoParameters, and POST mongoParameters.
SPARQL
Rfe11558 - Add query logging support
To add in query debugging and tuning, SPARQL queries using the 1.1 query engine can now be logged to the AllegroGraph log file. To log a query, use the prefix option notation by pre-pending this prefix to your query:
prefix franzOption_logQuery: <franz:yes> The log output will go directly to the AllegroGraph log. In this release, the output is relatively cryptic. Please consult with Franz ([email protected]) to interpret the data.
Rfe11492 - Add support for SPARQL CSV and TSV
Added CSV (comma separated values) and TSV (tab separated values) as native SPARQL output formats. Also corrected problems with using CSV output on result sets with unbound values.
Rfe11486, Rfe11538 - improve SPARQL 1.1 engine geospatial integration
The SPARQL 1.1 query engine did not implement the POINT extension for SPARQL FILTER functions so that queries like:
SELECT * {
...
} ORDER BY geo:haversine-km(?loc,POINT(105.842328, 21.0184040)) would fail. This has been corrected. Additionally, the geospatial query analyzer has been improved so that more computations are carried out at query analysis time rather than at query execution time. This can speed up some geospatial queries tremendously.
Bug21117 - Some SPARQL 1.1 path queries could parse incorrectly
The SPARQL 1.1 parser could combine some combinations of alternation and sequences incorrectly. For example, this query
SELECT ?xxx ?yyy {
?xxx (dc:aaa|dc:bbb/dc:ccc) ?yyy .
} would have been parsed as if it read (dc:aaa|dc:bbb|dc:ccc). This has been corrected.
Bug21093 - SPARQL UPDATEs could fail to save newly created strings
If run with chunkProcessingAllowed, a SPARQL UPDATE query that generated new strings could fail to save some of these new strings in the underlying triple-store. This has been corrected.
Bug21082 - Optional cross-product with no bindings could fail
If the final step of a query was a LEFT-JOIN and that LEFT-JOIN was also a cross-product (i.e., the right hand side shared no variables with the left hand side) and the right hand side produced no bindings of its own (e.g., it was all constant or its variables could be elided because they were not used anywhere else in the query), then it was possible for AllegroGraph to incorrectly remove answers. This has been corrected.
Bug21078, bug21083 - Handle sub-query SELECT expresions and filters correctly
SELECT expressions in sub-queries were not being handled correctly by AllegroGraph. This has been corrected so that queries like
SELECT ?a (?a * 2 as ?b) ?c ?d {
BIND(1 as ?a)
{ SELECT ?c (2* ?c as ?d) {
BIND(20 as ?c)
} }
} now perform properly.
It was also possible for AllegroGraph to incorrectly optimize FILTER expressions in sub-queries that returned all variables (*). This too has been corrected.
Bug21072 - unused variable check could remove variables in EXISTS filters
The SPARQL 1.1 query planner removes variables that are not used in the query from the set of bindings maintained during query execution. Unfortunately, the check to see whether or not a variable was used could fail to properly descend into EXISTS (and NOT EXISTS) filters. For example, the following query would have failed:
PREFIX : <http://example/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT DISTINCT (?s1 AS ?subset) (?s2 AS ?superset)
WHERE
{
?s1 rdf:type :Set .
?s2 rdf:type :Set .
?s1 :member ?x .
FILTER ( ?s1 = ?s2 || NOT EXISTS { ?s2 :member ?x . } )
} This has been corrected.
Bug21064 - Unneeded graph variables with DISTINCT query could error
If the variable in a GRAPH clause was not needed by the query (because, e.g., the query asks for DISTINCT results and the graph variable is not used anywhere else in the query), then the query planner will remove the variable from the plan. Unfortunately, the query executor could still try to assign a value to the graph variable which would lead to an error at query execution time. This has been corrected.
Bug21046 - Multiple MINUS clauses could lead to query execution error
A SPARQL query with multiple MINUS clauses could cause the SPARQL 1.1 query engine to incorrectly determine the variables on which to join the clauses. For example, this query would fail:
SELECT (?s1 AS ?subset) (?s2 AS ?superset) {
# All pairs of sets
?s2 rdf:type :Set .
?s1 rdf:type :Set .
MINUS {
?s1 rdf:type :Set .
?s2 rdf:type :Set .
# Assumes ?s1 has at least one member
?s1 :member ?x .
# If we want to exclude A as a subset of A.
# This is not perfect as "?s1 = ?s2" is not a
# contents based comparison.
FILTER ( ?s1 = ?s2 || NOT EXISTS { ?s2 :member ?x . } )
}
MINUS {
# If we don't want the empty set being a subset of itself.
?s1 rdf:type :Set .
?s2 rdf:type :Set .
# Choose the pair (empty set, empty set)
FILTER ( NOT EXISTS { ?s1 :member ?y . } )
FILTER ( NOT EXISTS { ?s2 :member ?y . } )
}
} This has been corrected.
Bug21037 - Construct queries with Turtle results format can cause string lookup error
If a SPARQL CONSTRUCT query used expressions to create strings that were not already in the underlying triple-store and if the RDF format requested with Turtle, then a UPI not in string table error could be generated. This has been corrected.
Note that the other RDF formats do not exhibit this behavior.
Bug21031 - Remove unstable optimization
The SPARQL 1.1 query engine attempted to reduce the amount of work needed to evaluate certain joins. Unfortunately, it was possible to create queries that exposed a conflict between this optimization and other optimizations which would lead to an error at query evaluation time. This optimization has been temporarily removed. It will be added again once the conflicts it exposed have been resolved.
Bug21023 - Correct datatype of the seconds() function
The SPARQL 1.1 seconds() function should return a value of type xsd:decimal but AllegroGraph was returning xsd:float. This has been corrected.
The date and time functions (like month() and seconds()) were also not behaving correctly on literals of type xsd:time. This has been corrected.
Bug21019 - Filter EXISTS with fixed triple patterns could fail
If a SPARQL query used FILTER (NOT) EXISTS and the EXISTS clause contained fixed triple-patterns, then AllegroGraph could signal an error. For example, this query would fail:
SELECT * {
?a a ex:Fruit .
FILTER EXISTS { ex:apple a ex:Plant . }
} This has been corrected.
Bug21016 - UPDATE with blank nodes could lead to error
A SPARQL UPDATE query that included blank nodes in the INSERT template could lead to an error. This has been corrected.
Bug20994 - Correct (in)equality test between plain and typed literals
Some comparisons between plain literals and typed literals could return false rather than signaling a SPARQL type error. This meant that a FILTER that tested for (in)equality could succeed where it should have failed. This has been corrected.
Bug20840 - Improve comparisons between xsd:dates and xsd:dateTimes
The SPARQL operator mapping table is quite strict in the sort of comparisons that are allowed. AllegroGraph was allowing inequality comparisons between xsd:dates and xsd:dateTimes even though these datatypes are not strictly comparable. These sort of comparisons will now signal a type error leading FILTER conditions to fail. I.e., a query like
... FILTER ( ?aDate > ?aDateTime ) ... will now fail (correctly) even though it may have resulted in results previously.
Bug20736 - Federated-triple-stores can fail to find equivalent literals
In SPARQL filters, typed-literal equality is based on the value of the literal, not its lexical form. I.e., these literals are all the same:
- '4'^^xsd:byte
- '4'^^xsd:long
- '4'^^xsd:unsignedByte
To reduce the amount of work during query execution, the SPARQL engine checks to see whether any triples exist with literals of these (encoded) types. This check was being handled incorrectly for federated triple-stores which could lead to answers being left out of federated result sets. This has now been corrected.
Bug20770, Bug21020 - improve SPARQL 1.1 aggregation support
SPARQL 1.1. allows assignment in a GROUP BY clause. For example:
... GROUP BY (str(?name) as ?theName) ... AllegroGraph did not allow for this syntax and required the work-around of:
... BIND(str(?name) as ?theName) ... } GROUP BY ?theName ... This omission has been filled.
Secondly, AllegroGraph did not allow aggregate expressions in the HAVING clause or SELECT expression involving more than one aggregate. For example, these queries would fail:
SELECT (avg(?value) as ?avg) ((min(?value) + max(?value)) / 2 as ?half {
...
} and
SELECT * {
...
}
group by ?value
having(sum(?value) > 20) Both of these omissions have also been corrected.
Bug20165 - Unnecessary variable in a BIND clause could cause error
If the variable in a BIND clause was not needed by the query (because, e.g., the query asks for DISTINCT results and the variable is not used anywhere else), then the query planner will remove the variable from the plan. Unfortunately, the query executor could still try to assign a value to the variable which would lead to an error at query execution time. This has been corrected.
AGWebView
Rfe11601 - MongoDB interface config
Added support in WebView for setting MongoDB parameters, in the Repository Overview page. See Mongo Interface.
Rfe11557 - WebView: serve doc files
AllegroGraph documentation, a collection of HTML files installed with the server, is now served by WebView, and the Documentation menu in WebView links to these local files in addition to those on franz.com.
Rfe11474 - Execute query without changing text
In the Query editor, a query may now be executed without editing the text.
Rfe11398 - Update query editor to CodeMirror v2.24
The query editor for SPARQL and Prolog, implemented by CodeMirror, has been upgraded from v2.15 to v2.24, including javascript and css.
Bug21085 - Webview: Recent queries should not add items redundantly
WebView Recent Queries now keeps the most recently used queries. If a previous query (including the same planner and limit) is repeated, that query is moved to the top of Recent Queries, with the number remaining the same. Limit and planner are displayed on the Recent Queries page.
Bug21062 - Recent queries in multiple browser tabs
Using WebView in multiple tabs of a browser caused a javascript error leading to a blank screen below the menu. This has been fixed by consistently using localStorage.
Note for using multiple browser tabs/windows: the last tab to execute query will overwrite localStorage of "recent queries" that will be seen if a new browser tab is opened.
Bug21050 - WebView: CodeMirror Query editor steals arrow keys
When the cursor is in Query Editor, Alt-LeftArrow and Alt-RightArrow will now perform the default browser behavior instead of navigating within the editor page.
Bug21036 - Editor empty after 40 recent queries
WebView holds up to 40 recent queries, but incorrectly restored the text when more than 40 were made. This is fixed.
Note that logout and login will clear the list of recent queries.
Bug20998 - Webview: failures in IE7
Many links to navigate between WebView pages did not work in Internet Explorer 7 (IE7) due to usage of browser history. In the Query editor page, limit and planner options did not take effect in IE7. These problems have been fixed.
Bug20982 - Query limit and planner selections ignored
In the Query editor screen, the limit and planner selections were ignored. Now, they are both used for Execute and Download. Also, they will remain set after Execute, and following a link from the Recent Queries page will remember these settings for that query (but they are not stored for a Saved Query).
Changes to the Lisp API
Bug21042, Bug21043 - standardize remote-triple-store SPARQL cursors
The cursors returned by run-sparql for remote-triple-stores behaved differently from ones returned for other triple-store classes. In particular, SELECT queries could not request a results-format of :cursor and the rows returned by cursors were lists of bindings rather than arrays of bindings.
The cursors for remote-triple-stores are now standardized to behave in the same way as cursors in the rest of AllegroGraph. This item is repeated above under AllegroGraph Server: General Changes, where it is also pointed out this applies to Lisp clients only.
Bug20067 - Make session ping threads more resilient
Remote clients ping their sessions to keep them from timing out on the server. Intermittent network errors could cause the ping thread to fail. This fix makes pinging attempts continue until the session timeout passes without successful pings.
Documentation
Bug21080 - Doc layout bad in IE intranet-mode
AllegroGraph docs now use html5 doctype and style. For Internet Explorer (IE), the highest mode available will be used because IE7 layout is not optimal.
Python Client
Rfe11244: Python Support for client X.509 certificate authentication
To support X.509 certificate authentication with SSL, AllegroGraphServer objects now have cainfo and sslcert keyword arguments for indicating the client security information. verifyhost and verifypeer can also be used.
These arguments correspond to the pycurl.CAINFO, pycurl.SSLCERT, pycurl.SSLVERIFYHOST, and pycurl.SSLVERIFYPEER settings on the Curl object if you want to read about them in the pycurl documentation for more information.
user and password keyword arguments can be omitted when using certificate authentication.
See src2/franz/openrdf/tests/tests.py's test_ssl for example use.
Java Client
Rfe11523 - Upgrade to Sesame 2.6.5 jars
With this change, the Java client has been updated to use the Sesame 2.6.5 jars.
Rfe11358 - Upgrade Clojure to 1.4.0
The Clojure client API now uses Clojure 1.4.0 and Leiningen 1.7.1.
Bug21099: java pool holds sockets open
Problem: the AGConnPool used one MultiThreadedHttpConnectionManager per AGRepositoryConnection, but only closed it with the pool rather than with the repo. This used multiple connections to the main AGraph port and left http socket connections open for them and a session port after the AGRepositoryConnection was closed. Session ports are not recycled by the AGraph server if SessionPorts is not specified.
Fix: use a SimpleHttpConnectionManager for each AGRepositoryConnection and close both at the same time. This minimizes socket connections and avoids the secondary pool within MultiThreadedHttpConnectionManager.
Bug21092 - Clojure: errors in tutorial
Enhanced repo-federation to work without rcon-args as was used in the tutorial. This fixes tutorial example16.
Enhanced value-factory to work with either Repository or RepositoryConnection. This reduces reflection warnings in the tutorial.
AllegroGraph 4.6
Server Changes
Rfe11449 - Timeout for database instances
Support has been added for making database instances linger for some amount of time after their last close. This allows for reducing the likelihood that a subsequent access to an otherwise idle database will require a long startup time. The time to linger is determined by the global and per catalog InstanceTimeout directive whose default is 0s.
InstanceTimeout is from the instant the last session or backend that has the store open closes it.
New function ensure-not-lingering of the same argument as open-triple-store.
close-triple-store has a new keyword arg :ensure-not-lingering.
Rfe11445 - Queries with large intermediate result sets could generate segfaults
Queries with very large intermediate result sets could generate segmentation faults. This has been corrected.
Rfe11440 - Remove trailing slash from server URL
With this change, AGServer now removes any trailing slash from the specified server URL. Previously, an errant trailing slash on the server URL resulted in an obscure error message about "No suitable response format available" that was difficult to diagnose.
Rfe11418 - improve efficiency of AllegroGraph's turtle parser
Improved the memory usage and speed of AllegroGraph's turtle parser making it more than twice as fast as previous versions.
Rfe11404 - Optimized opening stores
By reducing inter-process communication latency, opening triple stores has been made significantly faster.
Rfe11177 - Agload --relax-syntax flag
agload now uses --relax-syntax flag instead of --relax-for-btc flag. This flag now allows for underscores and dashes to be used in anonymous node strings for N-Triples and N-Quads. agload is documented in Data Loading.
Rfe11126 - SSL client certificate and CRL support
New https-verification keyword argument to open-triple-store and create-triple-store.
Rfe8881 - Optionally suppress duplicates at commit time
AllegroGraph now has a switch duplicate-suppression-strategy, which will prevent new duplicate triples (both duplicates already in the store, and duplicates within the same commit) from being committed to the store.
Bug20997 - Solve latency issue while starting replication.
Fixed a bug where transaction logs (tlogs) were repeatedly (and unnecessarily) examined when performing replication. Now each tlog is examined once. This results in the time required to be O(n) instead of O(n*n) (that is, time is linear with the number of tlogs rather than increasing by the square of the number of tlogs).
Bug20984: Turtle serializer emitted invalid blank node identifiers
The rdf-n3 (turtle) serializer was emitting blank node identifiers without prefixing them with ":_". This has been corrected.
Bug20981 - Exhausted cursor recycling happens too early
Exhausted cursors were being recycled too early which could sometimes result in program crashes. This has been fixed.
Bug20968 - Improve string dictionary cache performance
When heavily used, AllegroGraph's string dictionary cache could become bogged down in bookkeeping resulting in very poor performance. This change prevents this excessive bookkeeping and provides the expected near-constant cache performance.
Bug20943 - Free text index bug under replication
When a new free-text index was created while replication was enabled, it caused an error in the standby process. This is now fixed.
Bug20913: Xsd date and time coercion functions could signal error
If given invalid input, the type coercion functions for dates, times and dateTimes could signal an error other than a SPARQL type error. This would lead to a query error rather than a filter failure. For example, this query:
prefix xsd: <http://www.w3.org/2001/XMLSchema#>
select ?ds ?d {
bind( '2012-12-04 is my favorite day' as ?ds )
bind( xsd:date(?ds) as ?d )
} would signal an error rather than returning a single result for which the binding for ?d is empty.
This has been corrected.
Bug20910 - Queries with variable predicates could lose results
If AllegroGraph finds a query pattern with variable predicates and distinct results (and certain other conditions that are data dependent are also true), then it will optimize the pattern by converting it into an iteration over all of the distinct predicates in the store. A typical query of this form is
select distinct ?p {
?s a ex:Animal .
?s ?p ?o
} Because computing all of the distinct predicates is expensive, AllegroGraph caches this information. Previously, AllegroGraph could fail to update the cache correctly when new triples were added. This could lead to queries returning too few results.
This has been corrected.
Bug20896 - Hang when max chunks for an index reached
As part of their operation, the automatic index optimization processes will sometimes merge some number of input chunks into a single output chunk in order to keep the total chunk count below a certain threshold. However, under some circumstances, some indices would never undergo the optimization process so this chunk reduction would not occur. Eventually the chunk limit would be reached and the database would hang. This problem has been corrected.
Bug20887 - AllegroGraph XML parsers should not follow external references
AllegroGraph's RDF/XML and TriX parsers could try to follow external references in input files and fail. This has been corrected.
Bug20877 - xs:date could fail on some short plain literals
The xs:date constructor function could fail to correctly create a date if the argument passed to it was a small plain literal. E.g., constructs like
BIND (xsd:date( concat(str(?date), '-01-01')) as ?begin_date) could pass a string whose length was less than 10 to the xs:date function and this could lead to the failure.
This has been corrected.
Bug20833 - openFederated should handle remote stores
AllegroGraphServer.openFederated now works on remote stores. Previously it always attempted to open supplied arguments as local stores.
The openFederated API still has some limitations. To open a federated session with all federation minilanguage features such as reasoning and graph filtering, pass a properly formatted spec string to AllegroGraphServer.openSession directly.
Bug20822 - Cascading error can occur if database create/open fails
Previously, if there was an error during database open/create, another error could occur during the cleanup process, obscuring the actual problem. This has been fixed.
Bug20819 - problem with some combinations of multiple OPTIONAL clauses
Some query patterns with OPTIONALs whose left-hand side was empty could lead to incorrect results. An example of a query that could have failed was:
select distinct ?value ?type ?name {
OPTIONAL {
?value <http://example.com#isa> ?type .
}
OPTIONAL {
?value <http://example.com#name> ?name .
}
FILTER( REGEX(STR(?name),'Sa') && BOUND(?type) )
} This has been corrected.
Bug20814 - Make session urls unique
In order to avoid accidental sharing of sessions, include a unique session id in the session-url. Behind the scenes, what previously looked like http://localhost:55555 is now something like: http://localhost:55555/sessions/7e8df8cd-26b8-26e4-4e83-0015588336ea.
Bug20806 - fix loading of initfile and sitescripts
Initfiles and sitescripts longer than 4096 characters were presented with 4096 character long blocks in reversed order. This bug is now fixed.
Bug20799 - turtle-parser was case sensitive when parsing numbers
AllegroGraph's turtle parser was failing to parse floating point numbers that used an upper-case E before the exponent. This is now corrected.
Bug20793 - RDFS++ queries with free predicate could infer invalid triples
If a query involving RDFS++ reasoning used an unbound predicate, the reasoner could use inverse property reasoning to infer invalid triples. This has been corrected.
Bug20757 - Problems when starting agraph while another agraph is running
Previously if you tried to start AllegroGraph when it was already running, the second start would fail but it would cause the already-running AllegroGraph to stop operating correctly. This has been fixed.
It has always been acceptable to run two or more AllegroGraph servers simultaneously as long as they, at the very least, used different settings for the following parameters in their configuration files:
Port
SettingsDirectory
GlobalShmServerDirectory (defaults to the value of SettingsDirectory)
The operating system already enforced a unique Port setting. This change adds enforcement of unique SettingsDirectory and GlobalShmServerDirectory settings. Therefore some (dangerous) side-by-side AllegroGraph configurations that used to work may no longer start up. An instructive error message will be displayed in this case.
Bug20628 - Changes to warmstandby API
PUT /repositories/[name]/warmstandby
If one attempts to start a replication task that already exists, an http 409 Conflict response will be returned.
Bug20533 - Agload --rapper flag documentation
The --rapper flag is now specified in the group "Other options". Previously it was not given a group in the command line help. agload is documented in Data Loading.
Bug20508 - Remote triple store can't use the RDFS++ reasoner
Added support for apply-rdfs++-reasoner on a remote client triple-store.
HTTP Client
Rfe10812 - Improve http tracing
HttpTrace is now a global configuration directive of the same meaning as the --http-trace command line option. The global directive HttpTraceOptions and the command line option --http-trace-options were added, the value of which is a comma separated list of options. Options starting with the character + turn on the corresponding log category. Those starting with - turn them off. Use the max-message-size=<N> option to truncate overly long messages. The default is +log,max-message-size=1000.
Bug20798 - Improved http error handling
Previously, if the client that was performing an http request died and the backend couldn't transmit the response, then the backend was killed and the request was retried on another backend that also failed because the client socket was closed which wasted resources. The new behavior is to let the backend live on (without retrying), if the error is really from the client side.
SPARQL
Rfe11469: add line number to SPARQL parser error messages
The SPARQL parser will now indicate the line at which an error is encountered. The error message has also been reformatted to make it easier to understand the parsing problem.
Rfe11357 - enhance SPARQL 1.1 engine for changes to standard
The SPARQL 1.1 standards committee recently made several small changes to the standard. To accommodate these, AllegroGraph's SPARQL engine has:
- added support for the functions STRBEFORE, STRAFTER and REPLACE
- removed support for the function SHA224
- relaxed the type validation for REGEX so that language tagged literals and typed literals of type xsd:string can be searched without resorting to the STR function.
- returns rdf:langString as the datatype of a language tagged literal
- accepts solution modifiers for ASK queries
There are several other changes to the standard which AllegroGraph will support in a future release. These include:
- Grammar: Backslash character escapes in prefixed names
- Grammar: Fix: Allow ASK to take solution modifiers
- Grammar: Fix: AdditiveExpression: '?' => '*'
- Allow aggregates in ORDER BY clause
For additional information, please see the SPARQL release notes.
Bug20951 - SPARQL could fail to match plain and typed strings
SPARQL specifies that a plain literal (like "hello") is equivalent to a literal with type xsd:string (like "hello"^^xsd:string). The SPARQL 1.1 query engine was failing to provide this entailment in some cases. This has been corrected.
Bug20939 - CLEAR only recognized if it's the first verb
The SPARQL 1.1 UPDATE verb CLEAR was only recognized correctly if it was the first verb in a SPARQL statement. This has been corrected.
Bug20936 - OFFSET 0 and LIMIT did not mix
If a SPARQL query used an explicit OFFSET of zero, the LIMIT would be ignored. I.e., a query like:
SELECT * { ?a a ex:Animal . }
LIMIT 10 OFFSET 0 would return all ex:Animals rather than just 10. Leaving the OFFSET off entirely would produce the correct behavior. This has been corrected so that OFFSET 0 is treated as expected.
Bug20912 - Inequality filters could signal an error
Inequality filters in SPARQL queries can be transformed into fast range queries. If a query used a filter like
FILTER( ?a < 0.0 ) then the SPARQL 1.1 query planner could attempt to decrement an unsigned number below zero which would lead to a segmentation fault.
This has been corrected.
Bug20911 - SPARQL could lose precision on very small numbers
SPARQL numeric conversion routines could treat double-floats and decimals as single-floats which would lose precision. For example, a query with a filter like
FILTER( ?x > 4.5e-45 ) would have been interpreted as
FILTER( ?x > 0 ) This has been corrected.
Bug20898 - Complex queries could fail on federated triple-stores
Because of the way statistics were calculated on federated triple-stores, SPARQL queries with many clauses in a basic graph pattern (BGP) could fail when the query engine attempted to reorder them. This has been corrected.
Bug20891 - Improve SPARQL comparator selection
SPARQL specifies difference comparison operations depending on the type of the things being compared. The SPARQL 1.1 query engine could select the wrong operation if both the left and right hand sides were variables. This has been corrected.
Bug20884: improve Lisp client remote-triple-store SPARQL interface
The remote-triple-store implementation of the Lisp client was unable to execute SPARQL queries with non-trivial filter clauses. This has been fixed. (Repeated in the Lisp API section below.)
Bug20867: REPLACE function in SPARQL 1.1 used incorrect syntax for matches
The SPARQL 1.1 REPLACE function was implemented using a different syntax for pattern replacement than that specified by the standard. Replacements are now correctly specified using the $<number> syntax.
Bug20863 - Some SPARQL XSD numeric computations could fail
Depending on the datatypes of the numbers involved, it was possible for a SPARQL numeric computation to signal a type error rather than correctly returning a value. This has been corrected.
Bug20859 - Some filter combinations could cancel each other out
If a SPARQL query used IN (or several FILTERS like ?uri = constant) and if there were other filters on variables that overlapped with the variables in the IN construct, then it was possible for AllegroGraph to lose track of the additional filters in some circumstances. This happened due to two separate query optimizations handling the filters inconsistently. This problem has been corrected.
Bug20852 - remote-triple-store and SPARQL UPDATE don't mix
The Lisp remote-triple-store client could signal an error after evaluating a SPARQL UPDATE statement. The UPDATE would occur but the result would be mis-interpreted and signal an error. This has been corrected.
Bug20851 - SPARQL FILTERs could be too strict
Certain combinations of triple patterns in a BGP and FILTERs could cause AllegroGraph to misapply the FILTERs. For example, depending on the underlying triple-store, this query
SELECT ?first_film ?actor ?second_film {
?first_film dbpp:starring dbpr:Jack_Black , ?actor .
?second_film dbpp:starring dbpr:Joan_Cusack , ?actor .
FILTER ( ?actor != dbpr:Joan_Cusack )
FILTER ( ?second_film != ?first_film )
FILTER ( ?actor != dbpr:Jack_Black )
} could have misapplied the ?actor filter and produced fewer results than expected.
This has been corrected.
Bug20838 - SPARQL CONSTRUCT queries could fail for large result sets
Due to limitations in the internal mechanisms used to keep track of intermediate results, SPARQL CONSTRUCT queries that returned more than 100,000 rows would cause a segmentation fault. This has been corrected.
Bug20830 - Some inequalities could use the wrong comparisons
Some SPARQL 1.1 queries could select the wrong comparison code when comparing two variable clauses with either >= or <=. This could lead to incorrect results. This has been fixed.
Bug20821 - EXISTS or NOT EXISTS and additional filters did not mix
The new SPARQL 1.1 EXISTS and NOT EXISTS filters failed to parse correctly when combined with other FILTER clauses. This has been fixed.
Bug20805 - encodeforuri could produce incorrect results
The SPARQL 1.1 encode_for_uri could produce incorrect results for some UNICODE characters. This has been corrected.
AGWebView
Rfe10885 - Setting duplicate suppression from AGWebView
Added "Suppress duplicate statements" feature to the repository overview page. This sets the duplicate-suppression-strategy switch, which can be set to prevent new duplicate triples (both duplicates already in the store, and duplicates within the same commit) from being committed to the store.
Bug20996 - Webview: js error creating a new repo
This fixes a javascript error when creating a repository from the Catalog page.
Bug20993 - Webview: Internet Explorer (IE) compatibility mode
AGWebView now includes an html meta header for IE9 so that IE9 will use its documentMode instead of IE7 compatibility. Also tested with IE8.
Also using html5 doctype to get standards mode in all browsers.
Bug20963 - Fix session url in AGWebView
Urls for sessions are changed in Bug20814 (described under AllegroGraph Server: General Changes above). This fixes AGWebView and changes the url for its use of sessions to include the uuid.
Bug20894 - AGWebView not working with ie9 on intranet
When connecting to AGWebView on an intranet site IE may automatically use Compatibility Mode. A syntax error causes it to only display the "Loading" message and not continue. Work-around is to use a full url, or otherwise not use compatibility mode. The syntax error is now fixed so an intranet does not cause the error.
Bug20843 - AGWebView sometimes clears the editor
The execute button would clear the query editor (and the results area). This happened when the same query had been executed sometime before in the same browser followed by logout/login. The browser history mechanism had not been fully cleared. Recent queries are cleared after logout. This is now fixed.
Changes to the Lisp API
Bug20884: improve Lisp client remote-triple-store SPARQL interface
The remote-triple-store implementation of the Lisp client was unable to execute SPARQL queries with non-trivial filter clauses. This has been fixed. (Repeated in the SPARQL section above.)
Bug20883: improve the Lisp client's remote-triple-store caching interface
The Lisp client's remote-triple-store implementation did not support the spogi-cache-enabled-p or spogi-cache-size methods. This has been corrected.
Bug20185 - Remote lisp client used wrong value for restriction reasoning
For a remote reasoning store, this method used the wrong string internally, leading to an error when a query was made:
(setf (restriction-reasoning-enabled-p *db*) t) For a remote reasoning store, this method enabled restriction reasoning if the key :restriction-reasoning-enabled was present regardless of the key's value. Now respects t or nil.
(apply-rdfs++-reasoner :restriction-reasoning-enabled nil)
Documentation
No significant changes.
Python Client
Rfe11313 - Remove export*, use response capture
RepositoryConnection.export and exportStatements have been deprecated. They used client side serialization of statements that was buggy and not up to spec. The functionality has been replaced with a way to capture server responses of any supported mime type.
A new context manager function on RepositoryConnection is called saveResponse, and can be used as follows:
with open('out', 'w') as response:
with conn.saveResponse(response, 'application/rdf+xml'):
# The server response is written to response
conn.getStatements(None, None, None) You can reference the function testsaveresponse in src2/franz/openrdf/tests/tests.py for examples.
Java Client
Rfe11493 - Support writing N-Quads from an AGModel
With this change, the Jena adapter now supports writing to N-Quads via AGModel.write(out, "N-QUADS").
Rfe11438 - Provide N-Quads parser/writer META-INF/services in agraph.jar
With this change, the agraph.jar now provides N-Quads implementations for org.openrdf.rio.RDFParserFactory and org.openrdf.rio.RDFWriterFactory in META-INF/services (Sesame doesn't yet provide N-Quads implementations).
Rfe11436 - Add Sesame 2.6.2 src jar
With this change, the openrdf-sesame-2.6.2-onejar-src.jar has been added to eclipse project classpaths (repo and distribution) for convenience.
Rfe11397 - Support RepositoryFactory and Repository configuration
With this change, the Java client now supports Sesame's RepositoryFactory api and Repository configuration from RDF data. See javadoc for the new com.franz.agraph.repository.config package for details.
Rfe11240 - Support user/role management
With this change, AllegroGraph superusers can now dynamically manage users and roles. See the AGServer javadoc for details.
Rfe11230 - Support masquerading as another user
With this change, an AllegroGraph superuser can now masquerade as another user when issuing requests. See the javadoc for AGRepositoryConnection#setMasqueradeAsUser for details.
Bug20946 - IE javascript error in Mold: innerHTML is null
Work-around: the error dialog can be dismissed and the search results are still displayed. Fixed.
Bug20811 - Allow prolog queries from javascript
To run javascript the user must have the "execute arbitrary code" permission which should be enough to run prolog queries too. The bug that was fixed prevented the user from running prolog queries from javascript.
AllegroGraph 4.5
Server Changes
Rfe11338 - CJK bigram tokenizer
A new tokenizer :simple-cjk for Chinese/Japanese/Korean (CJK) text has been added. It uses bigrams for consecutive CJK characters as words. To use this tokenizer, specify :simple-cjk as the tokenizer option when creating or modifying a freetext index.
The bigram tokenizer can be used for CJK mixed text, but its simplicity may result in false positives. It also tends to index a much larger number of words compared to other tokenizers. See Free-text Indices and create-freetext-index.
Rfe11298 - Spread access to the same triple store across shared backends
Previously, when using shared backends, operations for a particular triple store tended to be routed to a single shared backend, even if it was busy. Now concurrent operations for a particular triple store are routed to a shared backend that is not already processing a request for that same store. Additional backends (up to the value of the "Backends" configuration parameter) will be started if necessary to achieve this goal. This is expected to increase concurrency in such situations.
Dedicated sessions are unaffected by this change.
Since this change is most beneficial when there are many shared backends available, the default setting of the "Backends" configuration parameter has been changed from 3 to 10.
Rfe11257 - Backup/restore user namespace settings
agraph-backup is enhanced in two ways:
The user namespace settings are now handled by the backup-all/backup-settings/restore-all subcommands.
The backup-settings subcommand can take a path to the executable of an older AllegroGraph's version of agraph-backup, and call it to archive the store data. With this feature, archiving an entire repository from an older AllegroGraph repository becomes a single command.
See Backup and Restore.
Rfe11099 - Add Apache Solr support to AllegroGraph
Now you can connect an external Solr text search engine from AllegroGraph server. With new prolog functions you can issue a query to the Solr engine to retrieve associated document IDs. See Solr Text Indices.
Bug20750 - Unused store closer holds lock too long
The thread which closes idle triple stores in backends held an important lock while trying to acquire locks on individual stores. If a lock on a store was being held by another thread (e.g., one running a long query), this could cause the whole backend to become unresponsive to other unrelated requests. This has been corrected.
Bug20736 - Federated-triple-store could mistreat xsd:strings in queries
In some cases a federated-triple-store could fail to account for the equality of plain literals and literals with a datatype of xsd:string. This has been corrected.
Bug20717 - Fix the UPI not in string table query issue
Depending on the result format and the client, queries that generated new strings (e.g., using SPARQL 1.1's BIND clause) could lead to a UPI not in string table error. This has been corrected.
Bug20712 - Implement xs:date and xs:time XPath constructor functions
The constructor functions for xs:date and xs:time are now implemented.
Bug20697 - Zero length property path with future parts
Zero length property path patterns now correctly match future parts that are not yet in the db:
?x :p{0} :a . Note that the same pattern with a literal worked even without this fix:
?x :p{0} "o" . Bug20690 - Property path results cardinality
In the presence of cycles, property path expressions did not always return results with the right cardinality. This has been fixed.
Bug20616 - Fix race condition in triple store termination
AllegroGraph now waits for a triple-store's subprocesses to exit when the last connection is closed instead of killing them while they could potentially be executing a clean shutdown.
Avoid Logging a backtrace during unclean shutdown
Under certain circumstances (such as when 'killall agraph' was used), there was a race condition during process termination that could result in an error and backtrace being logged to agraph.log. This has been corrected.
HTTP Client
Rfe11308 - Support UPDATE via POST /repository/ID/statements
The Sesame 2 protocol allows statements to be added to a repository by sending a POST request to /repository/ID/statements. AllegroGraph previously only supported update via POST requests to /repository/ID.
Now SPARQL update statements may be directed to either location. You can also POST an RDF document or data in the form of an application/x-rdftransaction to /repository/ID/statements.
Rfe11213 - Allow superusers to run queries with another user's security filters
AllegroGraph sessions belonging to superusers now allow their owners to use the X-Masquerade-As-User header in HTTP requests to activate another user's security filters.
Bug20671 - List the Hub process in the HTTP API
AllegroGraph now lists the Hub process in the HTTP API's process list and in WebView. The Hub process can not be killed or asked to perform a backtrace.
SPARQL
Rfe11320 - Trying to use SERVICE should signal an error
AllegroGraph's SPARQL engine does not support federated query (http://www.w3.org/TR/sparql11-federated-query/). Previously queries using the SERVICE keyword would fail silently. Now these queries will signal an error.
Rfe11302 - Support clauses with no variables
The SPARQL 1.1 engine now supports basic graph patterns where one or more of the clauses have no free variables. Previously, a query like
select * {
:a :p1 :c .
:p2 ?value .
} could produce incorrect results. This has been corrected.
Rfe11262 - Improve SPARQL 1.1 filter transformation
AllegroGraph's SPARQL 1.1 engine is able to move some filters out of the query processor and into the storage layer. For example, a query like
select * {
?a rdf:type ?class .
filter( str(?class) > 'z' )
} will move the inequality into a filter involving only the first clause. In some cases, however, the query processor could leave behind a stub filter which would unnecessarily add to total query time (without altering the results). This is now corrected.
Several other minor efficiency improvements have also been added to the query engine.
Rfe11254 - Improve SPARQL 1.1 sub-query implemenation
A problem in the sub-query implementation led to two failures in the DAWG test-suite. These have been corrected and additional tests have been added to the regression tests
Rfe11245 - Optimize Queries where the right-hand side of a join is a BGP
The SPARQL 1.1 engine now handles queries where the right-hand side of the join is a BGP more efficiently. Previously, it would compute the RHS separately and then join the two sets. Now, it merges the RHS BGP into the existing LHS set directly. This can provide some speed and memory improvements for some queries.
Rfe11135 - Implement SPARQL 1.1 DISTINCT aggregation
AllegroGraph's SPARQL 1.1 engine now implements DISTINCT in its aggregates. This means that a query like
select ?gender (count(distinct ?profession) as ?pCount) {
?person k:profession ?profession .
?person k:sex ?gender .
} group by ?gender order by ?pCount will now correctly return a count of the distinct professions for each gender rather than a count of all professions.
Rfe11098 - Integrate Apache Solr with SPARQL 1.1 query engine
Assuming that you have setup Apache Solr with an AllegroGraph triple-store, you can now query it using the SPARQL 1.1 query engine. See the description of using Solr in Solr Text Index. When used with SPARQL 1.1, there are two storage strategies: addinng a triple that associates the text with a Solr document ID and telling Solr to associate the text with the triple-id directly. Both are described in the referenced document.
Rfe11070 - Improve performance of variable predicate queries
AllegroGraph's SPARQL engine now recognizes queries that have variables in the predicate position and provides custom methods to optimize them. An example of such a query is:
SELECT ... {
... clauses ...
?s ?p ?o . Bug20778 - Several fixes for various SPARQL 1.1 issues
There were numerous issues in the initial release of the SPARQL 1.1 query engine. Some of the problems were caught by customers and others by internal review. This release of AllegroGraph fixes:
issues with several functions including: encodeforuri, some new mathematical functions, isLiteral, etc.
fixes several code paths that could lead to the UPI is not in string dictionary error,
fixed some edge cases in the SPARQL parser,
made some conversion functions more stringent so that, for examples,
10 dollarswould raise a SPARQL type error rather than being treated as the number 10.
There remain some known issues with SPARQL; please see the SPARQL release notes for details.
Bug20760 - Several improvements in SPARQL aggregation.
Do not include unbound variables in COUNT aggregation. Previously, AllegroGraph would include these in the count.
Remove bindings for SUM, AVG, etc. if any of the bindings in the group are unbound. Previously, AllegroGraph would simply ignore the unbound values are return the SUM of the remaining elements.
Allow non-numeric bindings for MIN and MAX. Previously, AllegroGraph assumed that MIN and MAX only applied to numeric bindings.
Bug20756 - Multiple OPTIONAL clauses for the same variable could fail
The SPARQL 1.1 query engine could fail to find some bindings when multiple OPTIONAL clauses referred to the same variable. For example, a query like
SELECT ?first ?last {
?s ab:lastName ?last .
OPTIONAL { ?s ab:nick ?first . }
OPTIONAL { ?s ab:firstName ?first . }
} could fail to find bindings for ?first that come from the second OPTIONAL. This is now corrected.
Bug20752 - Queries with both filters and property paths could fail
If a SPARQL 1.1 property path query used FILTERs and these FILTERs were moved into the triple patterns, then it was possible for the filters to be ignored. This has been corrected.
Bug20732 - SPARQL 1.1 property path improvements
In some cases, SPARQL 1.1 property path queries could do too much work and return too many results. This has been corrected.
Bug20723 - Improve handling of SPARQL filter predicates
Some of the SPARQL 1.1 filter predicates were failing to function properly when used in some contexts because their return value was not being correctly transformed from an internal representation to an external one. This has been corrected.
Bug20722 - SPARQL 1.1 parser treated whitespace differently
The SPARQL 1.1 parser required whitespace after a period and before an additional SPARQL keyword. This meant that a query like
select * { ?a ?b ?c.filter( ?c > 3) } would signal a parse error. This has been corrected.
Bug20721 - Improve handling of certain OPTIONAL edge cases
Depending on the data in the store, the SPARQL 1.1 engine could sometimes fail to process queries like:
select ?a ?b ?c ?d {
optional { ?a a ?b . }
{ ?c <urn:a:b> ?d . }
} where the OPTIONAL could lead to an empty group of bindings on the left-hand-side during query processing. This has been corrected.
Bug20711 - Improve some date and dateTime comparison issues
The SPARQL 1.0 and 1.1 engines behaved inconsistently with respect to one another for certain comparisons between xsd:dates and xsd:dateTimes. This could lead to differences in query behavior. These issues have been reconciled.
Bug20695 - Problem when computing range-queries on dateTimes
Both the SPARQL 1.0 and the SPARQL 1.1 engine could fail to correctly compare dateTime datatypes with fractional seconds. This has been corrected.
Bug20689 - Parsing of SPARQL property path expressions where a group is followed by a + as in
?x (ex:p)+ ?y no longer results in a error.
AGWebView
Bug20751 - Agwebview could submit queries twice in some browsers
On some browsers AGWebView could submit queries twice. This led to unnecessary work for SPARQL or Prolog SELECT queries and to duplicate data for SPARQL UPDATE queries. This has been corrected.
Bug20738 - AGWebView could not start remote sessions
A recent change in AllegroGraph broke AGWebView's ability to start sessions on remote stores. This is now corrected.
Changes to the Lisp API
No significant changes.
Documentation
Rfe11236 - Fix command line documentation of the agdebug tool
The agdebug tool now correctly indicates a non-optional argument.
Python Client
Rfe11229 - Add support for user masquerading to python client
Added support for user masquerading to Python client by adding RepositoryConnection.runAsUser(username).
Java Client
Rfe11329: AGConnPool, remove unnecessary dependency on Jena
If using AGConnPool with Sesame API, but not Jena, the Jena libraries are no longer required.
Rfe11310: AGConnPool should rollback when borrowing
Pool will now always rollback when borrowing a connection. This makes the transaction catch up to the most recent.
The 'testOnBorrow' property for the pool is now redundant, so generally not recommended.
Rfe11242: Upgrade to Sesame 2.6.2 jars
With this change, the Java client has been updated to use the Sesame 2.6.2 jars, including support for AGUpdate and AGRepositoryConnection#prepareUpdate methods. Java Sesame Tutorial example13 has also been updated to reflect use of prepareUpdate.
Bug20795: AGConnPool does not close connections
Closing the AGConnPool now closes sessions properly.
AllegroGraph 4.4.0.1
Server Changes
Bug20667 - XSD float exponent fix
Due to rounding issues when encoding, exponents of floats in the :sparql-xml result format were - very seldom - off by one. This has been fixed.
Bug20615 - TrxLogRetain replication now works on secondaries
The catalog directive: TransactionLogRetain replication now archives properly for replication secondaries.
HTTP Client
No significant changes.
SPARQL
Rfe11219 - Add query option support for all SPARQL constructs
AllegroGraph's SPARQL parser supports specifying query options as part of the query string. These take the form
prefix franzOption_OPTIONNAME: <franz:OPTIONVALUE> In particular, you can tell AllegroGraph which query engine to use with the queryEngine parameter. For example, to use the SPARQL 1.0 engine in a query, you would add the prefix
prefix franzOption_queryEngine: <franz:sparql-1.0> This extension is now supported for all SPARQL constructs: ASK, CONSTRUCT, DESCRIBE, SELECT and UPDATE.
Rfe11218 - Improve maximum solution size handling
Because the SPARQL 1.1 engine operates on the entire query solution simultaneously it can be more memory intensive than the SPARQL 1.0 engine for some queries. There was an internal check to warn if query solution sets were becoming too large. This check was causing some queries to fail rather than simply signaling a warning. This has been changed so that these queries will write a warning to the log rather than failing.
Rfe11202: SPARQL Update example with batch update
With this change, Java tutorial example 13 has been augmented to show how to evaluate SPARQL Update queries, including one that batches updates in a single request. [This item is repeated in the Java Client section.]
Bug20768 - SPARQL 1.1 property paths could return incorrect results
AllegroGraph was failing to filter non-matching rows for property path queries like
select ?a {
?a owl:sameAs{1} ?a .
} meaning that it would emit results when it should not. This has been fixed.
Bug20692 - Sparql 1.1 CONTAINS function can fail
Corrected a problem with some instances of the SPARQL 1.1 CONTAINS function.
Bug20676 - Geospatial datatype mappings could cause query errors
Changes to some query optimization code in the AllegroGraph SPARQL engines (both sparql-1.0 and sparql-1.1) were incompatible with geospatial datatype mappings (geospatial predicate mappings were not effected). This could lead to parse errors during query processing. This has been fixed.
Bug20665 - SPARQL-1.1 UNION doesn't see with-variable bindings on the RHS
The SPARQL 1.1 engine was losing track of the variable bindings on the right-hand side of a UNION. This could cause queries like
select ?s1 ?s2 {
{ ?s1 a ?p . }
union
{ ?s2 a ?p . }
} (with a variable binding for ?p) to produce incorrect results. This has been corrected.
Bug20664 - SPARQL-1.1 CONCAT function signaling errors with typed literals
The SPARQL-1.1 CONCAT function could signal an error instead of generating the correct result when one or both of the arguments with a typed literal. This has been corrected.
AGWebView
Rfe11198 - Improve WebView's bulk loading UI
WebView no longer conflates bulk load mode with agload usage. It's now possible to use agload's multi-core optimized loading mechanism and to optionally turn bulk load mode on. Some aspects of the file loading UI in WebView have also been optimized: The format selection element is now a combo box, and is located closer to the file input field.
Bug20662 - Reload triple count in WebView if load fails
WebView will now display the correct triple count in a store after an import got stopped by an error (e.g. the triple license limit).
Bug20641 - Improve restoring backups from WebView
When restoring from a backup in the WebView catalog overview, AllegroGraph will now automatically upgrade the newly-created store to the newest version. Additionally, there is now an option to set the store up for use as a replication secondary.
Bug20637 - Fix problem where SPARQL Update did not work with AGWebView
Changes to SPARQL UPDATE caused some UPDATE queries to fail in AGWebView. This has been corrected.
Changes to the Lisp API
Bug20640 - Fix printing of encoded-ids in remote lisp client
AllegroGraph's remote lisp client now correctly translates encoded IDs' UPIs back into strings. It can also correctly translate variable-width encoded IDs into UPIs now.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Rfe11202: SPARQL Update example with batch update
With this change, Java tutorial example 13 has been augmented to show how to evaluate SPARQL Update queries, including one that batches updates in a single request. [This item is repeated in the SPARQL section.]
AllegroGraph 4.4
Server Changes
Rfe11082: Transactional duplicate deletion
INCOMPATIBLE CHANGE: AllegroGraph no longer supports duplicate deletion at index optimization time. The database creation parameters indicating duplicate deletion strategy are no longer accepted.
Instead, AllegroGraph now allows deletion of duplicates using the delete-duplicate-triples API function, which is also accessible via the HTTP API call DELETE /statements/duplicates.
Rfe11039 - Include AllegroGraph version in ps output
AllegroGraph subprocesses modify their command line so that they can supply identifying information to programs like 'ps' and 'top'. Now, in addition to the name of the subprogram, the version of AllegroGraph is included. Also, for the AllegroGraph Service Daemon process, the port number that it is listening on is included.
Rfe11017 - Add an agdebug command line utility
AllegroGraph now ships with a command line utility called agdebug that will output a summary the contents of the transaction log files in a store. This is primarily useful for diagnosing problems with transactions and the database's storage system.
Rfe10984 - Fix serialization issues with encoded IDs
Both the turtle and the RDF/XML serializers failed to correctly handle encoded IDs. This has been corrected.
Rfe10981 - Improve upgrading/migration process
The agraph-backup tool has two new commands, backup-all and restore-all, to help migrating existing databases to the new version of AllegroGraph or copying all catalogs and repositories to a new location. See Backup and Restore and [Repository Upgrading[upgrade-guide].
Rfe10948 - Log secondary index creation
Adds log entries to the AllegroGraph log when a secondary index (also called a UPI map) is created.
Rfe10853 - Support fti charset in Python client
Keyword arguments innerChars, borderChars, and tokenizer have been added to createFreeTextIndex and modifyFreeTextIndex.
getFreeTextIndexConfiguration also returns these new configuration properties and values. See put-freetext-index.
Rfe10783 - Add per triple-store control of secondary indices
Modified the Lisp and HTTP APIs so that each triple-store can specify whether or not it should use existing secondary indices and/or create new secondardy indices during query. Since these indices must be maintained and updated when modifying the triple-store, automatic creation should be turned off until the store is quiescent. The default is to use secondary indices when they exist but not to create new ones. Use [create-secondary-indices-p[]/(setf create-secondary-indices-p), and use-secondary-indices-p/(setf use-secondary-indicies-p)* to control the behavior.
Rfe10568, Rfe11172 - Set the default query engine via agraph.cfg
AllegroGraph supports two SPARQL query engines. The algebra engine has been available for several years and is well tested, stable and robust. The newer, set-based, engine, named sparql-1.1 can be significantly faster than the algebra engine but may contain bugs that the algebra engine does not.
Out of the box, AllegroGraph uses the sparql-1.1 engine. This can be changed by using the QueryEngine parameter in the configuration file. If you include
QueryEngine algebra Then AllegroGraph will use the algebra engine. If you leave the setting off or use
QueryEngine sparql-1.1 Then AllegroGraph will continue to use the set based query engine.
The value of this configuration is read each time a triple-store is created or opened.
Rfe10179, Rfe10540 - SNA optimizations
Improved the efficiency of SNA operations in several ways. All SNA generator functions are now slightly faster. SNA generators that use select now save the select plan and re-use it. This can make these generators hundreds of times faster than before. The SNA system will also use secondary indices (UPI-maps) when available.
Rfe10398 - Improved speed of low-level AllegroGraph operations
Optimized the code used to copy UPIs and parts of triples. This produces a query speedup in SPARQL queries and most Prolog select queries. Some count-only Prolog select queries can run slightly slower.
Rfe10090, Rfe10541 - optimizations to AllegroGraph secondary indices
Improves both the creation of secondary indices (UPI-maps) and their efficiency in use. Queries that make heavy use of secondary indices will operate more quickly.
Rfe9786 - Make shm server global
A new configuration directive GlobalShmServerDirectory is added to specify a directory where the global shared-memory server creates its socket. When omitted, the value of SettingsDirectory is used.
Bug20619 Optionally not inherit trx log arch params during restore
The configuration file catalog directive TransactionLogArchive is no longer inherited from a backup database during a restore. In addition, when the --replica flag is set during agraph-backup restore, TransactionLogRetain and TransactionLogReplicationJobname are no longer inherited from the backup and must be set manually in the config file catalog section prior to restore. This is so that a replication primary and replication secondary can have independent transaction log archiving behavior. The flag --replica also ensures --nocommit is set and --newuuid is not set, and should be used as a replacement for --nocommit when restoring backups for replication client purposes.
Bug20460 - Encoded IDs and reasoning triple-stores mix not
If reasoning was used during a query, then encoded IDs were not correctly recognized which led to incorrect results. This has been corrected.
Bug20443 - Encoded IDs did not work with the TriX serializer
The TriX serializer was incorrectly emitting encoded IDs as if they were plain literals instead of resources. This has been corrected.
Bug20355 - Durably save transactions on warm-standby client
The warm-standby secondary machine now saves transaction log entries in a way that ensures they've been flushed to disk durably as soon as the entire commit has been read.
Bug20332 - Trix parser can get confused and try to encode plain literals
Under some circumstances, the TriX parser could attempt to encode plain literals as if they were typed literals which could cause a data import to fail. This has been corrected.
Bug20230 - Fix deletion slowdown under replication
The replication server used a connection to the replicated triple store that ensured it would never expunge deleted triples, slowing down operations if many deletes had occurred. Now, the replication server correctly expunges these triples and does not slow down.
Bug19957 - Turtle parser should support type-mapping
The turtle parser was not respecting a triple-store's type-mapping settings. This meant that triples added via turtle would not have their objects converted into encoded UPIs. This has been corrected.
Spr38044 - Restrict possible values for TransactionLogRetain
AllegroGraph will now signal an error when invalid values for TransactionLogRetain are encountered in the config file.
HTTP Client
No significant changes.
SPARQL
Rfe10985 - Spin functions and magic-properties
SPIN functions and magic-properties have been implemented. The SPIN implementation is described in Spin document.
Rfe10737 - SPARQL 1.1 BIND support
Add support for SPARQL 1.1's BIND language construct. In particular, queries like
select ?title ?costPlusTax {
?book <http://purl.org/dc/elements/1.1/title> ?title .
?book <http://example.org/ns#price> ?cost .
?book <http://example.org/ns#tax> ?tax .
bind( ?cost * (1 + ?tax) as ?costPlusTax )
} are now supported.
Rfe10736, Rfe10743 - SPARQL 1.1. exists, not exists, and minus
AllegroGraph's SPARQL engine now handles the MINUS operator of SPARQL 1.1. It also handles the new EXISTS and NOT EXISTS filter operators.
Rfe10700 - SPARQL 1.1 assignment
SPARQL 1.1 assignment is now implemented so that queries like
select ?title ?cost ?discount (?cost * ?discount as ?totalCost)
where {
?book :title ?title .
?book :price ?cost .
?book :discount ?discount .
} are now supported.
Rfe10699 - SPARQL 1.1 UPDATE support
The SPARQL parser and implementation now support the UPDATE specification at http://www.w3.org/TR/2011/WD-sparql11-update-20110512/.
Note: This change is incompatible with some existing uses of SPARQL UPDATE.
AGWebView
Rfe11131: Allow triggering deletion in WebView
It is now possible to invoke duplicate deletion from WebView.
Rfe10631 - Allow loading files via agload in WebView
WebView now has a "Bulk Load" option to speed up loading large files. Since this option uses the agload utility internally, it also allows specifying an error handling strategy other than "Cancel".
Bug20492 - Improve code editor in WebView
The text editor in WebView (used for e.g., queries) has been improved for browser compatibility (mainly with Internet Explorer) and functionality. It now expands automatically to fit the text contained, displays line numbers and automatically indents code.
Changes to the Lisp API
Rfe11082: Transactional duplicate deletion (repeated from the General section)
INCOMPATIBLE CHANGE: AllegroGraph no longer supports duplicate deletion at index optimization time. The database creation parameters indicating duplicate deletion strategy are no longer accepted.
Instead, AllegroGraph now allows deletion of duplicates using the delete-duplicate-triples API function, which is also accessible via the HTTP API call DELETE /statements/duplicates.
Documentation
No significant changes other than new features documented.
Python Client
Rfe11122 - Add "delete duplicates" API function to python client
RepositoryConnection.deleteDuplicates:
Delete all duplicates in the store. Must commit.
mode - "spog" or "spo"
Java Client
Rfe11121 - Support transactional duplicate deletion
With this change, the Java client now provides the method AGRepositoryConnection#deleteDuplicates(comparisonMode) for deleting spog and spo duplicates in a transactional manner. See the javadoc for more details.
See Change History for user-visible changes to earlier releases.
Rfe11117: java api for SPIN (adds to rfe10988)
New methods added to AGRepositoryConnection: listSpinFunctions() listSpinMagicProperties()
New classes: AGSpinFunction, AGSpinMagicProperty
Rfe10622 - Http response handlers now have their own package
With this change, the Java client's HTTP response handlers have been refactored and moved into a com.franz.agraph.http.handler package.
AllegroGraph 4.3.3
Server Changes
Rfe10984 - Fix serialization issues with encoded IDs
Both the turtle and the RDF/XML serializers failed to correctly handle encoded IDs. This has been corrected.
Rfe10967 - Nicer CSV output option
AllegroGraph now supports another CSV output format (MIME type application/processed-csv) that uses the UTF-8 encoding, prints literals with numeric types as numbers, and doesn't wrap string literals in quote marks.
Bug20460 - Encoded IDs and reasoning triple-stores failure
If reasoning was used during a query, then encoded IDs were not correctly recognized which led to incorrect results. This has been corrected.
Bug20451 - Fix encoded ID API on remote lisp client
The remote lisp client was missing a few functions in the encoded ID API: The functions that were missing are next-encoded-upi-for-prefix, encoded-id-prefix-registered-p and encoded-id-prefix-in-use-p. These are now implemented and available on the remote lisp client.
Bug20443 - Encoded IDs did not work with the TriX serializer
The TriX serializer was incorrectly emitting encoded IDs as if they were plain literals instead of resources. This has been corrected.
Bug20416 - Use proper n-triples syntax for text/csv
The CSV generator (when requesting MIME type text/csv) for query results will now always generate valid CSV such that the entries imported into a spreadsheet or a database are in correct n-triples part syntax.
HTTP Client
No significant changes.
SPARQL
No significant changes.
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Rfe10983 - Query execution with initial bindings
With this change, the Java client's Jena adapter now supports creating query executions with initial bindings using the method: AGQueryExecutionFactory#create(query,model,initialBinding);
AllegroGraph 4.3.2
Server Changes
No significant changes.
HTTP Client
Bug20424 - Disable 5-minute HTTP request timeout
AllegroGraph now allows HTTP API requests to run (and generate results) for longer than 5 minutes.
SPARQL
No significant changes.
AGWebView
No significant changes.
Changes to the Lisp API
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Bug20272 - NPE when no response stream is available
Previously, the Java client's HTTP response handler could throw an opaque NullPointerExeption if no response stream is available (this might occur due to a server crash). With this change, the response handler checks for a null stream and throws an exception advising the user that no response stream is available and to inspect the server log file for possible explanations.
AllegroGraph 4.3.1
Server Changes
Rfe10889 - Provide guaranteed duplicate removal behavior
As long as no triples are concurrently being added, completion of a level 2 database optimization will now ensure that all duplicate triples (as defined by the current duplicate triple removal mode) are deleted.
Rfe10850 - Ag3 plugins for TBC 3.5
TBC 3.5 no longer provides built-in support for AG3 databases. With this change, AG3 plugins are made available to work with TBC 3.5. The plugins are not compatible with previous versions of TBC due to conflicts with the built-in support for AG3.
AG3 plugins for TBC 3.5 will be available from this Eclipse update site:
http://franz.com/ftp/pub/agraph/eclipse/tbc/3.5
Bug20408 - Implement encoded-ids for lisp remote-triple-store
It is now possible to register, unregister and list encoded-id prefixes when using the remote lisp client.
Bug20397 - Load triples from files faster on remote lisp client
The remote lisp client now can load large files into the server, and will no longer run out of memory when doing so.
Bug20378 - Improve duplicate reduction in UPI-map based cursors
AllegroGraph reduces the number of triples examined during SPARQL queries by skipping subsequent duplicates when iterating over a cursor. This duplicate reduction was not occuring for cursors based on secondary indices (UPI-maps) which could lead to decreased query performance. This has been corrected.
Bug20372 - Fix part printing for literals and resources with same name
AllegroGraph would incorrectly print literals that had the same name as a resource. This is now fixed.
Bug20355 - Durably save transactions on warm-standby client
The warm-standby secondary machine now saves transaction log entries in a way that ensures they've been flushed to disk durably as soon as the entire commit has been read.
Bug20336 - Fix inconsistency on warm standby secondaries
Replication secondary instances for AllegroGraph would sometimes query and attempt to display inconsistent data while replication was copying transactions from the primary instance. This has now been fixed: AllegroGraph ensures that only data from transactions that have been fully been replayed is used.
Bug20230 - Fix deletion slowdown under replication
The replication server used a connection to the replicated triple store that ensured it would never expunge deleted triples, slowing down operations if many deletes had occurred. Now, the replication server correctly expunges these triples and does not slow down.
HTTP Client
Rfe10893 - Allow "unlimited" concurrent HttpClient connections
Previously, the Java client's AGServer class would default to using an HttpClient that limited the number of concurrent connections per host to 20 (up from the HttpClient's default of 2, per RFC 2616 sec 8.1.4), which could still result in unnecessary confusion and socket churn when interacting with application level connection pool management. With this change, the defaultMaxConnectionsPerHost and maxTotalConnections are both increased to Integer.MAX_VALUE to avoid conflict with application level connection pool management.
SPARQL
Bug20382 - Sparql queries that returned * could confuse remote-triple-stores
If a SPARQL query returned all of its variables (e.g.. SELECT * ...), then the remote-triple-store could fail to determine the actual variables returned by the query. This could in turn lead to incorrect results from clients that use this information (e.g., Gruff). This is now corrected.
AGWebView
Rfe10924 - Enable download of server log file from WebView
WebView now allows viewing and downloading the agraph.log server file for diagnostic purposes. To open the log, select the logfile entry in the Admin menu.
Changes to the Lisp API
No significant changes.
Documentation
No significant changes.
Python Client
No significant changes.
Java Client
Bug19798 - Illegal resource in Jena reified statements
Previously, the Jena adapter's createReifiedStatement method could create reified statements containing illegal blank node ids that were based on generated UUIDs, manifesting in invalid part errors from the server. With this change, these reified statements are now created using legal resources. AGGraphMaker instances can now be created with a specified ReificationStyle (the default style is ReificationStyle.Minimal).
AllegroGraph 4.3
Server Changes
Rfe10790 - improve logic for finding the sort-upi-map binary
There were cases when the Lisp client could fail to locate the path to the sort-upi-map external program. This has been corrected.
Rfe10665 - AG4 plugins for TBC 3.5.0
With this change, the AG4 plugins have been modified to work with TBC 3.5.0. The plugins are not backwards-compatible with previous versions of TBC due to changes in its API.
AG4 plugins for TBC 3.5 will be available from the update site: http://franz.com/ftp/pub/agraph/eclipse/tbc/3.5
Rfe10636, Bug20160 - signal an error if a blank node ID is too big, support for more blank nodes per store
AllegroGraph previously supported only 2^32 blank-nodes per triple-store. This is now increased to 2^56. Furthermore, if AllegroGraph tries to create a blank-node with an ID larger than the maximum supported, it will now signal an error rather than silently reusing existing nodes.
Rfe10632 - Enable per-user security-filtering of stores.
AllegroGraph now has a new class, security-filtered-triple-store, which allows specifying criteria that triples must fulfill in order to be accessible in that store.
AllegroGraph also automatically filters stores for users of a remote-triple-store if these users (or any of their roles) have security-filters defined.
Rfe10090, Rfe10541 - Optimizations to AllegroGraph secondary indices
Improves both the creation of secondary indices (UPI-maps) and their efficiency in use. Queries that make heavy use of secondary indices will operate more quickly.
Rfe10508 - Japanese full text search
An alternative text indexer for Japanese is provided. From the Lisp client, it is available by supplying :japanese as the :tokenizer keyword argument to create-freetext-index. When you use the Japanese tokenizer, we recommend you set :minimum-word-size to 2 or even 1 (the default is 3, but there are many Japanese words that consist of 2 characters).
The following is an example configuration for indexing Japanese text in literals of the object part of triples whose predicate is !<<description>.
(create-text-index "ja" :predicates '(!<description>)
:stop-words nil
:index-fields '(:object)
:tokenizer :japanese
:minimum-word-size 2)
Rfe10020 - add support for owl:SymmetricProperty
AllegroGraph's RDFS++ reasoner now supports owl:SymmetricProperty so that it will make inferences like:
P rdf:type owl:SymmetricProperty
a P b
===> b P a In the process of implementing this addition, several other small efficiency improvements were made so that the reasoner is now usually faster even though it is doing more work. For example, the time to run all 14 LUBM-50 queries went from 10.04 seconds down to 9.44 seconds.
Rfe8278 - Add ability to customize the tokenizer used by freetext indexing
The Lisp API create-freetext-index takes two additional keyword arguments, border-chars and inner-chars, to fine-tune the word tokenization behavior. Each argument takes a freetext-charset object, specifying a set of characters that can appear at the borders of a word, or inside a word. A freetext-charset object can be created by a new API freetext-charset.
The following example creates a freetext index in which only alphanumeric characters and a hyphen #\- consists a word; that is, #. and #_ breaks words.
(create-freetext-index "custom" :predicates '(!<description>)
:inner-chars (freetext-charset '(:alphanumeric #\-))) Bug20289 - Do not allow stored procedures to enter the debugger
If a stored-procedure written in lisp calls the function error with a non-error condition, the AllegroGraph backend process running the stored procedure would enter the debugger. This resulted in an unresponsive backend.
Now, the backend will catch invocations of the debugger, report the failure, and continue running.
Bug20273 - Suboptimal index queries broken
Under certain circumstances, suboptimal index queries could return triples that did not match the query pattern. This has been fixed.
Bug20160, Rfe10636 - signal an error if a blank node ID is too big, support for more blank nodes per store
AllegroGraph previously supported only 2^32 blank-nodes per triple-store. This is now increased to 2^56. Furthermore, if AllegroGraph tries to create a blank-node with an ID larger than the maximum supported, it will now signal an error rather than silently reusing existing nodes.
Bug20133 - Select queries should create filtered triple-stores only as needed
Prolog select queries were sometimes creating temporary graph-filtered triple-stores unnecessarily which wasted resources and time. This is now corrected: graph-filtered triple-stores are created only when necessary.
Bug19834 - Resource manager connection management bugs
Previously, global variables were used to maintain connection information between Lisp clients and one of the AllegroGraph subprocesses. This could cause problems if a single Lisp client made connections to two separate AllegroGraph servers or if an AllegroGraph server was stopped and restarted. This has been fixed.
Bug20037 - allow trailing slash on Server URL in AG4 dialogs
Previously, putting a trailing slash on the Server URL in AG4 dialogs would result in an error message that was hard to debug. With this change, dialogs are now robust to trailing slashes.
HTTP Client
Rfe10839 - Freetext index charsets and tokenizer in HTTP API
The HTTP API for freetext index management now supports the optional arguments "tokenizer" as well as "borderChars" and "innerChars", which specify the tokenizer and character sets for word boundaries.
Bug20250 - Fix federated remote stores in HTTP API
The /session resource didn't correctly create sessions when specified using the <http://example.com/repositories/test> syntax. Now, POSTing to /session correctly opens a session on a remote triple store.
SPARQL
Bug20305 - Encoded IDs were incompatible with SPARQL CONSTRUCT queries
SPARQL CONSTRUCT queries will not produce triples whose subject or predicate are literals. The check for this was incorrectly excluding triples with encoded IDs in the subject (or predicate) because these IDs have a different UPI type-code that the one used for resources. This has been corrected.
Bug20288 - Encoded IDs don't work with SPARQL/XML and SPARQL/JSON output
The SPARQL/XML results format would crash when used with encoded IDs; The SPARQL/JSON results format generated output but emitted encoded IDs as typed literals rather than as resources. Both of these problems are corrected.
AGWebView
Rfe10728 - Improve WebView store overview user interface
The AllegroGraph WebView store overview page now is better structured and allows easier access to commonly-used functions.
Bug20304 - Fix codemirror for Internet Explorer 9
Internet Explorer 9 users can now use WebView's text editor with syntax highlighting.
Bug20302 - Fix loading more results in Internet Explorer
WebView can now load more than 100 results (using the [more] link) in Internet Explorer.
Bug20296 - Fix query result downloads in Firefox 3.
Fix a JavaScript incompatibility that caused downloading query results from WebView to fail in Firefox version 3.
Bug20203 - Fix WebView for newer Internet Explorer versions.
Internet Explorer 8 and 9 support a new history tracking mechanism, and deprecated features that WebView used to keep track of history before. Now, WebView makes use of this new mechanism when it's available. This allows Internet Explorer to correctly run AllegroGraph WebView again.
Bug20069 - Uploading file into a context from WebView is broken
The error " is not a valid part" would always appear. This is now fixed.
Changes to the Lisp API
Bug20146 - Export UPI hash-table functions from db.agraph.user
In the transition from AllegroGraph 3 to AllegroGraph 4, the UPI hash-table functions ceased being exported from db.agraph.user. This is now corrected.
Bug20145 - (Lisp API only) loading AllegroGraph should provide module :agraph4.
AllegroGraph previously only provided the module :agraph which meant that using
(require :agraph4) could load agraph4.fasl unnecessarily. This is fixed. AllegroGraph 5.0 note: the module to require is :agraph5 and the file is agraph5.fasl starting in version 5.0.
Bug20144 - Fix interference from remote-triple-store ping
To ensure that a remote-triple-store remains opened, AllegroGraph's lisp API sends a ping message in the background in regular intervals. Previously, it would use the same session object to record responses from the server as a user's requests. This would cause errors when a ping message interrupts a long-running request.
Now, AllegroGraph uses a separate session object on the client's side to send these ping messages.
Documentation
Bug20149 - Correct documentation of datatype and predicate mappings (Lisp)
The Lisp reference manual had several typos in its discussion of datatype and predicate mappings. These have been fixed.
Python Client
No significant changes.
Java Client
Rfe10135 - Improve javadocs
Added overview pages, more docs for classes and methods, and links to AllegroGraph documentation for introduction, catalog, federation, etc.
Rfe10745 - Add Jena text indexing code example
With this change, a Jena text indexing code example is now available in the distribution in JenaTutorialExamples.java as example 12; this will eventually also be written up in the Jena tutorial document, but the focus here is on making the code example available as it is useful on its own.
Rfe10624 - AGQuery limit and offset modifiers
With this change, the Sesame and Jena AGQuery classes now allow limit and offset to be specified by the query object, outside of any limit or offset specified in the query string, making it easier for apps to override these modifiers.
Rfe10623 - Jena AGQueryExecution count methods
With this change, the Jena api now supports counts queries via the AGQueryExecution#countSelect() and countConstruct() methods. Solutions are counted server-side and only the count is returned to the client.
Rfe10492 and Rfe10493 - javadoc for AGRepositoryConnection
Add javadoc that points to SessionPorts for relevant methods. Add javadoc for type mappings, linking to http and lisp docs.
Rfe10452 - support N-Quads responses in the Java client
With this change, the Java client now supports N-Quads responses from the server. The property -Dcom.franz.agraph.http.defaultRDFFormat=NQUADS can now be used to configure the client to request N-Quads format by default for RDF statement responses; when this property is not present, the default format is TriX. The method AGHttpRepoClient#setPreferredRDFFormat() can also be used to override the default RDFFormat on a per connection basis.
Rfe10450 - Jena Prolog and transaction code examples
With this change, Jena Prolog and transaction examples are now available in the distribution in JenaTutorialExamples.java as examples 17, 18, and 22; they will eventually also be written up in the Jena tutorial document, but the focus here is on making the code examples available as they are useful on their own.
Rfe10447 - add AGGraphQuery#count() method
With this change, the java client now supports counting for graph queries (e.g CONSTRUCT and DESCRIBE) with the AGGraphQuery#count() method. Also added javadoc for the AGTupleQuery#count() method.
Rfe10446 - add count() examples to the Java tutorial
With this change, the Java tutorial shows how to use the AGTupleQuery#count() and AGGraphQuery#count() methods to to obtain query result counts without returning results (counting is done server-side, only the count is returned).
Rfe10445 - add example/test getting triple ids via prolog
With this change, example17 in TutorialExamples.java shows how to get triple ids via Prolog. TripleIdTests has also been augmented to get triple ids via Prolog.
Rfe10411 - Streaming large query results
When AGTupleQuery.evaluate() is used for sparql and prolog queries, the results are returned by the server in sparql/xml format. This is parsed by the Sesame library using a SAX XML parser and the results are collected in an ArrayList before the TupleQueryResult is returned. For very large result sets, this will cause an OutOfMemoryError.
Alternatively, AGTupleQuery.evaluate(handler) can be used where the handler methods are called as results are parsed. This can handle any length results without a memory error, but the API is less convenient. A new example has been added to the tutorial to demonstrate this: TutorialExamples.example3a()
A new class AGStreamTupleQuery can be used to wrap an AGTupleQuery so the TupleQueryResult will be streamed back. This provides the familiar iteration API rather than a handler, but the http response is only parsed as the iterator is advanced. The usage of this API is explained further in the javadocs: AGTupleQuery query = conn.prepareTupleQuery(QueryLanguage.SPARQL, "SELECT ..."); query = new AGStreamTupleQuery(query); TupleQueryResult results = query.evaluate();
There is also a new method in AGRepositoryConnection that will cause the connection to automatically wrap all tuple queries to use streaming: AGRepositoryConnection.setStreamResults(boolean streamResults)
Instead of SAX, the AGStreamTupleQuery uses an XMLStreamReader for parsing. Initial performance results suggest that in addition to being resistant to OutOfMemoryError this uses less overall memory than the SAX parser for small and medium result sets (10-500).
This approach of streaming results has not yet been applied to getStatements().
Rfe9445 - Upgrade slf4j, funnel through jcl, and add source jars
With this change, the java client now uses slf4j-api-1.6.1, and uses slf4j-jcl-1.6.1 to funnel sesame and jena logging through commons-logging-1.1.1 (used by httpclient-3.1). Source jars are also included for javadoc and to faciliate debugging.
AllegroGraph 4.2.1c
Server Changes
Clarify what retain for replication means
If the TransactionLogRetain is set to replication then all transaction logs will be saved until replication starts at which point those log files no longer needed will be archived. This behavior may be further modified in a future release.
Bug20257 - fix convertdb for some rare cases
The convertdb utility failed to upgrade some stores that had transaction logs with a certain size. Now it can correctly upgrade these stores.
Bug20236 - Fix startup when HTTPProxy is specified in config
AllegroGraph now starts up correctly when a HTTP proxy specification is present in the configuration file.
Bug20223 - Fix agload for Turtle files
A change in agload broke loading Turtle-formatted files with agload. Now, agload can correctly load the Turtle format again.
HTTP Client
No user-visible changes.
SPARQL
No user-visible changes.
AGWebView
No user-visible changes.
Bug20200 - Clarify format support in import forms in WebView.
AllegroGraph now supports loading .gz-compressed files (and autodetecting their file format) from the server's local file system. The file import dialogues in WebView also have a note about the file name format for autodetection.
Changes to the Lisp API
Rfe10790 - improve logic for finding the sort-upi-map binary
There were cases when the Lisp client could fail to locate the path to the sort-upi-map external program. This has been corrected.
Documentation
No user-visible changes.
Python Client
No user-visible changes.
Java Client
No user-visible changes.
AllegroGraph 4.2.1a
Server Changes
Rfe10689 - Increase unique string limit to ~4B
The limit on the number of unique strings in the string table was previously 1,431,655,765. This limit has been increased to 4,294,967,295 entries.
Rfe10562 - Change encoded-id marker to @@ (from @#)
The marker used to indicate that a URI represents an encoded-ID has been changed from @# to @@.
Rfe10532 - Return encoded ids in ascending order
The .../encodedIds?prefix=""&amount=# http request was returning the ids in reverse order. This has been fixed to return them in ascending sequential order.
Rfe10521 - Deleted hash table management based on generation
Deleted hash tables are now cleaned up as soon as they are no longer needed, preventing the store from accumulating unnecessary files.
Rfe10200 - Regulate speed of mergers
Triple index mergers now regulate their speed based on the number of triples to be merged and the rate at which triples are added after a merge has started.
AllegroGraph 4.2.1.
Server Changes
Rfe10483, Rfe10528, Bug20012 - Inadequate transaction log durability
When using the default TlogSyncMethod of "odirect", the durability of transaction log files was not correctly ensured. This release fixes that issue by preallocating transaction log files and fsyncing them before using them. Additional transaction log files are created as needed. In addition, transaction log files that are no longer needed are now recycled instead of deleted.
This change will have an impact on database creation time due to the initial preallocation. To compensate for this, the default TransactionLogSize parameter has been changed from 1 gigabyte to 256 megabytes.
There is a new configuration parameter DesiredTlogFiles which specifies the number of transaction log files which should be preallocated at database creation time. The default value is 2. Specifying a larger value helps lower the probability of additional transaction log files being created during commits. This parameter is documented in Server Configuration.
Rfe10475 - Improve I/O behavior under memory pressure
When under memory pressure, Linux can sometimes generate far more I/O than expected when minor page faults occur during accesses to memory mapped files. This can result in very poor performance, especially when the I/O device is already congested. This change instructs the operating system not to perform these extra I/Os.
Rfe10439 - RDSF++ reasoner now does not create duplicate inferred triples
In some cases, the RDFS++ reasoner could follow multiple owl:sameAs paths and generate many duplicate inferred triples. This increased AllegroGraph's memory footprint and decreased RdFS++ reasoning performance.
The reasoner now generates only a single inferred triple for each owl:sameAs path when searching for sameAs statements with a fixed subject and predicate or for ones with a fixed predicated and object.
Rfe10434 - Rename Lisp API function to duplicate-deletion-strategy
The Lisp API function duplicate-deletion-strategy alters the way AllegroGraph manages duplicate triples during index optimization operations. It can be one of:
t- the same as:spog(see below).nil- do nothing to duplicates.:spo- ensure that no more than one triple with a given Subject, Predicate and Object exists after index optimization operations are complete. A triple may be deleted if it differs from another only in its graph.:spog- ensure that no more than one triple with a given Subject, Predicate, Object and Graph exists after index optimization operations are complete.
Previously, this function was called delete-duplicates-during-merge-p.
This setting is also available via the HTTP interface.
Rfe10413 - Add check for 64-bit Linux
Added a check to install-agraph and configure-agraph scripts to ensure that they're running on a 64-bit Linux. This should reduce confusion in the case where AllegroGraph Server was accidentally installed on a 32-bit Linux host.
Rfe10400 - Minor cursor implementation changes
Some changes were made to enhance AllegroGraph's cursor mechanisms. These do not alter existing cursor behavior but help pave the way for future improvements.
Rfe10399 - Verify lisp client and server version match
As the lisp client is very sensitive to changes in the backend API exposed by the AllegroGraph server, it now checks that it is connecting to the correct version of the server.
Rfe10363 - Improved error message when a file cannot be found
The error message generated when a database file is not found has been improved to make it more clear what is causing the problem.
Rfe10354 - Redesigned --quiesce feature of agtestclient
The quiesce functionality in agtestclient was unnecessarily complicated and inefficient. The feature has been redesigned to require running on the host of the secondary server in question and also require the expected count and a timeout. This is sufficient for the purposes for which the quiesce function is used. Expected count must either be queried from the primary or the triple-count must be known beforehand.
A bug was also fixed with the order of the arguments in the delete function. (They had been reversed.) Also, help documentation was created, which can be accessed by running agtestclient with no arguments. Finally, the --add argument was renamed to the more appropriate --add-random.
Rfe10345 - Don't log warnings when opening NFS triple-stores.
Opening a triple-store on NFS caused unnecessary warnings to be issued. These warnings are no longer logged.
Rfe10342 - Command line wrapper for the lubm-generator Lisp module
Note: lubmgen can easily contaminate or erase a production database if used without clear understanding of its purpose.
Rfe10337 - Improve cursor resource management
Improve the management of the cursors AllegroGraph uses internally for triple counting and triple deletion.
Rfe10335 - Improve logging of (adjusted) store parameters
AllegroGraph will now log human-readable configuration values when opening a store if a human-readable presentation is available. (The system determines whether a value has a human-readable presentation or not. That is not under user control. Values which do not have a human-readable presentation are also logged but not in a human-readable way.)
Rfe10308 - AllegroGraph can use UPI-maps to run queries in more cases
Increased the number of times when AllegroGraph can use UPI-maps (secondary indices) to run queries. This improves the enhancements made in Rfe10141 and Rfe10142.
Rfe10243 - opening database should honor existing merge mode as default
It may have been possible that the duplicate removal mode is rolled back accidentally by reopening the database. The problem is fixed.
Rfe10184 - Improve custom hash table efficiency
Improve the way AllegroGraph handles some internal hash table computations. This will provide some very small performance improvements to general AllegroGraph operations. Note that none of these operations occur in inner loops so the performance gains will not be markedly noticeable.
Rfe10114 - Change default for TransactionLogRetain to "recovery"
In prior releases, the default value for the TransactionLogRetain setting was "all", which meant that, unless configured otherwise, all transaction log files since the the database was created would be saved. This release changes the default to "recovery", which means that only the transaction log files required for instance crash recovery are retained. See Transaction Log Archiving.
Rfe9338 - Improved error message due to session port connection failure
When using methods that start a session (setAutoCommit, addRules, and registerSNAGenerator), if there is a connection error, the error message points to the documentation about SessionPorts.
Bug20122 - Handle disk full on transaction log writing
Previously, encountering a full filesystem while writing to a transaction log would result in a crash. Now, AllegroGraph will (if possible) log a message to the logfile and to stderr noting the problem and will periodically retry the write operation until it succeeds.
Bug20118 - graph type-mapping could lead to incorrect data
If a triple was added with a graph field that was mapped to an encoded UPI (for example, a geospatial point or a typed literal) and the object field was not mapped (for example, a plain literal or a resource), then AllegroGraph would fail to correctly store the triple's object. This has been fixed.
Bug20089 - encoded-ids not working in non-object fields for clients
AllegroGraph was failing to translate encoded-IDs in fields other than the object field for any non-direct Lisp client. This has been fixed.
Bug20050 - Improve speed of bulk load mode in agload
agload was incorrectly handling the --bulk parameter which caused it to perform significantly more slowly when bulk mode was turned on. With this change, bulk mode is 6 to 7 times faster than it was before.
Bug20031 - agraph-backup backs up the same file twice
Previously, agraph-backup was backing up one of the database files twice. Though this did not result in a corrupt backup, it was wasting time and space. This problem has been fixed.
Bug19996 - Repair durability guarantees for NFS stores.
The change to implement rfe10216 in AllegroGraph 4.2 accidentally broke durability guarantees on NFS stores: Transaction logs on NFS were opened in an asynchronous write mode, which could lead to commits getting lost in the case of a crash.
Bug19969 - Correct problem printing times with fractional seconds
If a fractional second had a value less than 0.001, then AllegroGraph would incorrectly print the time (or dateTime) using scientific notation. It now prints these values correctly. E.g., it now prints 16:46:40.000133 rather than 16:46:40.1.33e-4.
Bug19954 - Commit/rollback could sometimes result in crash
Fixed a programming error which could potentially cause a commit or rollback to result in a program crash.
Bug19950 - Handling of tranaction log files corrected
In this release the 'recovery' and 'archive' values for the TransactionLogRetain catalog parameter function correctly. Previously these two values were treated as if 'all' were specified.
Bug19933 - Made agload recognize "nq" file format (nquads).
Previously, agload would reject files with extension .nq. Now they are correctly identified as nquad files and loaded.
Bug19916 - Improve :terse output
Modified the heuristic for the :terse output so that it will not emit empty strings. Previously, a URI like 'http://www.foo.com/bar/' would have a terse representation of ''; now it will use 'bar/'.
Bug19836 - Date and datetime with negative year
XSD requires BCE dates, for example, these are now accepted:
(value->upi "-0002-04-12" :date)
(value->upi "-0002-04-12T12:23:00Z" :date-time) Bug19682 - Federated-triple-stores error for type-mappings
Type mappings can now be used with federated stores where all component stores have the same type mapping, for example in methods such as collect-all-type-mappings and get-triples. Function collect-all-type-mappings returns the intersection of mappings for all component stores.
Mappings can still only be set on component stores, not a federated store.
Bug19116 - Datatype-mappings are not applied to graph
When adding a triple, if the graph is a literal with a datatype that can be mapped like object, it will be. Subjects and predicates are not mapped at all.
At most 3 triples can be added, which is the same as before:
- with object and graph mapped by datatype
- with object mapped by predicate-mapping and graph by datatype if this results in a different triple than #1
- with no mappings if :preserve-strings is t
HTTP Client
Rfe10360 - Better HTTP error messages for encoded-id errors.
Stack traces are no longer logged for too-wide encoded-id formats or bad re-registration of such ids.
Bug20124 - Make number of initial HTTP workers configurable.
The number of initial HTTP workers started by the AllegroGraph server used to be exactly the number of backends. However, that doesn't take into account any frontend sessions like WebView. This would cause long-running requests on startup (like opening a triple store) to cause shorter-lived requests to wait until the longer-running request was finished.
AllegroGraph now has a new top-level configuration parameter HTTPWorkers (defaulting to 50), which lets administrators specify how many HTTP worker threads to launch on startup.
Bug20022 - Add location-rewriting to the AG reverse-proxy.
The HTTP reverse-proxy that dispatched requests to server back-ends did not handle redirect responses with Location headers in any special ways. This could sometimes lead to clients being redirected to a backend URL, which did not work. The proxy now properly rewrites the Location header to point at the front-end URL.
Bug20013 - Requests to back-ends with Expect: headers fail
Due to a bug in the HTTP system, sending a request with a body an Expect: header (which CURL does by default) would cause a deadlock in the intra-server communication, and thus a timeout. This has been fixed.
Bug19948 - Allow full HTTP delete-duplicates control
The HTTP API used to expose deleteDuplicates as a boolean value, but there are actually three states: off, delete by SPO, and delete by SPOG. The strings "spo" and "spog" (as well as the existing booleans) can now be used in the HTTP API when inspecting or setting duplicate deletion.
SPARQL
Rfe10486 - SPARQL query planning should not require the PSOGI index
When a query involves filtering on potentially encoded triples (e.g., numeric or date ranges), then the SPARQL algebra query engine checks to see if certain ranges of triples exist during its planning phase. The checks that SPARQL makes requires the POSGI index and are inordinately slow if that index is not available. Previously, AllegroGraph would make the checks regardless. Now, it will now skip these checks if the index is not there.
Rfe10374 - Improve efficiency of some SPARQL queries
SPARQL's notion of equality requires AllegroGraph to examine several variants of some IRIs. For example, the literal "4" should be equivalent to the literal "4"^^xsd:string.
This change improves the efficiency of AllegroGraph's SPARQL equality computations. The performance improvement will depend on the query.
Rfe10371 - Improve SPARQL's handling of DISTINCT LIMIT queries
Improve SPARQL's performance for queries with a LIMIT whose solutions must be DISTINCT.
Rfe10348 - Iri= for subject and predicates (generalize SPARQL constraints)
Extended IRI= constraints for subjects and predicates. This allows queries like:
select * where { ?s ?p ?o . filter( ?s = ex:foo ) } to operate as if they were written:
select * where { ex:foo ?p ?o . } If multiple constraints are simultaneously operative, then only one will be used for a given pattern. Currently, these will be selected in the order: subject, object, predicate.
Queries for which this optimization is relevant will see greatly increased speed.
Rfe10344 - Improve SPARQL statistical BGP reordering logic
The statistical BGP reordering function could produce sub-optimal results if a SPARQL query had multiple levels of nested UNIONs or OPTIONALs. This could cause queries to take much longer than necessary.
Rfe10312 - Optimize IRI= for object fields in SPARQL queries
This optimization converts IRI= constraints for object fields in SPARQL queries into the equivalent storage level get-triples calls. Previously, a query like
SELECT ?article
WHERE { ?article rdf:type ?type .
?article swrc:pages ?value
FILTER (?type = bench:Article ) } would fetch all of the rdf:type triples and then filter for those whose object field was bench:Article. Now the query will be executed as if it had been written as
SELECT ?article
WHERE { ?article rdf:type bench:Article .
?article swrc:pages ?value
}
Rfe10118 - Added interface to SPARQL query analysis to Python client
After calling RepositoryConnection.prepareTupleQuery() on a SPARQL select query, one can call analyze() on the resulting query object to get a string returned with information about which indices would be used to satisfy the query.
Currently, the query object's method is:
def analyze(self, analysisTechnique=None, analysisTimeout=None):
Analysis is only available for SPARQL queries.
analysisTechnique defaults to "executed", which executes
the query to perform dynamic analysis. "static" analysis is
the other option.
For analysisTimeout, pass a float for the maximum number of
seconds to run the query if executed. Rfe9507 - Add initial support for SPARQL subqueries
SPARQL subquery support includes ORDER BY, LIMIT, and DISTINCT. An example query would be:
PREFIX fn: <http://www.w3.org/2005/xpath-functions#>
PREFIX boat: <http://example/boats/vocab/>
PREFIX boats: <http://example/boats/>
SELECT ?boat ?boatName
FROM boats:boats1
FROM boats:boats2
WHERE {
# Select the 2 boats with the shortest names.
{ SELECT ?boat WHERE {
?boat boat:placed ?position ;
boat:name ?name .
} ORDER BY asc(fn:string-length(?name)) LIMIT 3 }
# Select their crew and the boat's name.
?boat boat:name ?boatName .
}
ORDER BY desc(fn:string-length(?boatName)) desc(?boatName) Note that subqueries are a pre-release SPARQL 1.1 feature; documentation and conformance tests are not yet available from the W3C, semantics are subject to change, and the implementation might be incomplete in a manner that is visible to user code. Please contact Franz support if you suspect that you have found a discrepancy.
Bug20070 - fix problem with geospatial encoded UPIs and some output formats
If a SPARQL query returned geospatial encoded UPIs and used the SPARQL-XML result format, then an error would occur. The same error could occur when serializing geospatial UPIs in the turtle format.
There was also a problem correctly serializing geospatial output using the SPARQL-JSON format: incorrect latitude and longitude values were sometimes generated.
All of these problems are corrected.
Bug20047 - Can't download query results with missing values as CSV
Taking a SPARQL query that produced unbound variables (say, with OPTIONAL) and asking for a comma-separated value return format would return an error. This has been fixed.
Bug19989 - Sparql CONSTRUCT could generate invalid blank nodes
SPARQL CONSTRUCT using the N-Triples format could generate invalid blank nodes.
Bug19832 - Correct bug in low level SPARQL equality code
Correct a bug in some rarely executed SPARQL equality code. The bug would have caused some queries in some datasets to fail to run.
AGWebView
Rfe10370 - Use only relative URLs in WebView
This makes it possible to set up a reverse-proxy that proxies, for example, host:80/allegrograph/ to host:10035/, without breaking WebView.
Bug20051 - Downloading query results from WebView was broken
After recent patch (bug20022), trying to download query results from WebView always resulted in a 404 response. This fixes that.
Bug20016 - Fix horizontal scrolling in WebView for Firefox
A bad CSS rule was preventing Firefox from ever displaying a horizontal scrollbar for WebView. This made it hard to inspect, for example, query result tables that were wider than the window. This is now fixed.
Bug20009 - 4.2 WebView can't handle some data saved by 4.1
Saved queries saved in 4.1 would cause an error when used in 4.2, because of a change in the way the client-side code is handling JSON data. This change fixes that, by falling back to the old parsing method if the new one fails.
Bug19972 - Webview breaks when it doesn't find JSON support
On older browsers without a JavaScript JSON object, WebView's detection of that object was failing, causing the whole script to abort. It now properly detects, the functionality, replacing it when not present.
Changes to the Lisp API
Rfe10420 - Add lisp client function optimize-indices
AllegroGraph's Lisp client now has a function optimize-indices that will cause the server to rearrange the indices for optimal query performance. This optimization pass can take a long time and cause a lot of I/O operations to be performed on large stores.
Rfe10066 - Add option to use rapper for load-turtle
The lisp API load-turtle now has a :use-rapper-p keyword parameter. See load-rdf/xml documentation for argument usage details.
Rfe9715 - Lisp client package now includes Lisp tutorial examples
The subdirectory tutorial now holds the Lisp and data example files discussed in the documentation.
Bug19672 - Triple-store-exists-p leaks server information
The Lisp client's successful return value from triple-store-exists-p sometimes provided the store's path on the server's filesystem.
The API has been fixed to return only the values nil or t.
Bug19133 - get-triple now applies geospatial mappings
get-triple now correctly applies geospatial mappings to the triple returned. Also, the get-triple now returns a fresh triple (rather than one which might be reused by internal mechanisms).
Documentation
New JavaScript document for programming the server
See JavaScript.
New Security document describing data security
Rfe10341 - Improve documentation search
Added basic documentation searching to the AllegroGraph documentation.
Python Client
Rfe10505 - use session to get new blankNodeIds
AGValueFactory was using the main repository url to get blank node ids. Use connection session instead if available. Using the session can reduce load on the primary server.
If the ValueFactory was created by Repository instead of RepositoryConnection, the old behavior remains.
Rfe10418 - Call indices/optimize from Python
Added an optimizeIndices(level, wait) method to RepositoryConnection for exposing the http indices/optimize POST request to Python client.
Rfe10326 - Support deleteDuplicates setting from Python client
The Catalog.createRepository function now has a deleteDuplicates keyword argument for setting the store's deleteDuplicates behavior on creation. If unspecified, the behavior is the server's default setting.
The store's deleteDuplicates setting is also available from a Repository object instance either as a property variable (delete_duplicates) or as setDeleteDuplicates and getDeleteDuplicates methods. This is a per-store setting.
See the HTTP Protocol - SPARQL Endpoint documention for setting values and behaviors (text search for deleteDuplicates).
Rfe10325 - Added evalJavaScript to Python client
Added an evalJavaScript function to the RepositoryConnection class for evaluating server-side JavaScript code.
See the server's JavaScript documentation for details on programming the server in JavaScript.
Java Client
Rfe10389 - Upgrade to sesame 2.3.2
With this change, the java client distribution has been upgraded to use openrdf sesame 2.3.2, a minor upgrade from 2.3.1.
Rfe10373 - Support deleteDuplicates mode in java
With this change, the Java Client supports setting a repository's policy for deleting duplicates using methods AGRepository#setDeleteDuplicatesMode and AGRepository#getDeleteDuplicatesMode.
Rfe10372 - Support periodic commit during add/load
With this change, the java client now supports the /statements commit parameter for periodic commits during add/load via the AGRepositoryConnection's setUploadCommitPeriod and getUploadCommitPeriod methods.
Rfe10257 - Support script loading in sessions
Added method to AGRepositoryConnection: addSessionLoadScript(String scriptName)
When a dedicated session is created, the named scripts on the AllegroGraph Server will be loaded.
Rfe10256 - Support loadInitFile switch in java
Added method to AGRepositoryConnection: setSessionLoadInitFile(boolean loadInitFile)
When a dedicated session is created, the initFile on the AllegroGraph Server will be loaded if this param is true.
Javadocs on AGRepositoryConnection reorganized to describe sessions.
Rfe10201 - Support adding N-Quads from java
With this change, the java client now supports adding N-Quads data from Sesame and Jena using the RDFFormat AGRDFFormat.NQUADS and the language "NQUADS", respectively.
Rfe10177 - Support getting statements by id
With this change, the java client now supports getting statements by triple id using the AGRepositoryConnection getStatements(String... ids) method. Note that there is currently no method for obtaining a statement's id in the java client, so this api is for experimental use by developers and subject to change.
Rfe10174 - Support adding Turtle from Sesame and Jena
With this change, Turtle formatted data can now be added to a repository using the RDF4J and Jena APIs.
Rfe10059 - SPOGI triple cache management
With this change, the SPOGI-cache can now be managed from the java client using AGRepositoryConnection methods enableTripleCache(size), getTripleCacheSize(), and disableTripleCache().
Rfe9436 - Session lifetime management
With this change the lifetime (maximum idle time) of a session can be set using the AGRepositoryConnection#setSessionLifetime method. The default session lifetime (in effect when not set with the method above) can be set with the AGHttpRepoClient.setDefaultSessionLifetime method. When none of these are set, the default session lifetime is 1 hour.
Rfe8362 - Allow geospatial queries against graph/context field
A boolean useContext parameter can be given to /geo/box, /geo/circle, /geo/haversine, and /geo/polygon, to make them query the context field, rather than the object field of the triples.
Bug19906 - Writing a JavaScript NaN value as JSON loops forever
When running, for example, the JavaScript program 'NaN', and asking for a JSON response, the server would never respond, because the float-printing code didn't handle 'special' float values. We now convert NaN and Infinity to strings in JSON output.
Bug19342 - Java LUBM Prolog and SPARQL queries
With this change, Java LUBM Prolog and SPARQL queries are updated to use distinct in appropriate places and located in the test.lubm package as AGLubmProlog and AGLubmSparql; ant targets lubm-prolog and lubm-sparql have been added to facilitate running the queries from the command line.
AllegroGraph 4.2
The new (in 4.2) document Release Notes has information about the 4.2 release. The document you are reading simply lists, for the current and for previous releases, user-visible changes (new features, bug fixes, and other modifications) in an abbreviated format. All the 4.2 changes listed here are also listed in the Release Notes but that document does not list changes in earlier releases. We suggest that users start with that document and refer to this one only if they need information about earlier releases.
Welcome to AllegroGraph 4.2. See Release Notes for general release information. The following, organized by category, are user-visible changes in AllegroGraph 4.2.
Server Changes
Rfe10293 - Warn against using ReiserFS.
As ReiserFS causes long delays on checkpoints, AllegroGraph now warns (in the server log file) if a store resides on ReiserFS. To ease debugging, it also logs the file system types of every store directory now.
Rfe10267 - Do not request duplicate removal if there are no duplicates detected
Internal handling of duplicate removal process is slightly improved. This fix also eliminates excessive "Dupes deleted: 0" messages from the agraph.log file.
Rfe10255 - Allow minimum text-index word size of 1
It is now possible to create freetext indices with a minimum word size of 1 character (the minimum used to be 2).
Rfe10216 - Support alternative tlog synchronization methods.
AllegroGraph now supports a new per-catalog configuration option, TlogSyncMethod, which allows users to specify one of three synchronized writing methods for transaction logs: ODIRECT, SYNC, and fsync.
As before, ODIRECT is the default and the recommended choice on ext3 file systems. For catalogs residing on non-ext3 file systems, the other choices may yield performance benefits.
Rfe10191 - Have install-agraph pass its arguments to configure-agraph
You can now do './install-agraph /dir/ --non-interactive ...' to install and configure agraph in a single command.
Rfe10182 - Improve efficiency when waiting for an in-memory chunk
Previously, when waiting for an in-memory chunk to become available, a commit operation would poll once per second. The polling has been replaced with a wait on a semaphore which will result in immediate notification when an in-memory chunk becomes available. This results in lower average times for commits that must wait for in-memory chunks.
Rfe10173 - Upgrade plugins to support TBC 3.4.0-Beta1
Previously the AllegroGraph plugins were only supported on TBC 3.3.x; with this change, the plugins can also be used with TBC 3.4.0-Beta1.
Rfe10167 - Log euid and runas information at daemon startup.
AllegroGraph now logs which user started the AllegroGraph service daemon, which user it was meant to switch to once running, and the currently effective user ID.
Rfe10162 - Perform a clean shutdown on agraph-control stop
Previously, running the agraph script with the stop argument would not verify that open databases were shut down correctly, leading to a forced recovery from transaction logs upon next startup.
Now, running the agraph script with the stop argument will wait for up to 60 seconds for open triple-stores to shut down cleanly.
To get the quicker shutdown behavior, use force-stop argument or kill the Service Daemon process with the QUIT signal.
Rfe10156, Rfe10157 - Improve lock acquisition and store creation on NFS.
Previously, creating a triple-store left open a very narrow window in which another process could clobber the store parameters. Now, the triple-store is locked immediately on creation. Additionally, newly-created triple-stores can now be cleaned up correctly when created on NFS.
Rfe10149 - Support setting a graph for inferred triples in HTTP sessions
Using the syntax <store>[rdfs++#
Rfe10147 - Export 'custom-service' macro from the db.agraph package
This macro was crudely inserted directly into db.agraph.user before. It is now imported there from db.agraph, like all other AllegroGraph functionality.
Rfe10143 - support bulkMode in connections
With this change, connections can now be put into bulkMode. When in bulkMode, data can be added/loaded more quickly, but there is no guarantee of durability in the event of a crash. Added method AGRepositoryConnection#setBulkMode to set the connection's bulkMode (defaults to false), and added method AGRepositoryConnection#isBulkMode to check the mode.
Rfe10141, Rfe10142 - Use UPI-maps more often inProlog queries
Improved the Prolog query optimizer to use UPI-maps in more cases. Also added a new function, qm, that will use a UPI-map whenever possible. The optimizer changes will help existing queries run more quickly; the new functor will allow more hand-tuned query optimizations.
Rfe10117 - implement run-time (dynamic) query index analysis
Query index analysis now comes in two flavors: the existing :static flavor (from rfe10075) and the new :executed flavor. The later provides an accurate account of the index usage because it actually runs the query while monitoring which indices are touched. Because the query is run, this flavor of analysis can be much more time consuming.
Note that the default analysis technique is to use the new :executed flavor. You can use a different technique by using the queryAnalysisTechnique query parameter. You can also specify a timeout using the queryAnalysisTimeout parameter. See Query Analysis.
Rfe10069 - AllegroGraph v4.0.6 introduced a dependency on a patch to update message digest functions. Now, loading the AllegroGraph client will fail early if this patch is missing, and will report an actionable error.
In addition to the check for the SHA1 patch, AG will now report missing patch names to the user, and offer to install these patches specifically.
Rfe10021 - Support Soundex freetext indices
This adds a supported freetext wordfilter 'soundex', which will reduce words to their Soundex equivalent when indexing or searching. Soundex is described in this Wikipedia article.
Rfe9877 - Lazier collection of FTI results
The triple-ids matching a free-text query will no longer all be accumulated at the start of the query, but (as far as possible) lazily, while triples are being fetched. This should improve performance of limited queries with lots of results.
Rfe9861 - Improve method of sending an x-rdftransaction
Previously, sending an application/x-rdftransaction request required escaping to the client's http layer. With this change, the method AGRepositoryConnection#sendRDFTransaction can now be used; this method is useful for bundling add/remove operations into a single request and minimizing round trips to the server. Changes are committed iff the connection is in autoCommit mode. For increased throughput when sending multiple rdftransaction requests, consider using autoCommit false and committing less frequently.
Rfe9746 - Various improvements to agload
- Added the --rapper command line option to tell agload to use the external rapper program when parsing RDF/XML.
- Added :trix to the list of supported input formats
- Added :save as an error strategy to write errors to the file agload.log and continue.
- Added --recursive to process source directories recursively.
Rfe9657 - Unattended execution of configuration script
The configure-agraph script now has --non-interactive mode, where all its parameters are specified on the command line, so that it can be called from a script without human intervention.
Rfe9631, Rfe10028 - db-get-triples improvements, UPI map usage
Improved the performance of AllegroGraph's internal get-triples functors and extended them to use secondary indices (UPI maps) when they are available.
This greatly improves the performance of many standard queries for both SPARQL and Prolog select. For example, the SP2 benchmark for 10,000 authors goes from a total time of 112.58 seconds to 37.91 seconds.
Rfe9622 - support getting the server's version information
With this change, the client can now get the server's version, build date, and revision information using the AGServer methods getVersion(), getBuildDate(), and getRevision(), respectively.
Rfe9517 - Unify AServe and AGraph messages in the AGraph log
Our HTTP server used to output log messages in its own format. We now route these through the AGraph logging system, causing them to follow a unified format.
Rfe9515 - Drop deleted triples from text index during merge.
The freetext index used to never un-index anything it had indexed. It now leaves out triples that have been deleted when merging index chunks, which will prevent unbounded index growth in stores where lots of triples are being deleted.
Rfe9470 - Reduce madvise calls during compressed index lookup
Small performance improvement during load and query.
Rfe9296 - Transactionally safe duplicate triple elimination
There are improvements in the functionality to remove duplicate triples. Removal is done more eagerly and more robustly. When opening a database, you can pass one of the following values as the :delete-duplicates-during-merge argument:
- nil - do not remove duplicates (default)
- :spog - duplicate triples with the same s, p, o and g are deleted.
- :spo - duplicate triples with the same s, p, o are deleted
- t - same as :spog, for backward compatibility.
Rfe8414 - Add encoded-ids to AllegroGraph
Adds support to register and query encoded-ids and to use these to generate sequences of UPIs in a triple-store.
Bug19910: agraph-recover now returns 0 when successful, 1 otherwise
Agraph-recover now correctly returns 0 to shell when successful. (see Point-in-Time Recovery).
Bug19897 - SHM hangs if socket creation fails
Previously, if a socket creation failure occured in the SHM server (which is a critical subprocesses associated with every database instance), database startup would hang. Now the error is immediately reported, allowing AllegroGraph server to otherwise continue operating.
Bug19891 - Fix log level test for log-backtrace and when-debugging.
Error logging no longer ignores the preferred log level for certain messages.
Bug19888 - Prevent logging gratuitous errors on forced shutdown.
AllegroGraph no longer leaves error messages in the server log file when force-terminated with the killall command. (Trying to write such log messages could result in the service daemon trying to communicate with dead processes.
Bug19864 - Some cursors could fail to correctly free resources
There were situations in SPARQL querying where a cursor could fail to correctly release its resources even when it was correctly discarded. This could lead to resource starvation. This is now fixed.
Bug19848 -- Improve behavior of qm when UPI-maps are not available
The qm functor now gracefully drops back to the behavior of the q functor when UPI-maps are not available. Previously, it would signal an error.
Bug19842 - Error message server-pid-and-client-pid-must-match-error
Fix error message when server is running as agraph and client is running as root.
Bug19813 - Fix store parameter merging for configured values.
Opening a triple-store by passing configuration parameters to open-triple-store now correctly overrides those values that are already present in the store's parameters configuration file.
Bug19816 - FTI Merger sometimes produces corrupt chunks
In a database with deleted triples, it was possible for the text-index merger to output a chunk that could not be read, which would bring down the whole FTI subsystem.
Bug19800 - support empty enumerations in owl:oneOf
AllegroGraph's RDFS++ reasoner signaled an error when an owl:oneOf claimed that some class X was one of the empty set. Now it works correctly for this case too.
Bug19791 - Fix checkdb's --port argument.
Previously, the checkdb utility could check triple-stores only if they were served on the default port, 10035. Now, it uses the value from its --port argument to determine the port to connect to.
Bug19786 - Checkdb now exits with 0 on success.
The checkdb utility now follows the unix convention of returning 0 if no problems were detected.
Bug19785 - Convertdb now exits with 0 on success.
The convertdb program now follows the unix convention of returning the status code 0 if there were no errors.
Bug19777 - Replication to multiple secondaries improved
The speed of replication will no longer be determined by the slowest secondary. Each replication job will now proceed as fast as each secondary database can process the data
Bug19766 - Sub-optimal index selection
Previously, in certain circumstances (such as when a range query was being used and one or more of the standard index flavors were not available) the index flavor selection code would exclude some index flavors that it could have used to satisfy a query more efficiently. This has been fixed.
Bug19773 - Dropping an index is not crash recovery proof
Fixed a bug where dropping an index followed by uncleanly shutting down a database would result in an error during restart recovery.
Bug19765 - Detect more cases of string table corruption.
The checkdb program can now detect many more inconsistencies in string tables.
Bug19764 - N-Quad output was incorrect
The output of AllegroGraph's N-Quad serializer when printing single triples was incorrect.
This is now fixed.
Bug19697 - improve performance of part printing
Increased the speed and memory efficiency of part printing.
Bug19657 - Fix race-condition in text-index merger
The merger could sometimes pick the chunks it was to merge from an outdated set of chunks, resulting in chunks being merged multiple times. This would cause duplicate triples to show up in FTI search results.
Bug19600 - direct-upi wire format does not handle nil values
The wire format used to communicate query results to a remote-triple-store broke when given a nil value, though the result of SPARQL queries with an OPTIONAL part may return them. This is now fixed.
Bug19587 - Suboptimal encoding of some characters in shm-queue.
Some characters were being encoded with an extra unnecessary byte. This problem has been corrected.
Bug19578 - Clearnamespaces XMLTransaction tag is broken
There was a bug in the XMLTransaction parser that causes requests containing a <clearNamespaces/> tag to fail. This has been rectified.
Bug19550 - Fix reification in rdf/xml parser
Fixed a bug in which certain statements imported from RDF/XML files would not be reified correctly.
Bug19549 - RDF/XML parser loses the trailing /
When importing from RDF/XML sources, AllegroGraph used to rewrite URIs that came without a path, but with a trailing slash and import them without the trailing slash (e.g., "http://franz.com/" would be imported as "http://franz.com"). Now, it imports all URIs verbatim.
Bug19530 - Fix off-by-one error in text-index code.
Adding a 255-character (the maximum) word to a text index would cause searching and merging of index chunks to break, because a too-small buffer was allocated when traversing the index. This is now fixed.
Bug19522 - Prevent spurious warning in some Prolog select queries
If both :limit and :count-only were specified in a Prolog select query, AllegroGraph would signal a spurious warning. This is now corrected.
Bug19517 - improve Lisp API for remote-triple-stores
Previously, if a :url parameter was used, AllegroGraph would ignore other parameters to create-triple-store like :user and :port that may be inconsistent with :url value. AllegroGraph now ensures that the :url and any other parameters agree and merges parameters into the :url if necessary. This means that you can now use constructs like:
(create-triple-store "foo"
:triple-store-class 'remote-triple-store
:url "http://blade4.franz.com"
:catalog "java-tutorial"
:user "test" :password "xyzzy"
:port 11101) Bug19515 - Correct precision loss in printing of RDF time and dateTimes
Certain RDF time and dateTime values with fractional seconds could fail to print all of the digits stored for the value. This patch corrects this problem.
Bug19508 - Prevent text-index from leaking files on crashes
When processes crashed 'hard' (hang, or kill -9), the text index would sometimes leave files around that it never cleaned up. It now takes care to (at some point) delete every file it creates.
HTTP Client
Rfe10278 - Alphabetize store and catalog lists in HTTP responses
The lists of repositories and catalogs returned through the HTTP API were ordered more or less randomly. This made finding a repository in a long list in WebView cumbersome. They are now sorted by name.
Rfe10234 - Inefficient limited Prolog queries over HTTP
The HTTP server used to fetch all results, then throw away the extra ones, when executing a Prolog query with a limit argument. It now only fetches needed results.
Rfe10230 - Accept application/x-turtle as a synonym for text/turtle
Some systems are still using the pre-standard MIME type for Turtle documents. The AllegroGraph HTTP server now accepts that as a content-type when uploading triples.
Rfe10220 - Provide an HTTP interface for controlling bulk mode
It used to be only possible to turn on bulk-load mode per load request. This was slow when using many small requests. In a session, it is now possible to persistently turn bulk mode on and off through requests to /session/bulkMode.
Rfe10210 - Don't backtrace on unsupported-db-version error
The HTTP server no longer logs a backtrace and returns a server-error code when trying to open a store with the wrong version. Instead, it returns a PRECONDITION FAILED error.
Rfe10165 - Expose turtle loading over HTTP.
The /statements HTTP service now accepts request bodies in Turtle format, using a Content-Type header of text/turtle. Such files can now also be loaded from WebView.
Rfe10078 - Make HTTP protocol no longer hide inaccessible repositories.
It used to be that HTTP services /repositories and /catalogs hid all repositories and catalogs not accessible by the current user. This was unnecessarily confusing. Now all present objects are returned, with readable and writable flags indicating whether the current user has access to them.
Rfe10009 - Implement Turtle loading
Loading of files in the Turtle (Terse RDF Triple Language) format is now supported.
Rfe9986 - Support Sesame 2.2-style XML transactions
The Sesame project, which is the origin of the XML transaction format our HTTP server supports, have slightly changed the format to be simpler and more flexible. We now support this update, while remaining compatible with the old-style transactions.
Rfe9912 - Remove SharedMemSize parameter from HTTP protocol and HTTP docs
The SharedMemSize parameter was still mentioned in the HTTP protocol docs, and understood by some HTTP services, even though it is no longer used.
Rfe9910 - Reuse sockets for backend-frontend communication
Use HTTP keep-alive to reuse the sockets that the front-end uses to talk to the back-end.
Rfe9838 - Use HTTP keep-alive in the remote Lisp client.
Lisp remote-triple-stores now re-use their sockets when making HTTP requests.
Bug19909 - Inter-server HTTP communication should use local addresses
When the server tried to address itself over HTTP, it would often use the URL provided in the HTTP request that trigerred this communication. If that came from outside the local network, the address in this URL might not be useable. It now uses a local address.
Bug19871 - HTTP server does not unescape \u in resources
In resources passed to the HTTP server either through HTTP parameters, or through JSON data, N-Triples backslash escape sequences were left in the literals, rather than being unescaped. The server now properly handles these.
Bug19846 - Ensure the HTTP client explicitly frees SPARQL cursors
SPARQL queries from the HTTP client did not explicitly discard the cursor they created. This could cause resource exhaustion in situations where many, many queries were operating simultaneously. This is now fixed.
Bug19801 - HTTP character encoding should not depend on system locale
The system's default character encoding was accidentally being used when writing HTTP responses. This worked fine on systems with a UTF-8 locale, but caused the server to return inconsistently encoded responses on other locales.
Bug19661 - HTTP JSON output for query results truncates row on first unbound result.
The code that wrote out SELECT query results in application/json format would stop writing a row at the first null it encountered, leaving that null and any values after it out of the result.
Bug19616 - Allow multiple contexts to be specified in HTTP /size
Contrary to the Sesame docs, we only accepted a single context argument to our /size HTTP service. This has now been fixed -- you can pass multiple context parameters, and the returned value will be the sum of their sizes.
SPARQL
Rfe10207 - Efficiency improvement for some SPARQL graph queries
Improve the way the SPARQL engine handles simple GRAPH clauses with a variable graph argument that does not appear within the rest of the clause. For example:
select ?someGraph ?o
where {
ex:a ex:inGraph ?someGraph .
graph ?g { ex:a a ?o . }
} SPARQL semantics dictate that the query act as if the graph clause is checked for every graph in the dataset. In this case, however, it is clear that the value of ?g cannot alter the behavior of the inner query and that the intent is to determine the graph slots of any of the selected triples.
Note that this optimization is currently very conservative and only occurs when the graph clause is a simple BGP. Future work will extend the behavior to more complex queries.
Rfe10061, Bug19663 - Improve SPARQL negation-as-failure parsing
Nested instances of the SPARQL negation-as-failure idiom were not being parsed correctly leading to incorrect query results. This fix corrects the problem.
Rfe10030 - Share statistical planner between Prolog and SPARQL
Use the foundations of the Prolog select statistical planner in SPARQL queries to reorder BGPs. This improves the speed of many SPARQL queries. Note that this planner requires the POSGI index. If this index is not available, then the coverage planner is used instead.
Because the planner is statistical in nature, it will not always choose the best plan. In general, performance is improved (for example, the 10,000-triple SP2 benchmark queries almost double in speed (from 56-seconds down to 33-seconds) but some queries go slightly slower using the new planner. We will continue to enhance the statistics used and the planner's capabilities in future releases.
If your queries are running more slowly than expected, you can tell SPARQL to use the older coverage-based heuristic planner or tne identity planner (i.e., the planner that does no clause rearranging. Planner selection varies by client. See the client SPARQL documentation for more details.
Bug19809 - fti:matchExpression is sometimes treated like fti:match
There was a bug in the SPARQL code that sometimes caused S-espression-style text queries (as per fti:matchExpression) to be parsed as textual queries (fti:match).
Bug19681 - fix SPARQL problem in queries using both FROM and FROM NAMED
A SPARQL query that used FROM and FROM NAMED together to create the dataset could incorrectly omit some of the the graphs mentioned in the FROM clause. This could cause the query to produce the wrong results.
This problem has been corrected.
Rfe10061, Bug19663 - Improve SPARQL negation-as-failure parsing (repeated from Rfe10061 listing)
Nested instances of the SPARQL negation-as-failure idiom were not being parsed correctly leading to incorrect query results. This fix corrects the problem.
AGWebView
Rfe10300 - Allow optimizing of stores from WebView
There is now an optimize store control in the store overview in WebView. This can be used to optimize a store for querying after a bunch of data has been added.
Rfe10280 - Expose scripting in WebView
There is now a 'Scripts' link in the navigation bar, where one can run JavaScript and (with the right permissions) Lisp scripts interactively, as well as store and edit server-side user script files.
Rfe10277 - Improve help text for WebView part input
The examples given now demonstrate that both backslash-escapes (\n, \uHHHH) and unicode characters are allowed.
Rfe10244 - Add JavaScript server-scripting support
It is now possible to send an /eval request with content-type "text/javascript", and have the body be evaluated as JavaScript. Custom services can also be written in JavaScript.
Rfe10236 - Disable query planner selector in WebView for Prolog queries
To make it clear that the query planner dropdown doesn't apply to Prolog queries, it is now greyed out when selecting a non-SPARQL language.
Rfe10185 - Allow the user to specify a limit for queries in WebView
For heavy queries, the default limit (200) imposed by WebView might be too big. There is now a dropdown above the query input that allows one to choose the amount of results that should be retrieved.
Rfe10181 - Leave WebView widgets in the default browser style
This removes the custom CSS styles for WebView inputboxes and buttons, making the the application look a little more native in modern browsers.
Rfe10155 - Allow a query planner to be selected in WebView
Adds a drop-down menu to the WebView query input page where one can select which planner to use.
Rfe10128 - Rework WebView namespace management
Both the store's default namespaces and the user's namespaces are now managed in the same place. This to reduce confusion---people assumed they could use the namespaces they defined for the store in their queries, without first selecting them.
Rfe10126 - Show PIDs in WebView's process list
Under #processes, the Unix PID of each of the processes is now listed after the process name. The /processes HTTP service now returns tuples instead of just names.
Rfe10116 - Show running sessions in WebView
Once you navigated away, the browser history used to be the only way to get back to a session through WebView. There is now a list of active sessions (if any) in the catalog overview page.
Rfe10039 - Support backup and restore through WebView.
There is now (for superusers) the option to create stores from backups, and to back up existing stores, through the WebView interface. (This is not recommended for large stores.)
Bug19880: Resizing the query input field in WebView is broken
You could drag the resize corner widget, but when you released it nothing happened. This is now fixed.
Bug19868 - Webview tries to write to the installation dir
The server would try to cache gzipped JavaScript files in its installation directory, which fails when it is running under a user that doesn't have write access to that directory. It now uses the settings directory (called the Directory to store data and settings by the configuration program, with default name .../data/) for the cache.
Bug19858 - AGWebView accidentally shows Turtle as export format
We don't currently support exporting to Turtle, so this shouldn't be selectable.
Bug19762 - Fix problem in process page in WebView
A bug had snuck into the WebView code that prevented the #processes page from coming up when in a catalog view.
Bug19741 - Add example to context input field help in AGWebView import
It used to say only that it accepted 'N-Triples' syntax.
Bug19523 - Fix several bugs in WebView's query result display.
Prolog queries returning the default graph now display properly. Prolog queries returning non-flat-lists (such as (?x (?y ?z))) now produce an error message rather than no feedback at all. Literals ending in a backslash are now parsed properly, rather than treated as invalid.
Changes to the Lisp API
Rfe10296 - Check minimum build id on client startup.
The AllegroGraph lisp client will now check if the lisp has all the required runtime patches installed, and will present an error and resolution options if that is not the case.
Rfe7148 - Optimize away keyword scanning in get-triples
db.agraph:get-triples now uses a compiler macro to optimize the calls so no runtime CPU time is spent keyword scanning.
In Lisp if you want to trace get-triples calls, you should declare the function notinline. Alternatively, you can trace db.agraph::get-triples-positional, the non-exported internal function, which the compiler macro uses.
Bug19890 - load-rdf/xml-from-string (Lisp API) could cause memory error
If called with a very long string and no explicit base-uri, load-rdf/xml-from-string tried to create a base-uri by querying the file system with the string. In some situations, this could cause a buffer overflow which would lead to abnormal client termination. This commit fixes the base-uri resolution algorithm so that it handles strings without querying the file system. The underlying buffer problem has also been corrected.
Bug19678 - serialize-rdf-n3 fails with encoded UPIs
The Lisp API function serialize-rdf-n3 would fail if it was asked to serialize triples with any encoded UPIs. This is now corrected.
Bug19069 - improve error handling of N-Triples parser
The continue-on-errorp function to load-ntriples can now specify how to behave when an error occurs more precisely. It can return one of
- :continue (the default)
- to ignore an error and continue
- :stop-parsing
- to ignore the erorr and stop parsing the rest of the current source
- :error
- to signal the error in any case
Documentation
Rfe9657 - server-installation.html document changed to describe non-interactive running of configure-allegrograph.
Python Client
Rfe10221 - BulkMode support added to Python client
Bulk Mode is a repository-wide setting that affects all connected clients.
Use the bulk_mode property on a repository object to set and get the bulk_mode state. Here are some examples.
repository.bulk_mode = True print "The bulk mode state is ", repository.bulk_mode
If you only have a connection object, you can access the property with conn.repository.bulk_mode.
Any commits while repository.bulk_mode is true are done so using BulkMode on the server.
Java Client
Rfe10303 - Support revised bulk-load mode interface
With this change, the Java client now supports the revised bulk-load mode interface. Bulk mode is now persistent (sticks even after closing and reopening the store), and is controlled with the AGRepository#setBulkMode and AGRepository#isBulkMode methods, rather than per session.
Bulk-load mode can now only be controlled through a store-wide toggle (/bulkMode over HTTP), since it is a store-wide setting, and turning it on and off a lot is very slow.
Rfe10197 - Support for encodable-ids and generated URIs
With this change, a namespace (viewed as a set of URIs) can now be registered as encodable. URIs in the namespace have local parts that follow a declared format, allowing for efficient encoding and in some cases generation of unique URIs in the namespace. Once registered, a conformant URI in the namespace is auto-encoded when added to the store, resulting in a smaller string table and faster query speed. (This is a Java client version of the encoded-ids feature, see Encoded IDs for more information.)
Rfe10189 - Java Interface for Stored Procedures
The stored proc feature and API are experimental, and subject to change in a future release. For this reason, the new methods are marked as deprecated.
The AllegroGraph server defines a new API for defining Stored Procedures and they are installed like Custom Services.
The primary API is:
AGRepositoryConnection.callStoredProc(functionName, moduleName, args) A low-level API is also exposed:
AGHttpRepoClient.callStoredProcEncoded()
AGSerializer.serializeAndEncode()
AGDeserializer.decodeAndDeserialize()
Rfe10119 - Add analyze() method to provide query analysis
Sesame AGQuery subclasses can now call the analyze() method to obtain a query analysis for the query rather than evaluating it.
Rfe9971 - JAVA API for HTTP protocol checkVariables
Related to bug19157: SPARQL error if ORDER variable isn't in query
Add method setCheckVariables(boolean) to both query classes: com.franz.agraph.jena.AGQuery com.franz.agraph.repository.AGQuery
Rfe9858 - Support ResultSet nextSolution() method
Previously, the legacy ResultSet API method nextSolution() would throw an UnsupportedOperationException; with this change the method is now supported.
Rfe9846 - JENA graph unions and graph-scoped reasoning
With this change, the method AGGraphMaker#createUnion can be used to create a union of named graphs. AGInfModels can now be created over models backed by a union graph, and reasoning will be properly scoped to just the union graph; this change allows sound reasoning over a proper subset of data in the store.
Rfe9534 - Add source jar to the client download
With this change, an agraph-[version]-src.jar file is now included in agraph-[version]-client-java.tar.gz downloads. The download's Eclipse project's .classpath is modified to attach the source jar to the agraph-[version].jar to facilitate browsing and debugging.
AllegroGraph 4.1.1
Welcome to AllegroGraph 4.1.1. This release of AllegroGraph provides overall product stability and efficiency improvements, with special attention to improvements in AGWebView and documentation.
Server Changes
Rfe9796 - Make tlog-write-checkpoint-indicator use :allocate-temporary
The checkpoint indicator file, which records the location of the last checkpoint record that has been written to the transaction log file, is now written using the new internal API for creating temporary files. This ensures atomic updates on disk.
Rfe8994 - Change behavior when a filesystem is full
A change has been made to the AllegroGraph server that makes the database server block if one of the file systems is found to be full during a write operation. Operations automatically continue once space has been made available by the system administrator.
Rfe9319 - Freetext index configuration and search support
With this change a free text index can be created by specifying its configuration, and free text performed can be performed w.r.t particular freetext indices.
Rfe9534 - Add sources for jena, arq, and json jars
Previously the source code for these 3rd party libs had to be obtained separately; this change includes them as a debugging convenience in the git repository, and sets up the .classpath to reference them in Eclipse.
Rfe9969 - Update clojure api
AGraph Clojure API updated for agraph v4, clojure 1.2. Build support for leiningen 1.3.1, as well as ant.
Clojure and Java source are in the Git repository: http://github.com/franzinc/agraph-java-client Added README.md for use and development instructions.
Clojure and Java jars are available in the Clojars Maven repository: http://clojars.org/groups/com.franz agraph-java-client and openrdf-sesame-onejar When AGraph v4.1 is released, these will be updated and agraph-clj-4.1.jar added.
Changed license to EPL, same as the java code.
Removed agclj.sh in preference for 'lein repl' or 'lein swank'.
Rfe9999 - Auto-adjust MinMergeChunks
If MergeThreshold is configured to a value lower than MinMergeChunks, the value of latter is now automatically adjusted to the value of the former. This ensures that a merge will actually be done when the threshold is reached.
Rfe10119 - Add analyze() method to provide query analysis
Sesame AGQuery subclasses can now call the analyze() method to obtain a query analysis for the query rather than evaluating it.
Bug19658 - Agload needs bin directory symlink
The symbolic link for the agload program was not being created in the 'bin' directory by install-agraph and the RPM installation. This has been fixed.
Bug19674 - After-the-fact index creation should mind the amount of files it opens
Fixed a bug that caused "too many open files" error and terminating the server when large number of chunks had to be merged.
Bug19698 - Changes to prepareGraphQuery and prepareBooleanQuery
prepareGraphQuery and prepareBooleanQuery now also accept the string "SPARQL" or "PROLOG" as a query language parameter. Previously, only prepareTupleQuery allowed the string values as a convenience in place of the QueryLanguage.SPARQL or QueryLanguage.PROLOG objects.
Bug19720 - Fix "don't attach as root" check
The check which ensures that you don't attach to a triple store as "root" was checking the real uid, not the effective uid, preventing it from working properly under su/sudo. This has been fixed.
Bug19739 - Compressed index files not unmapped when closed
Triple index files which were mapped during queries were not being properly unmapped when they became obsolete. This problem has been fixed.
HTTP Client
Rfe9756 - Return 400, not 500, when trying to delete an open store over HTTP.
The HTTP server was returning (and logging) an internal server error when it tried to delete a store that was still being held open (through a DELETE or PUT request to the store's URL). It now returns a 400 (bad request) response.
Rfe9971 - Java api for http protocol checkVariables
Related to bug19157: SPARQL error if ORDER variable isn't in query
Added method setCheckVariables(boolean) to both query classes: com.franz.agraph.jena.AGQuery and com.franz.agraph.repository.AGQuery.
Rfe10120 - Make sure custom-service exists in agraph4.fasl
This adds the HTTP backend code to the agraph fasl, so that it is possible to compile custom services in an image that has this fasl loaded. AllegroGraph 5.0 note: the file is agraph5.fasl starting in version 5.0.
SPARQL
Bug19157 - SPARQL error if ORDER variable isn't in query
When the order by clause has a var not known from the query, ignore it (the default), or identify it in an error message (when sparql-check-variable-errors-p or checkVariables in the http protocol is true).
Bug19618 - Properly escape strings in SPARQL/JSON output.
Newlines and other special characters are now escaped according to JSON rules in SPARQL/JSON results returned by the SPARQL engine.
Bug19619 - Fix typed literal writing in SPARQL/JSON output.
The SPARQL engine was not writing out typed literals in SPARQL/JSON documents properly. Those in UPI form got a wrong type property ("literal" rather than "literal-typed"), those in future-part form were written out as XML.
AGWebView
Rfe9492 - Add export to AGWebview
Adds the possibility to download repositories and query result sets through AGWebView. Several output formats are supported, depending on whether triple data (n-triples, n-quads, rdf/xml, trix) or select results (sparql+xml, csv) is downloaded.
Rfe9587 - Show progress indication in WebView uploads
Rather than just seeing a spinner animation, users doing an upload in WebView now see how much data has been uploaded already.
Rfe9898 - Add drop-down menus to WebView's navigation bar.
To make it possible to link more functionality from the navigation bar without running out of space, groups of links are now put into drop-down menus.
Rfe9900 - Optimize loading of WebView scripts and style-sheets.
Serves the JavaScript and CSS files used by WebView in concatenated form, allowing gzipped responses. This causes the base webview page to go from 25 files totalling over a quarter megabyte to 2 files and 68 kilobyte.
Rfe9950 - Provide a way to start exploring a store in WebView
This adds a paragraph to a repository overview containing links to some simple queries that can help the user explore the store.
Rfe9951 - In WebView user management, show drop-downs for stores.
When granting permissions on a store, the user no longer has to type the names, but can select them from a drop-down menu.
Rfe9964 - Ask confirmation before letting user un-superuser himself in WebView
Once your user loses its superuser flag, it can no longer manage users, so one wrong click here could lock yourself out. There is now a confirmation dialog to ensure you intended to do this.
Rfe9990 - Do not display \uHHHH character escapes in WebView
For non-ASCII characters, WebView showed the N-triples escape sequence instead of the character itself, even when 'short' part display was turned on. This is now fixed.
Rfe10025 - More robust recent-query saving in WebView
On modern browsers, you'll now be able to list your recent queries even after navigating away from WebView and back. On top of that, the list of recent queries is no longer tied to a specific repository.
Rfe10026 - WebView now shows a spinning animation when loading a file
Before, there was no proper feedback indicating that something was happening during a dataset upload.
Rfe10027 - In WebView's graph view, fix nodes after dragging
Nodes can be rearranged by dragging in the graph view. A node now stays fixed after having been dragged, making it much easier to lay out the graph.
Rfe10050 - Make specifying a graph when adding from WebView easier.
The graph input field in the add-triple dialog is now always visible. WebView will try to retrieve a list of graphs that are used in this store, and allow you to choose them from a drop-down.
Rfe10051 - Allow a graph/context to be specified when importing in WebView.
This extends the import file and import URL dialogs with a context field, which is used to specify a graph to import into.
Rfe10052 - Add a general delete-triples UI to WebView
This adds a 'delete statments' control to the repository overview page, from which you can delete triples, by wildcard. Also includes the existing-graph select control from rfe10050, to make it easy to drop a graph.
Rfe10127 - Allow init files to be loaded in sessions created from WebView
There is now a checkbox 'load initfile' in the widget that creates new sessions from a catalog view in WebView.
Bug19581 - Newly registered users in WebView don't get enough access
When user-registration is turned on, new users were supposed to get the same access permissions as the anonymous user, but due to a bug they did not get any access. This is now fixed.
Bug19536 - In WebView's graph view, clicking on a node was broken.
Clicking on a node in the graph did not actually take you to that node's overview page, but raised a script error instead. This is now fixed.
Bug19552 - Can't start a session from catalog page in WebView
There was a bug that would prevent users from starting a session from a (non-root) catalog page in WebView. This has been fixed.
Documentation
Rfe9574 - Lisp client documentation
The AllegroGraph Lisp Client Quick-Start guide has been expanded to include explicit Lisp Client set-up instructions and an explanation of local vs. remote triple-store connections.
Rfe9750 - Improve document navigation
The navigation panel (to the right side of most documentation pages) and the document map page have been revised and extended to provide more organized and thorough access to the many documents and web pages that describe AllegroGraph.
Rfe9883 - Javadoc included in the distribution
Javadocs for com.franz.agraph are in the javadoc folder in the distribution file, and they link to external javadocs for java, jena, and sesame.
Rfe9929 - User manual for 4.1.1 AGWebView
The AllegroGraph documentation has been extended to include a User's Guide for WebView. This document presents task-oriented guidance on using the WebView interface for managing users and repositories.
Rfe9936 - Document TopBraid Composer plugin
In this release the TopBraid Composer (TBC) plugin installation notes got an upgrade with numerous screen-shot illustrations and further notes on how to integrate TBC with the AllegroGraph Virtual Machine.
Rfe10046 - AllegroGraph Text Indices
There is a new, high-level document describing AllegroGraph's free-text indexing features.
Rfe10077 - Add user-selected triple indices to tutorials
The Python, Java, and Jena tutorials have an augmented example1() showing how to add and remove triple indices for greater efficiency.
Rfe10108 - AllegroGraph Triple Indices
There is a new, high-level document describing AllegroGraph's system of triple indices and how to manage them.
Rfe10139 - Emphasize "agraph" user vs. AG superuser
The server-installation document now emphasizes the differences between the "agraph" operating system account and the AllegroGraph superuser account. Users sometimes confused the two accounts, which caused connection issues.
Rfe10159 - Rapper documentation
There is a new document describing the use of the Rapper command-line utility for pre-processing incoming data files to make them load more quickly in AllegroGraph.
Bug19620 - Fix typos in documentation of create-freetext-index
The description of the index-fields argument was a jumble of a sentence.
AllegroGraph 4.1
Welcome to AllegroGraph 4.1. This release of AllegroGraph provides overall product stability and efficiency improvements, with special attention to backup/restore, point-in-time recovery, and warm standby features.
Rfe9535 - Use managed files throughout the storage layer
The storage layer has been changed so that all files are allocated through the central resource manager mechanism. This makes all files eligible for placement through FilePlacementRule parameters in the database configuration. Previously, some files could not be placed using such rules.
Rfe9772 - More advanced server scripting
This introduces server-side script files, which can be loaded into sessions, and a convenient way to define custom HTTP services from such files. See the HTTP protocol documentation for details.
Rfe9784 - Keep track of which pages of deleted bitmap have been touched
In order to speed up backup operations, the AllegroGraph server now keeps track of which pages in the deleted bitmap file are used.
Rfe9785 - Revise backup and restore
AllegroGraph now uses a new, single-file archive format for storing backups. The new backup and restore functionality is described in the server documentation.
Rfe9789 - Support callback parameter when writing JSON output
Requests returning JSON data can now specify a callback=[x] HTTP parameter, which will cause the response to be wrapped in [x]([response]), so that the resulting document can be loaded as a JavaScript program (which is useful for working around cross-domain restrictions in browsers).
Rfe9798 - Allow relative directories in config files
In the AllegroGraph server configuration file, relative pathnames can now be specified for file and directory parameters. Such pathnames are considered relative to the directory where the server configuration file is located.
Rfe9805 - Point in time recovery
The point in time recovery facilty allows the a database backup to be restored to its state at a particular point in time.
Rfe9835 - Use stream API to allocate blocks in sparse deleted bitmap
The database server code has been updated to explicitly allocate blocks in the deleted bitmap file using file system calls. That way, full filesystem conditions will be handled in a graceful manner.
Rfe9840 - Improve "disk full" behavior in agraph-backup
When agraph-backup detects a "disk full" condition, it now performs an interactive recovery if it was started from a terminal. When started non-interatively (i.e. from cron), no recovery is performed and an error message is generated.
The exit code of agraph-backup is now always 0 if the operation succeeded, 1 if it failed.
Progress reporting in agraph-backup has been improved.
Rfe9842 - Agraph-backup: merge parameters.dat while restoring
When restoring a database from a backup, the parameters established in the parameters.dat file stored in the backup database are copied to the restored database. File placement related parameters are skipped in this process.
Rfe9888 - Do not log backtrace for HTTP proxy pipe errors.
When a back-end crashed, this often caused two back-traces to be logged, one for the back-end and one for the front-end. The second one was just noise, so it is now suppressed.
Rfe9889 - Allow resource manager to forget a file
A new internal API function has been added to the resource manager library to support the archival of transaction log files.
Rfe10055 - Properly expose the no-commit flag
The no-commit mode, used primarily by the warm standby replication process, is now something that can be explicitly turned on and off.
Rfe10056 - Require authentication for warm-standby connections.
The socket on which a warm-standby server listens used to be accessible by anyone. It now requires a valid username and password when initiating a connection.
Bug19391 - Agraph-install no longer outputs paths with double slashes
The helper script agraph-install would output paths with double slashes when given a path with a trailing slash as argument. This was mostly harmless, and is now fixed.
Bug19420 - Chunk writers leak disk space when db crashes
A problem has been corrected that could lead to stray files being left on disk when a database was brought down uncleanly.
Bug19446 - Catalog parsing problem - confusing error message
The error message that the configuration file parser generates when overlapping directories are specified in the configuration file has been improved.
Bug19487 - Solve ':void external-format' errors in logs.
The AG log files occasionally showed unexpected errors of 'Attempt to translate characters using a :void external-format'. These were harmless, but they are now handled properly, and no longer logged.
Bug19526 - Back up tlog file only up to backup checkpoint
The backup code has been changed so that transactions which executed while the backup was running are not included in the backup.
Bug19684 - Include the triple-id index in the standard indices
If specific indices are not requested when a store is created, then it will have the standard set of seven: :ospgi, :posgi, :spogi, :gospi, :gposi, :gspoi, and :i.
Prior to this change, the :i index was not included in this standard set.
Lisp Client
Bug19442 - Make sure remote-triple-store accepts :server argument.
The :server argument to the remote-triple-store class was being incorrectly rejected, making it impossible to open stores on other machines.
SPARQL
Bug19474 - Improve SPARQL describe queries
excess variables are no longer included in the output of DESCRIBE
CBD (Concise Bounded Description) logic is now implemented so that we correctly handle blank nodes
Many general efficiency improvements
Bug19499 - Fix SPARQL XML and JSON output of geospatial values
Retrieving SPARQL SELECT results containing geospatial values as SPARQL+XML or SPARQL+JSON was broken because an internals change hadn't been propagated to the SPARQL engine. This is now fixed.
AGWebView
Rfe9815 - Fix accept-encoding issue for WebView in Internet Explorer 7
This fixes a problem where Internet Explorer 7 would display "Internet Explorer cannot display the webpage" when opening WebView (or any other page served by AG). Internet Explorer fails to handle deflated content, so we now serve it gzipped responses instead.
Rfe9899 - Fix typo and wrong word choice in WebView.
When adding a notable node, 'Desription' was corrected to 'Description'. When looking at a text index that indexes all predicates, it used to say 'None' under predicates, now it says 'All'.
Bug19500 - Fix problem with displaying geospatial results in WebView
When a query result contained geospatial data, a bug in WebView was triggered that caused the result to fail to display. This bug has been fixed.
Bug19525 - Fix failure when opening a federated session in WebView.
Trying to create a federated session in WebView resulted in a PRECONDITION FAILED error. This was caused by a bug in the way session data was being returned by the server, causing WebView to believe it was not on a federated store. The problem has been fixed.
Documentation
Spr37148 - Question about connecting using java
Two lines of code were inconsistent between the Java tutorial HTML file and the tutorial examples code. The tutorial has been edited to correct this issue.
Rfe9783 - AllegroGraph's isolation model
The AllegroGraph Overview document has been augmented with a new section on AllegroGraph's ACID-compliant isolation model.
Rfe9806 - Enhance SPARQL examples in tutorials
In the Python, Java and Jena tutorials, example13() now uses the SPARQL "contruct" to create has-grandchild triples, and then "describes" a grandparent and a grandchild.
Rfe9821 - Make SPARQL examples more robust
Example10() of the Python/Java tutorials has been extended to show examples of SPARQL's FROM, FROM DEFAULT, and FROM NAMED operators, and an example of the UNION operator.
Rfe9831 - Python/Java tutorials, illustrate use of conn.size() with contexts.
Example10() of the Python and Java tutorials has been extended to show examples of the conn.size() method returning the number of triples found in specific subgraphs.
Rfe9915 - Java and Python tutorials demonstrate SPARQL and RDFS++
Example19() of the Java and Python tutorials has been augmented to demonstrate how to use a SPARQL query with RDFS++ reasoning.
Rfe10104 - Point-in-time recovery document
There is a new document describing AllegroGraph's point-in-time recovery feature.
Rfe10105 - Replication and warm-standby document
A new document on triple store replication and warm stand-by has been added to the document set.
Rfe10106 - Update backup/restore document
The backup/restore document has been updated to reflect the new features available in AllegroGraph 4.1.
Rfe10107 - Transaction Log Archiving document
Transaction logs increase without limit unless archived. A new document describes the transaction log archiving feature of AllegroGraph.
Bug19454 - Python tutorial explains how "int" becomes "long"
In example15() of the Python tutorial we show registerDatatypeMapping() requesting a nativetype "int" and AllegroGraph responding with XMLSchema.LONG. This is correct, and the text now explains why.
Document MinimumCheckpointInterval and MaxRecoveryTime parameters
The meaning of these parameters have been documented in the daemon configuration documentation.
AllegroGraph 4.0.6a
Welcome to AllegroGraph 4.0.6a. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9837 - Avoid log warning when opening an existing repo
Added method AGCatalog.hasRepository(id) which tests for existance. Added AGCatalog.openRepository(id) which will open but not create. Added AGCatalog.createRepository(id,strict) where the strict option will throw an exception if the repository already exists.
Rfe9980 - getCatalog now works with the root catalog
Added AGCatalog.isRootId(id) to identify id's for the root catalog. Enhanced AGServer.getCatalog(id) to accept root catalog identifiers. Also added AGServer.getCatalog() to return the root catalog.
Rfe10041 - Implement owl:oneOf
AllegroGraph's RDFS++ reasoner now includes the following inference:
if the class X is owl:oneOf A, B and C
then each of A, B and C are of type X. Rfe10075 - Implement query index use analyzer, version 1
If analyzeIndicesUsed=true is added to a SPARQL HTTP query string, then AllegroGraph will analyze the query to determine which indices it will use. The results are returned as plain text and written to the AllegroGraph log. They will look like
(desired spogi optimal 1 actual spogi optimal 1)
(desired posgi optimal 6 actual ospgi suboptimal 4) Where each line refers to one of the indices required by the query. The line indicates which index flavor the query really wants and which index the query used based on what is available to the store. The additional information indicates whether or not the index choice was optimal, suboptimal or required a full scan. The numbers indicate whether or not additional filtering was required and can be ignored for now.
Bug19650 - Fix problem federating repositories from the root catalog
Previously an exception was thrown when attempting to federate repos from the root catalog; this change addresses that.
AllegroGraph 4.0.6
Welcome to AllegroGraph 4.0.6. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9535 - Use managed files throughout the storage layer
The storage layer has been changed so that all files are allocated through the central resource manager mechanism. This makes all files eligible for placement through FilePlacementRule parameters in the database configuration. Previously, some files could not be placed using such rules.
Rfe9620 - Increase limit on number of index chunks
Previously, the maximum number of index chunks was based on the limit on the number of file descriptors (RLIMITNOFILE) which, in a default Linux configuration, resulted in a limit of 100 chunks. Now the maximum is based on the limit on the number of memory mappings (determined by reading /proc/sys/vm/maxmap_count), which usually results in a much higher limit.
If the limit on the number of index chunks is reached, further commit activity will be blocked. Therefore this higher chunk limit will reduce the likelihood of blocked commits.
Rfe9731 - Change transaction log aligment calculation
An internal change was made to the way the alignment for transaction log records is calculated so that opening transaction log files can be defered until records are written.
Rfe9762 - Add someValuesFrom and allValuesFrom restriction reasoning
AllegroGraph's hasValue reasoning has been renamed restriction-reasoning and now covers owl:hasValue, owl:someValuesFrom and owl:allValuesFrom. In particular, AllegroGraph will now infer:
C owl:onProperty P
C owl:hasValue v
a rdf:type C
==> a P v
C owl:onProperty P
C owl:hasValue c
a P c
==> a rdf:type C
C owl:onProperty P
C owl:someValuesFrom D
a P b
a P x
b rdf:type D
(and will not infer x rdf:type D)
==> a rdf:type C
C owl:onProperty P
C owl:allValuesFrom D
a type C
b P x
==> x type D Because the reasoner now implements more than owl:hasValue, it has been renamed the restriction-reasoner. The accessor function has-value-reasoning-enabled has been deprecated and the new accessor is restriction-reasoning-enabled.
The HTTP protocol changes to use restriction instead of hasvalue as the name of the reasoner that supports hasValue reasoning (and the other restrictions).
Rfe9769 - Log merge statistics
In addition to logging how long a merge took, the following additional information is logged: Number of triples read, number of triples written, number of deleted triples expunged, and number of duplicate triples dropped.
Rfe9772 - More advanced server scripting
This introduces server-side script files, which can be loaded into sessions, and a convenient way to define custom HTTP services from such files. See the HTTP protocol documentation for details.
Rfe9776 - Standardize blank node serialization
Use the same blank node syntax in all of AllegroGraph's serializers (N-Triples, RDF/XML, TriX, etc.).
Rfe9789 - Support callback parameter when writing JSON output
Requests returning JSON data can now specify a callback=[x] HTTP parameter, which will cause the response to be wrapped in [x]([response]), so that the resulting document can be loaded as a JavaScript program (which is useful for working around cross-domain restrictions in browsers).
Rfe9803 - Bulk loading support
AllegroGraph now supports a bulk loading mode that allows changing the triple-store without updating the transaction log. Both the HTTP statements service (used to load data into a store) and the agload bulk loader have been enhanced to use this new feature.
Use the bulkMode parameter to specify that you want bulk loading on in the HTTP service
Use the --bulk option to specify that you want bulk loading on when using agload.
Depending on hardware and the actual data, loading with the transaction log turned off is between 10 and 33% faster than loading with it on.
Rfe9812 - Add client API for triple-store index manipulation
Add the following new functions to the Lisp API:
- add-index - add a new index to a store
- drop-index - remove an index from a store
- triple-store-indices - return a list of current indices for a store
- valid-index-flavors - return a list of all valid index flavors
Note that changes made by 'add-index' and 'drop-index' are not made until 'commit-triple-store' is called.
Rfe9889 - Allow resource manager to forget a file
A new internal API function has been added to the resource manager library to support the archival of transaction log files.
Rfe9907 - Add with-indices support to agload
agload now supports custom index selection using the --with-indices command-line option.
Rfe9913 - Improve single-float printing
AllegroGraph could lose precision when printing single float values. For example, 0.0018333651 would print as 1.833365E-3 rather that 1.8333651E-3. This has been corrected.
Rfe9930 - Support adding, dropping, listing repository indices
Added methods addIndex, dropIndex, listIndices, and listValidIndices to AGRepositoryConnection to support managing repository indices.
Rfe9953 - Share transaction log buffer across attachees
Each process (backend) that attached to a store used to allocate its own in-memory buffer for writing to the transaction log.
A single buffer is now allocated in shared memory and used across all backends, saving valuable RAM.
Rfe9966 - Front-end server-socket backlog parameter is too low
The kernel will only save a certain amount of pending connection attempts. If the server doesn't accept them quickly enough, they get discarded. The default backlog size was too small. It is now set to use the maximum the operating system allows, making the chance of discarded connections much, much lower.
Rfe9970 - Upgrade path for configurable indices
Code has been added to convert pre-v4.0.6 databases to be useable with v4.0.6 in terms of configurable indices.
Rfe9972 - Updates to database upgrade code
The internal API for database upgrades has been improved to make more information about the upgraded database available to the code that performs (part of) the upgrade.
Rfe9992 - Backup/restore improvements
agraph-backup now accepts a --catalog command line parameter. Also, agraph-backup now only backs up the used portions of the string table files, resulting in faster backup and restore times.
Rfe10012 - Consistency checker for sstab
A utility has been added to the AllegroGraph distribution that validates the contents of the on-disk string table data structures.
To validate the integrity of a database, the new "checkdb" utility should be used.
Rfe10019 - Setup automatic datatype-mapping for triple-ids
There is now an automatic datatype mapping between triple-id UPIs and the datatype <http://franz.com/ns/allegrograph/4.0/triple-id>. This means that these UPIs will be be read and written in this format. For example, the triple
ex:test rdf:refersTo "18"^^<http://franz.com/ns/allegrograph/4.0/triple-id> . will be imported such that its object is an encoded UPI that refers to triple ID 18 using the AllegroGraph specific triple-ID reification.
Bug19171 - Importing RDF files with long text literals is much faster.
AllegroGraph now uses a string output stream to collect characters in text literals instead of string+'ing them together. This speeds up importing long literals with many XML escapes. Files that previously took 4 hours to import now take ~2 seconds.
Bug19181 - Correct newline parsing for n-triple imports
Previously, the AllegroGraph n-triple format parser would count newline and carriage return characters as white space, allowing incorrectly-formatted triples to be imported. Now AllegroGraph allows a triple to be specified on one line only.
Due to this change, the AllegroGraph n-triple parser now reports the correct line number when a parse error occurs. This line number can now be used in a text editor to jump to the location of the problem.
Bug19253 - Accept resources for :base-uri in load-rdf/xml.
Previously, the RDF parser disregarded resources used as :base-uri in the RDF import. Now, it accepts all resources and other future-parts and uses their value to determine the base URI for the import.
Bug19266 - Correct cursor re-use problem
AllegroGraph was aggressively re-using storage layer cursors which could lead clients to point at cursors that were no longer valid. This could lead to incorrect query results.
Bug19391 - Agraph-install no longer outputs paths with double slashes
The helper script agraph-install would output paths with double slashes when given a path with a trailing slash as argument. This was mostly harmless, and is now fixed.
Bug19407 - Allow the direct Lisp client to log to the server log file
The direct client (which is used internally in the HTTP server) can now write to the server log file directly.
Bug19420 - Chunk writers leak disk space when db crashes
A problem has been corrected that could lead to stray files being left on disk when a database was brought down uncleanly.
Bug19442 - Make sure remote-triple-store accepts :server argument.
The :server argument to the remote-triple-store class was being incorrectly rejected, making it impossible to open stores on other machines.
Bug19446 - Catalog parsing problem - confusing error message
The error message that the configuration file parser generates when overlapping directories are specified in the configuration file has been improved.
Bug19460 - Issue installing agraph 4.0.5 over 4.0.4
A bug has been fixed that resulted in errors when trying to open a database that was created with AllegroGraph v4.0.4. This problem has been resolved.
Bug19569 - Java client now helps delete orphaned saved queries
Preparing a Sesame AGQuery currently has the side effect of saving the prepared query on the server in memory for the duration of the session. A client app that prepares many queries during a session thus uses up server memory. If an AGQuery instance goes out of scope, the saved query is no longer needed on the server and is currently only deleted when the session expires. With this change, the client library no longer uses the saved query service during a prepare query, pending further cost-benefit analysis of the saved query service. Client code can explicitly request that a query be saved using an experimental AGQuery.setSaveName() method. The client library also tries to help delete orphaned saved queries sooner by tracking when AGQuery instances are garbage collected on the client and issuing delete requests for the orphaned saved queries in subsequent query requests.
Bug19474 - Improve SPARQL describe queries
Excess variables are no longer included in the output of DESCRIBE
CBD (Concise Bounded Description) logic is now implemented so that we correctly handle blank nodes
Many general efficiency improvements
Bug19487 - Solve ':void external-format' errors in logs.
The AG log files occasionally showed unexpected errors of 'Attempt to translate characters using a :void external-format'. These were harmless, but they are now handled properly, and no longer logged.
Bug19489 - Directory where pidfile resides is chown'd to agraph user
In prior releases, when running as root, the agraph-control script would chown the directory where the pidfile resides to the agraph user. This was okay when the recommended pidfile path of /var/run/agraph/agraph.pid was used, but if another path that was shared with non-agraph files was used, the chown was undesirable. This problem has been fixed.
Bug19497 - Fix convertdb
As AllegroGraph v4.0.6 changes the on-disk database format, existing databases need to be converted. The conversion process is performed by the "convertdb" utility, which has been brought up to date with respect to recent storage layer changes.
Bug19499 - Fix SPARQL XML and JSON output of geospatial values
Retrieving SPARQL SELECT results containing geospatial values as SPARQL+XML or SPARQL+JSON was broken because an internals change hadn't been propagated to the SPARQL engine. This is now fixed.
Bug19509 - Agload could load inconsistently on virtual machines
Some virtual machines could reveal a fault in agload that caused inconsistent triple loading. This could result in agload reporting a negative triples-per-second load rate and in different amounts of triples being loaded on different runs. This patch corrects the problem.
Bug19539 - Support dynamic catalogs
Support for dynamic catalogs as documented in the AllegroGraph manual has been restored.
Bug19548 - Correctly import default namespaces in rdf/xml literals.
Previously, when importing XML literals that had a default namespace set, the RDF/XML import in AllegroGraph would silently drop the default namespace. Now, the correct xmlns clause is set for these xml literals.
Bug19551 - Fix handling of blank nodes in RDF/XML parser.
Previously, the RDF/XML parser failed to import empty xml elements like <foo x="y"/> by creating references to empty string literals instead of blank nodes like the RDF/XML syntax specification mandates. Now, the RDF/XML parser correctly creates blank nodes.
Bug19560 - Provide sstab upsizing upgrade path
v4.0.5b increased the sizes of the string table files without changing the database version number. Databases created with v4.0.5a or earlier must be upgraded using the convertdb utility to have this problem corrected.
Bug19562 - Improve error message when AllegroGraph loaded on an unsupported platform
AllegroGraph is currently only supported on linux-64 but the error message produced when trying to load agraph4.fasl on a non-supported platform was nonsensical. This patch improves the error message to make it clear what has gone wrong and what remedies to take. AllegroGraph 5.0 note: the module to require is :agraph5 and the file is agraph5.fasl starting in version 5.0.
Bug19565 - Crash can occur when file descriptor limit is large
When running as a user with a large file descriptor limit (larger than 1024), it was possible for AllegroGraph to crash under certain circumstances. This problem has been corrected.
Bug19567 - Log handles remain open after SIGHUP
Due to a mistake in the way the daemon detached from the terminal, the initial log file would never be closed, even when a HUP signal was sent to the daemon. This is now fixed.
Bug19595 - Repair createGraph() for anonymous graphs
Previously createGraph() in the no-arg case was trying to use an illegal blank node id based on a Jena-internal AnonID; this fixes that problem.
Bug19602 - String table fails to recover short strings from transaction log
An error has been corrected that could have resulted in the loss of string table data if a database was uncleanly shut down.
Bug19614 - Bus error during prolog query
Some queries that returned AllegroGraph specific UPI types (e.g., for reified triple-ids or default-graphs) could cause a bus error rather than signaling the correct message that these types cannot be serialized using RDF/XML. Corrected.
Bug19632 - N-Quads serializer was emitting the default-graph incorrectly
The N-Quads serializer was needlessly including the graph even for triples in the default-graph of the triple-store. Furthermore, it was emitting these graphs using invalid syntax. Both of these issues are fixed.
HTTP Interface
Rfe9885 - Fix HTTP connection keep-alive for 204 responses.
HTTP requests that caused a 204 (no content) response incorrectly had their connection closed by the server, even when they requested keep-alive. This meant the client had to create a new socket for their next request, causing unnecessary overhead.
Rfe9888 - Do not log backtrace for HTTP proxy pipe errors.
When a back-end crashed, this often caused two back-traces to be logged, one for the back-end and one for the front-end. The second one was just noise, so it is now suppressed.
Rfe9906 - Support index inspection and manipulation over HTTP.
Under /indices, a triple store's indices can now be inspected, created, and dropped over HTTP. Adds an index argument to store-creation requests, to set the indices at creation time.
Bug19506 - Make HTTP server more cautious about closing stores
It was possible for the HTTP server to close a triple store that it was still reading from. This would cause the reading request to fail. The server now waits until pending requests have finished when closing one of its stores.
Documentation
Rfe9936 - User document for Top Braid Composer and AG
Added an installation page for the AllegroGraph TopBraid Composer plugin.
Lisp Client
Bug19452 - Race condition in lisp client startup
While connecting to a database, the Lisp client could sometimes crash with the following error message:
The value of db.agraph.spawn::hub is nil,
which is not of type db.agraph.spawn::hub A fix has been developed to prevent that problem from occuring.
Python Client
Rfe9906 - Add support for index changes to the Python client
A connection object in the Python client now has listIndices, addIndex, and dropIndex methods for inspecting and manipulating the indices on a triple store.
Rfe9965 - Python miniclient now supports retry on ECONNRESET
The python miniclient now retries on connection failures with errno ECONNRESET. Previously it would just fail the request.
Java Client
Rfe9845 - Support owl:Restriction reasoning in the Java client
The Java client now supports hasValue, someValuesFrom, and allValuesFrom reasoning. In the Sesame AGQuery API one can now specify which entailment regime to use when including inferences in query results; the new Restriction regime is now available as an option, while RDFS++ remains the default. Similarly, using the Jena AGReasoner API, one can set up AGInfModels with either a Restriction reasoner or the default RDFS++ reasoner.
SPARQL
Bug19474 - Improve SPARQL describe queries
Excess variables are no longer included in the output of DESCRIBE
CBD (Concise Bounded Description) logic is now implemented so that we correctly handle blank nodes
Many general efficiency improvements
Bug19499 - Fix SPARQL XML and JSON output of geospatial values
Retrieving SPARQL SELECT results containing geospatial values as SPARQL+XML or SPARQL+JSON was broken because an internals change hadn't been propagated to the SPARQL engine. This is now fixed.
WebView
Rfe9815 - Fix accept-encoding issue for WebView in Internet Explorer 7
This fixes a problem where Internet Explorer 7 would display "Internet Explorer cannot display the webpage" when opening WebView (or any other page served by AG). Internet Explorer fails to handle deflated content, so we now serve it gzipped responses instead.
Bug19500 - Fix problem with displaying geospatial results in WebView
When a query result contained geospatial data, a bug in WebView was triggered that caused the result to fail to display. This bug has been fixed.
Bug19525 - Fix failure when opening a federated session in WebView.
Trying to create a federated session in WebView resulted in a PRECONDITION FAILED error. This was caused by a bug in the way session data was being returned by the server, causing WebView to believe it was not on a federated store. The problem has been fixed.
AllegroGraph 4.0.5d
Welcome to AllegroGraph 4.0.5d. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9966 - Front-end server-socket backlog parameter is too low
The kernel will only save a certain amount of pending connection attempts. If the server doesn't accept them quickly enough, they get discarded. The default backlog size was too small. It is now set to use the maximum the operating system allows, making the chance of discarded connections much, much lower.
AllegroGraph 4.0.5c
Welcome to AllegroGraph 4.0.5c. This release of AllegroGraph v4.0 provides efficiency improvements in the area of reasoning.
Server Changes
Rfe9887 - Exclude inferences from Sesame queries by default
Sesame queries (Tuple, Graph, and Boolean) will now be set to exclude inferences by default (this applies when the setIncludeInferred method has not been called on the query instance).
Bug19484 - Jena always uses reasoning
Fixed so reasoning is off by default and is enabled when InfModel is used.
Bug19485 - setAutoCommit(true) causes exception
True is the default, but calling this would raise an exception, so this has been fixed.
Bug19491 - Jena reasoning not on full graph
In the Jena API, AGInfModel, reasoning was previously done on graphs (default or named), and is now done on the full repository. Reasoning on scoped graphs will be included in a future release.
AllegroGraph 4.0.5b
Welcome to AllegroGraph 4.0.5b. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9866 - Increase limit on string file sizes
Previously, the files that held string data were limited to 32GB each. This release expands that limit to 512GB each.
Bug19503 - Reclaim cursor resources more aggressively
Certain SPARQL operations could acquire cursor resources and not release them immediately. Depending on garbage collection and server load, this could eventaully hang the server.
This patch ensures that SPARQL releases cursors immediately during triple-existence checks which prevents the resource issues from arising.
AllegroGraph 4.0.5
Welcome to AllegroGraph 4.0.5. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9334 - License support for agraph with triple-limit and expiration
AG now requires a license code which includes a limit on the number of triples and an expiration date, both of which may be unlimited.
If the license code is missing, invalid, or expired, it defaults to Free version, 5 million triples with no expiration.
Rfe9348 - Added offset keyword argument to getStatements/evalFreeTextSearch
The getStatements and evalFreeTextSearch methods on RepositoryConnection objects now support an optional offset keyword argument for skipping a number of results before returning values from the result set.
Rfe9471 - Limit maximum checkpoint frequency by configuration variable
A new configuration variable, MinimumCheckpointInterval, has been added that can be set to modify the frequency at which checkpoints are written by AllegroGraph. It defaults to the the CheckpointInterval period or five minutes if interval based checkpointing is not used.
The SkipPostMergeCheckpoint configuration variable has been removed.
Rfe9564 - Alternate mode for checkpoint triggering
In addition to triggering checkpoints in regular time interval, AllegroGraph now offers an alternate mode for triggering checkpoints based on the expected recovery time. It is enabled by the MaxRecoveryTime catalog/database parameters and needs to be set to the maximum number of milliseconds that AllegroGraph should spend recovering transactions from the transaction logfile during recovery. When AllegroGraph detects that the established maximum recovery time would be exceeded, a checkpoint is written. Note that the maximum recovery time period does not include the flat startup fee for a database that is independent of the number of transaction to recover from the transaction log file.
Rfe9604 - Suppress unnecessary checkpoints
In previous releases, AllegroGraph automatically created checkpoints when a database was started, stopped or a certain time interval had elapsed. This has been changed so that checkpoints are only written in these situations if there had been any commits since the last checkpoint has been created.
Rfe9608 - Have HTTP /eval service handle multiple returned values
Extra values returned by evaluated forms used to be discarded. The server now attempts to return them as a list of values, if the Accept format supports such a thing.
Rfe9691 - Remove race condition in internal API
In order to support a new use case, a race condition has been removed from the internal resource manager API.
Rfe9709 - Improved automated test for database recovery
Automated test coverage for database recovery has been improved.
Rfe9713 - Document load-nquads
Added the documentation for load-nquads and load-nquads-from-string to the Lisp Reference Guide.
Rfe9726 - Minor agload improvements
- improve agload output so that it is easier to understand,
- correct agload failure to load the TriX format,
- expose support for N-Quad format, and
- other minor internal improvements
Rfe9729: Make it possible to disable interval based checkpointing
Interval based checkpointing can now be disabled by setting the CheckpointInterval database parameter to zero.
Rfe9732 - Disable interval based checkpointing when recovery time based checkpointing is enabled
When recovery time based checkpointing is enabled, AG now switches off interval based checkpointing by default.
Rfe9766 - Ensure that the AllegroGraph server and the Lisp client use the same UID
Because the Lisp client must have full read/write access to a database's files and shared memory data structures, it must run under the same UID as the AllegroGraph server. This change enforces this requirement to avoid potentially confusing errors.
Bug19101 - with-variables->bindings-array is too strict
AllegroGraph's SPARQL engine raises an error when imposed bindings (the :with-variables keyword argument to run-sparql) are not a subset of the variables mentioned in the body of the query.
This error checking failed to include variables mentioned in a CONSTRUCT template or DESCRIBE variable list, which meant that query invocations which imposed bindings for variables mentioned in the CONSTRUCT template alone would raise an error.
This is now corrected.
Bug19168, Bug19332 - Ensure that user functions are compiled correctly
This patch improves the way that functions generated at run-time are compiled. This includes SNA generators and select query plans. These should all perform slightly better with this change.
Bug19209 - Fix bug with reasoner and rollback
In certain situations, the RDFS reasoner could become confused and fail to update its data caches after the triple-store was rolled-back. This is corrected.
Bug19339 - Update UPI serialization
There was a printer escape problem in the N-Triple serializer such that only one backslash was output when two should have been. I.e., \a instead of \a. This prevented serialized triples from being re-imported into AllegroGraph. This problem has been corrected.
Bug19429 - Problems with catalog option FilePlacementRule in agraph.cfg
A defect has been fixed that resulted in an error when a database created from a catalog that specified a FilePlacementRule was opened.
Bug19431 - Wrong parameters logged during database startup
A defect has been fixed that made the AllegroGraph server potentially log wrong parameters to the server log file during database startup.
Changes to the Lisp API
Rfe9643 - Remove synchronize-automatically from the Lisp API
In AllegroGraph 3, synchronize-automatically controlled whether or not a triple-store was immediately synchronized after the addition of new triples. Because this level of control is unnecessary in AllegroGraph 4, the variable is being removed.
Bug19188 - [Lisp API] create-triple-store :if-exists :error problem
The Lisp API function create-triple-store behaved incorrectly if the :if-exists keyword argument was :error. An error would occur whether or not the triple-store already existed. Fixed.
Documentation
Rfe9475 - Document catalog parameters vs database parameters
The AllegroGraph server configuration documentation has been updated to include information on how existing databases can be reconfigured.
Rfe9721 - Restore Lisp freetext tutorial
Adds a tutorial that describes the way freetext indexing works in version 4 of AllegroGraph.
Bug19411 - Configuration and performance documentation updated
The server documentation has been updated to reflect the removal of the 'Aux' catalog directive and the addition of the StringTableDir and FilePlacementRule directives.
Java Client
Rfe9743 - Upgrade to Sesame 2.3.1
Previously the Java client was using Sesame 2.2.4; this upgrades the client to the latest version of Sesame 2.
Jena Client
Rfe9744 - Upgrade to Jena 2.6.2
Previously the Java client was using Jena 2.6.0; this change upgrades the client to the latest version of Jena.
AllegroGraph 4.0.4
Welcome to AllegroGraph 4.0.4. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe8792 - Clean up partially created triple stores
When creating a triple store fails, the AllegroGraph server now deletes any files and directories that it may have created up to the point where the error occured. This makes it easier to retry creating the triple store once the condition that made creation fail has been removed.
Rfe9123 - Extend test functions to start and stop db server
The AllegroGraph server can now be stopped by providing the --stop-server switch to the agraph executable. The --config or --pid-file parameters must be present on the command line.
Rfe9292 - AllegroGraph Loader
Added the AllegroGraph Loader tool for parallel loading of large files.
Rfe9367 - AllegroGraph log should include the config file
A change has been made to AllegroGraph so that configuration file directives are written to the server log file at "info" log level during configuration file parsing.
Rfe9451 - Pause text-index merges during checkpoints
We make sure the text-index merger is paused when a checkpoint is taking place, to prevent resource contention from making the checkpoint take long, which would make other subsystems wait.
Rfe9497 - Ensure that select queries release temporary indices
Previously, select queries that used temporary indices (UPI maps) would not free them when the query completed. This could result in poor memory management and eventual poor system responsiveness. This is now corrected because queries release their temporary indices when they complete.
Rfe9508 - Add configuration options for manual control of file placement
Two new configuration options, StringTableDir and FilePlacementRule, have been added to the catalog and database configuration. They can be used to control the placement of individual database files.
Rfe9518 - SHM server handles client failures
Increased robustness of the background shared memory server so that it won't crash when a client shuts down prematurely while communicating with the server.
Rfe9579 - Include catalog name in conditions signaled by the storage layer
Error messages generated by the storage layer now include the catalog name in addition to the name of the database that was being acted upon when the error occured.
Rfe9580 - Use less strict permissions on UPI map and transaction log files
In prior releases, UPI map and transaction log files were created with mode 0600, which prevented them from being read by other users. This made is difficult to share databases. Now, UPI map and transaction log files are created with mode 0666 and the umask is used to determine the final file permissions. This keeps the permissions on these two types of files in line with the other database files.
Rfe9602 - Optimize various UPI coercion functions
Optimized some of the code paths that AllegroGraph uses to convert strings and future-parts into UPIs. This will provide a small performance benefit (1-2%) in some aspects of triple-store loading and querying.
Rfe9642 - Transaction logging adjustments
The recovery code has been adjusted so that automatic recovery after a machine crash is possible if the crash happened while a record was written to the transaction log.
Rfe9642 - Transaction logging adjustments for warm standby
The transaction log now includes marker records for the beginning of a commit that is used by the warm standby mechanism. Existing databases must be upgraded to accomodate for the new transaction log format.
Rfe9654 - Remove Aux configuration parameter
The Aux configuration parameter that was used in previous versions of Allegrograph to make a database use multiple disks for its files has been removed. In order to make use of multiple disks, the TransactionLogDir, StringTableDir and FilePlacementRule should now be used.
Rfe9710 - Make warning about uploading big files through WebView clearer.
Since browsers aren't very good at uploading multi-gigabyte files, WebView makes it clear that there are better alternatives for doing that.
Bug19199 - Lock database directories
When an Allegrograph server opens a database, it will now lock the database directory so that if a second server is configured to use the same directory, it cannot open the database. This protects the database from corruption which would be the result of two servers opening it at the same time.
Bug19203 - Restore the :use-planner option of the select query planner
The :use-planner option of the select query planner was being ignored. It now functions correctly.
Bug19221 - Durability issue related to file deletion
A problem has been identified that could result in database files to be wrongly deleted during checkpoint processing, resulting in database corruption. The problem has been fixed.
Bug19251 - Spurious warning message about nonexistent file ctl0
During database creation, the server logged a spurious warning message about a file "ctl0" that could not be deleted. This has been fixed.
Bug19256 - Database recovery fails with partially written checkpoint
A problem was identified that could cause database recovery to fail when the transaction log contained an incomplete checkpoint record. This problem has been corrected.
Bug19263 - Server log contains cut-off lines
Due to a programming error, it could happen that lines in the server log file were garbled or cut off. This program has been corrected.
Bug19264 - Opening databases is slower than it needs to be
A bug has been found that caused opening databases to be much slower than needed. The reason was that the "df" command was invoked by the database server for every file instead of once per directory. This has been corrected.
Bug19279 - Deleted triple data structure maintenance only happens once
After each merge (but only if 15 minutes has elapsed), a task runs which performs maintenance on the data structures which track deleted triples. Due to a logic bug, this operation was only happening once, even if subsequent merges occurred. This has been corrected.
Bug19282 - Improve parsing of XSD single and double floats
AllegroGraph was parsing XSD doubles using a single float parser which led to some values being incorrectly converted to infinity. AllegroGraph was also treating numbers outside the range of IEEE floating point representations inconsistently. It now always converts numbers smaller than the minimum representable value for a float to zero and numbers larger than the representable value for a float to positive or negative infinity (as appropriate).
Bug19303 - Improve algorithm used to find sort-upi-map program
In some cases, AllegroGraph was unable to find an external program needed for UPI map creation. This patch fixes the problem.
Bug19308 - Make namespaces available to code evaluated server-side.
Code run through the eval-in-server service is now given access to the namespaces that the user defined. It used to run in an environment without namespaces.
Bug19328 - Make remote stores raise an error when created while open
It is now an error to call create-triple-store on a store that is still being held open as a remote-triple-store (in Lisp).
Changes to the Lisp API
Bug19298 - Improve the printing of future-parts
(Lisp only) Future-parts were not printing properly when print-circle was bound to true. In addition, future-parts like !<foo> were not printing correctly (the angle brackets were left off). This patch corrects both of these problems.
Documentation
Rfe9339 - Improve 3-to-4 code conversion document
The Java and Jena sections of the 3.x to 4.x code conversion document have been greatly expanded.
Rfe9549 - Many improvements in the Lisp documentation
Improved and clarified many minor issues in the Lisp reference guide.
Rfe9690 - Documentation for AllegroGraph Loader
A documentation page for the AllegroGraph Loader has been added to AllegroGraph 4.0.4.
Rfe9711 - More detail in Java client installation instructions
The instructions for importing the AllegroGraph Java client into Eclipse have been augmented with more detail to make the process easier for new users.
WWW495 - Add LUBM Benchmarks to documentation navigation list
A link to the web page on AllegroGraph's performance on the Lehigh University benchmark (LUBM) has been added to the navigation bar of the Franz documentation pages.
Text Indexing
Rfe8932 - Allow sorting of freetext-searches by relevance.
freetext-get-triples and the HTTP /freetext service now support an argument that will cause the results to come back in sorted order of relevance, using a simple weighting scheme that takes into account word frequencies in the document and commonness of words across documents.
Rfe9356 - Add fuzzy text-index searching.
Freetext search expressions can now contain fuzzy terms, written as (fuzzy "term" [max-distance]) in s-expression queries, and term~ or term~2 in textual queries, which will match anything with a Levenshtein distance less than max-distance (or 1/5 word length when no explicit distance is given).
WebView
Rfe9504 - Improve handling of WebView file uploads to use less memory.
File uploads through WebView went through a few too many memory buffers, causing them to fail on systems without a lot of free memory. They are now directly streamed, and never stored in memory in their entirety.
AllegroGraph 4.0.3
Welcome to AllegroGraph 4.0.3. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe8999 - Use full blank-node IDs in RDF/XML output.
The nodeID properties found in RDF/XML documents created by AllegroGraph can now be sent back to the server to identify the blank node they refer to. They used to be shorter IDs, which were meaningless outside of the document.
Rfe9363 - Revise use of fsync and fdatasync and move to checkpoint time
In previous versions, AllegroGraph used the fsync() system call in various places to synchronize the on-disk state of database files. This could result in unexpected I/O activity when databases resided on ext3 file systems, as fsync() on ext3 always synchronizes all files, not just the file that has been requested to be synchronized. To improve this, AllegroGraph now syncs files only at checkpoint time.
Rfe9490 - increase application's initial heap size
Increase AllegroGraph's initial heap size to reduce garbage collection frequency and thereby improve application performance.
Rfe9505 - Improve efficiency of string dictionary lookup
Increased the efficiency of string lookup by roughly 20%.
Rfe9541 - Server installation script avoids writing double-slashes to agraph.cfg
Previously in configure-agraph, if the user gave a directory name with a trailing slash, the cfg file would be written with double-slashes in various dir names.
Now, the trailing slash is removed to avoid double-slashes. A double-slash is legal in Unix, but is visually unappealing.
Rfe9560 - Truncated cursors free resources more aggressively
Truncated cursors now clean up their environment more quickly which improves general AllegroGraph resource management.
Rfe9567 - Don't print a backtrace on socket timeout errors.
The server no longer logs a backtrace when a socket timeout is raised.
Rfe9586 - Prevent unneeded file reads in metadata-rollback
This provides an increase in efficiency when rolling back a triple store. The metadata system was re-reading its status file on rollback, even when it didn't have to.
Rfe9593 - Optimize string-dictionary
Optimized and stream-lined AllegroGraph's string-dictionary code. This provides a 5 to 6-percent speed increase in many string related operations.
Rfe9598 - Improve efficiency of string table intern operations.
Previously when strings were interned, more work than necessary was being performed if an entry already existed for a string. This extra work is now avoided.
Rfe9665 - Adjusted merge invocation algorithm
Changes have been made to the algorithm which determines when index merges should occur. These changes should make merge behavior more predictable and efficient.
Rfe9673 - Make sort-upi-map operation more verbose
In order to aid in tracking down a hard-to-reproduce bug, the sort-upi-map program now logs information about which files it is manipulating.
Bug19041 - Delete-triple-store only deletes files from the main directory (not the aux)
Previously, if a database used Main and Aux directories and was deleted using delete-triple-store, the files in the Aux directories were not deleted. This has been corrected.
Bug19085 - Fixed race condition in delete-list.dat handling
A file named "delete-list.dat" is used to maintain a list of which files should deleted at the next checkpoint. There was a race condition in the handling of this file which could result in it being deleted at an inopportune time. This problem has been corrected.
Bug19178 - Fix race condition in metadata system
In some circumstances, opening a store or rolling back could cause the system to try to read a non-existent file. This is now solved.
Bug19182 - Performance regression with concurrent writes.
A performance problem has been identified in the mechanism that Allegrograph uses to translate from internally-used logical file names to real on-disk path names. Previously, such lookups were made by a centralized process using a remote procedure call. Now, lookups are solely made within the requesting process without requiring an inter-process exchange.
Bug19192 - Merger may not notice that the database is shutting down
Previously, under certain circumstances, the merger may fail to shut down in an orderly manner during an otherwise normal database shutdown. This problem has been fixed.
Bug19189 - Prevent HTTP back-ends from holding their lock too long
This bug was most visible when using cursors in remote-triple-stores -- the client had to read out the whole cursor before it could make other requests. This is now fixed.
Bug19201 - Catch and log errors raised in the process-services thread.
Errors raised within the per-process thread that managed, for example, back-trace requests were not properly logged, but caused the debugger to be entered. This is now fixed.
Bug19206 - Unrecoverable database due to checkpoint file id reuse
A problem has been identified that could result in an unrecoverable database in situations where there have been unclean shutdowns during recovery. This problem has been fixed.
Bug19222 - Do not fsync temporary files during checkpoint processing
Previously, AllegroGraph called fsync for all files, including temporary files, that make up a database. This has been changed so that temporary files are not fsynced.
Bug19223 - Recovery hangs
A problem has been identified that could cause AllegroGraph to hang during database recovery afer an unclean shutdown. This problem has been fixed by starting the merger processes earlier so that they are available during the recovery process.
Bug19236, Bug19237 - improve geospatial subtype mapping support
The geospatial sub-type mapping code was occasionally being called incorrectly which could lead to spurious errors during query. In addition, the error message reported when a geospaital subtype was not found was being displayed incorrectly. Both of these problems are now corrected.
Bug19244 - Delete old deleted-shmem-files
It has been discovered that the deleted-shmem-files, which are used by the triple deletion functionality, have never been deleted when they became obsolete. This wasted disk space with stale data. The problem has been fixed.
Bug19257 - Deleted hash file deleted at the wrong point in time
A problem has been identified that could lead to the deletion of a deleted hash table file while it was still being referenced. This has been fixed.
Bug19276 - Possible unnecessary post-merge full scan of indices
After an index merge, a task sometimes runs to perform maintenance of the data structures used to track deleted triples. This operation requires a full scan of indices. Previously, this full scan was performed even if it was unnecessary. This has been fixed.
Changes to the Lisp API
Rfe9595 - Reduce unnecessary type-checking from Lisp API functions
The exported Lisp API functions (like get-triples or add-triple) were performing unnecessary type checks. This commit removes these checks and thereby increases the speed of code that uses this API.
This change also means that the Lisp API no longer supports using triple-store names to designate stores. That is, functions now require a triple-store instance and will no longer accept a triple-store name.
Bug19173 - Corrected bug in Lisp API for remote-triple-stores
Previously, cursors for remote-triple-stores could produce an error when a query returned too many unique strings. This error will no longer occur.
Bug19177 - Fix broken interpretation of namespaces in Sesame transaction documents
The 'name' and 'prefix' attributes of the addNamespace and removeNamespace tags were interpreted the wrong way. ('name' should hold the URL, and 'prefix' the short name).
Documentation
Rfe9339 - 3.x to 4.x code conversion document added
At the request of users, we have provided a document that lists the changes in the AllegroGraph Lisp, Python, and Java APIs in the upgrade from AllegroGraph 3.x to AllegroGraph 4.x versions.
Rfe9432 - Updates to Python API document
There were several updates to the Python API documnent to better delineate the distinction between public and private methods in 4.0.3.
Rfe9504 - Document TempDir configuration directive
The location where the server creates temporary files can now be controlled by specifying a TempDir option in a configuration file. This option is now described in the Daemon Configuration web page.
Rfe9540 - Audited the Python API doc for currency
The Python API document has been updated to reflect recent changes in the Python API.
Rfe9599 - Update Java section of 3-to-4 conversion document.
The v3-to-v4 code conversion document, convert3to4.html, has been updated to enlarge on the information about the new AllegroGraph Java Client. [5.0 note: the convert3to4.html document is no longer part of the documentation set.]
Bug19320 - Include links to upgrading information in documentation
Added links from the navigation section of each document to the "converting from AllegroGraph 3 to 4" documentation.
SPARQL
Rfe9561 - Improvement in SPARQL recycling of storage-layer cursors.
The SPARQL query engine now more aggressively discards and recycles cursors over on-disk indices. This improves the behavior of the system under load.
Gruff
New overview/scroller panes
The View menu has three new commands for using an overview/scroller pane.
New submenu File | Export Displayed Data As ...
A new "Export" child menu on the File menu allows exporting the currently-displayed triples or nodes to a file in one of several standard formats.
Child menu names are now included in the documentation.
Until now, a menu command that's on a cascading child menu, such as "Drawing Options | Derived Node Labels | Capitalize First Word", appeared in the docs as simply "Drawing Options | Capitalize First Word", which could make it hard to find on the menus. The docs now include the full path.
New freetext command "Add | Edit the Set of Registered Predicates".
There are new commands for registering some or all predicates for freetext queries at one time, and for unregistering some in agraph4.
Rfe9090 - Initial port to the agraph4 freetext API
The freetext commands now work on agraph 4.
Export commands failed when bridge lines were present
The new File | Export commands failed when bridge lines are present. This is now fixed.
AllegroGraph 4.0.2
Welcome to AllegroGraph 4.0.2. This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9084 - Return Prolog parse errors as bad requests, not internal server errors.
When issuing a Prolog with a syntax error, the response no longer indicates an internal server error, but is returned as a bad request error, tagged "MALFORMED QUERY".
Rfe9313, Bug18592 - Improved clause reordering in select queries
Modified the clause size estimation code to make it work better with AllegroGraph 4.0. The change greatly increases the speed of some Prolog-based select queries.
Rfe9345 - Ensure that distribution does not contain extra debug scaffolding
We now ensure and test that certain developer tools and debugging aids are not part of the distributed AllegoGraph. Removing these from the released code will have a small performance gain in some parts of the system.
Rfe9351 - Don't write deleted hash table if no triples have been deleted
Previously, the shared memory hash table that is used to track deleted triples was written to a file at every checkpoint. With this change, the table is only written when deletion of triple actually has happened. This speeds up trivial checkpoints as well as database open and close operations by about 35%.
Rfe9404 - Improve error message when bulk loading N-Triple data
The error signaled when loading invalid N-Triple data has been made clearer.
Rfe9435 - Support offset argument to geospatial query services
For consistency, all services that support a limit argument now also have an offset argument, which allows results to be skipped.
Rfe9440 - Improve cursor code efficiency in some situations
Improved some low-level cursor implementation details that provide a small but measurable speed and space boost for some queries.
Rfe9447 - Improve detection of SPARQL Negation-As-Failure idiom.
This change allows for SPARQL's idiom for Negation-As-Failure to be detected more reliably, and thus executed more optimally.
In particular, this allows for the OPTIONAL component to contain FILTERs, so long as they refer only to variables bound within the OPTIONAL itself.
One such query might be:
PREFIX foo: <http://www.example.com/foo/foo_model#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?foo ?type
WHERE {
?foo a foo:Bar .
?foo a ?type .
OPTIONAL {
?subType rdfs:subClassOf ?type .
FILTER(?subType != ?type)
}
FILTER(!BOUND(?subType))
} Rfe9448 - Add upi< and upi<= functors to prolog select
Added additional functors to the Prolog-based select language to make some operations easier to use.
Rfe9462 - Improved database shutdown process
The shutdown process that AllegroGraph uses when a database is closed has been improved to be more orderly and reliable.
Rfe9464 - Support adding and removing of namespaces from Sesame XML transactions
A client program that uses the <setNamespace> and <removeNamespace> tags has been shown to exist (TopBraid), so they have been implemented.
Rfe9493 - Added mixed workload to events test
A -m parameter was added to the events test to run a Phase 6 mixed workload after the normal phases complete. This runs a mixture of queries, adds, and deletes until it is interrupted.
Bug19070 - /etc/init.d/agraph 'status' doesn't work
Previously, running "/etc/init.d/agraph" resulted in an error message instead of reporting the status on the agraph service. This has been fixed.
Bug19078 - Improve error message for invalid use of create or open-triple-store
The error message signaled when creating (or opening) a triple-store with incorrect parameters has been improved. The documentation of both functions is also clearer.
Bug19089 - Make sure daemon doesn't hang when it encounters an error
Turns off a debugging switch that was, unintentially, switched on by default, causing the service daemon to enter the debugger when encountering an error.
Bug19090 - Improve handling of graph-filtered-stores
Graph filtered triple-stores are used in various places in AllegroGraph's query engine. Prior to this fix, they did not implement some of the necessary API and could generate an error.
Bug19091 - Prevent server shutdown from locking up when a session is open.
The server would lock up when sent a sigterm if an HTTP session was still open. This is now fixed.
Bug19101 - Error detection for with-variables and CONSTRUCT queries.
AllegroGraph's SPARQL engine raises an error when imposed bindings (the :with-variables keyword argument to run-sparql) are not a subset of the variables mentioned in the body of the query.
This error checking failed to include variables mentioned in a CONSTRUCT template or DESCRIBE variable list, which meant that query invocations which imposed bindings for variables mentioned in the CONSTRUCT template alone would raise an error.
The resolution for this bug corrects the issue.
Bug19121 - Fix bug in namespace caching for HTTP back-ends.
Occasionally (depending on the system clock rolling over to the next second at the wrong moment) a newly defined (or deleted) namespace wasn't being seen by a back-end, causing 'unknown namespace' errors. This is now fixed.
Bug19123 - Allow reasoning-triple-stores to create new-blank-nodes
Reasoning-triple-stores (and other encapsulated-triple-store classes) were unable to mint new blank IDs. This prevented them from loading any RDF data that contained blank nodes. This fixes the problem.
Bug19145 - sendfd library does not handle "out of file descriptors" condition gracefully
When a client process experienced file descriptor shortage while connected to an Allegrograph store, memory corruption could occur. This has been corrected.
Bug19151 - Refcounts on open indices not updated aggressively
Previously, if regular query operations were performed against a database that was continually growing, file handle exhaustion could occur due to incorrect reference counts on indices. This problem has been fixed.
Bug19155 - Correct printing of invalid-characters-in-name condition
This fixes a syntax error in the formatting of the condition used to signal an invalid triple-store name or catalog. Previously, the error message produced was difficult to interpret; now it is clearer.
AllegroGraph Server: HTTP Interface
Rfe9227: Allow an HTTP proxy to be configured for the server
The HTTPProxy configuration directive can now be used when the server has to use an HTTP proxy to connect to the internet (for example, to load data from a URL).
Rfe9318 - Remove persistent federations from the HTTP protocol
Accessing federations of stores is now done by opening a session for such a federation, not by creating a persistent federated store and opening that.
Rfe9450 - Allow committing and rolling back through HTTP headers
As a performance optimization, it is now possible to have commit or rollback commands piggyback on other HTTP requests, using the x-commit and x-rollback HTTP headers.
Rfe9458 - Expose a full-featured textual free-text query language over HTTP
The 'pattern' argument to the /freetext service now understands strings like 'foo | bar' (foo OR bar), 'foo*' (everything starting with foo), '"foo bar"' (matching the exact string "foo bar") and even 'quux (foo | bar)' (quux AND (foo OR bar)).
Rfe9458 - Tweak free-text query expression format
When searching from the Lisp API, a phrase search is now done with '(phrase "foo bar") rather than "\"foo bar\"", and a wildcard match is done with '(match "wild") rather than just "wild".
Rfe9483: Add continueOnError parameter to HTTP statement loading
For N-Triples and N-Quads data, this argument can be used to make the load continue when malformed data is encountered, skipping the broken triples.
Rfe9496 - Support clearNamespaces tag in Sesame XML transaction documents
This tag was overlooked in the original implementation. It is now supported.
Rfe9499 - Support prepared queries in the HTTP protocol
It is now possible to 'prepare' SPARQL queries, either as a side-effect of executing them, or as a separate action. The requests for executing the prepared query will be more lightweight, since they don't have to pass in the query itself and most of its arguments, and skip the query-parsing stage.
Bug19088 - Allow a default namespace to be defined over HTTP
Due to the server stripping off trailing slashes from URLs and redirecting, the PUT request to ../namespaces/, which is required to create a namespace without a name, was redirected to ../namespaces, and didn't work. The redirecting behaviour is now turned off for this URL.
Changes to the Lisp API
close-triple-store has new 'commit' keyword
close-triple-store has a new 'commit' keyword argument.If true, then the commit-triple-store will be called before the store is closed.
user-reasoner keyword removed
The deprecated use-reasoner keyword argument has been removed from make-tutorial-store; use apply-reasoner-p instead.
sync-triple-store has been removed
The deprecated sync-triple-store has been removed; use commit-triple-store instead.
Rfe9576 - :expected-size argument to open-triple-store and create-triple-store no longer supported
To specify the expected size of a new database, edit the catalog configuration file before creating the database. To change the expected size of an existing database, change the ExpectedStoreSize parameter in the parameters.dat file of the database.
Bug19046 - Fix error generated by future-part printer
If a namespace mapping did not exist, then the future-part printer could signal an error instead of correcty printing the part. [Lisp API]
Bug19081 - Change return type of ego-group-layers from vector to list
The ego-group-layers function was incorrectly returning a vector of results instead of a list. Its behavior now matches the documentation: it returns a list of lists where each sublist contains nodes discovered at that level. [Lisp API]
Documentation
Improved Lisp reference manual documentation
Made several minor formatting changes and added documentation for the upi<= function and functor.
Document new free-text index HTTP API.
The creation and configuration of multiple text indices over HTTP is now described in the protocol document.
Rfe9169 - Add links to Javadocs from AllegroGraph documentation
Added links from the navigation bar of each AllegroGraph documentation webpage to the Javadocs.
Rfe9314 - Update Lisp API docs to reflect changes in freetext-indexing API
Indexing is no longer managed by keeping a list of predicates, but by creating one or more indices, each with multiple configurable parameters.
Rfe9318 - Update the HTTP protocol documentation for changed federation API
The services for creating and deleting persistent federations are gone, a more flexible way to open sessions, which also allows federation of stores, has been added.
Rfe9377 - Update free-text indexing example in Python tutorial
The 12th example in the Python tutorial now shows how to create multiple indices and search for both text literals and URIs. The example introduces these new methods:
createFreeTextIndex
getFreeTextIndexConfiguration()
evalFreeTextSearch() Rfe9378 - Update Python documentation to reflect new federation API
The Python tutorial has been updated to reflect recent changes in the Python API for creating federated sessions.
Rfe9383 - Document new-style federation in the java client
The 16th examples in the Java tutorial has been updated to reflect the new API for creating a federated repository.
Rfe9500 - Update free-text-index entries in Python API html doc.
The connection object methods for free-text indexing have been updated to reflect recent changes in the API.
Java Client
Rfe9318 - Support session-based (rather than persistent) federation
The methods in AGServer for creating and deleting federations have been replaced by a single method, federate, which takes any number of repository objects, and returns a 'virtual' repository that federates all of them.
Rfe9392 - Use accept-encoding gzip to get back compressed results
Get and Post requests now use the Accept-Encoding: gzip header to request that responses be compressed, to improve performance when results are large.
Modified both tutorial examples files to call closeAll prior to deleteRepository in exampele1, as not doing so can result in orphaned sessions on the toClose list.
Rfe9436 - Make session lifetime configurable
Session lifetimes can now be configured for each connection, and should be set prior to making a method call that initiates a dedicated session. The default session lifetime is 1 hour. Tutorial example 22 sets the session lifetime to 120 seconds to demonstrate.
Rfe9442 - Add basic information to the jar manifest
The Java client jar manifest now includes information about the current implementation, including name, version, vendor name, vender URL, time of creation, etc.
Rfe9499 - Support query preparation
Calls to prepareTupleQuery, prepareGraphQuery, and prepareBooleanQuery now return query objects that will, during the next call to evaluate(), request that the server save the query for future reuse (in dedicated sessions only), eliminating some parsing and other overhead in subsequent calls to evaluate(). Tutorial example 14 has been modified to use non-autocommit mode so that it demonstrates query preparation in a dedicated session and multiple calls to evaluate() with different variable bindings.
Bug19086 - JenaTutorialExamples now uses Graph#getName()
The tutorial examples now use Graph#getName() to get the graph's name, rather than using Graph#toString(), which now prints out additional human readable information about the graph.
Bug19092 - Jena ResultSet and PrefixMapping compliance
AGPrefixMapping now supports methods qnameFor, removeNsPrefix, samePrefixMappingAs, setNsPrefixes, shortForm, expandPrefix, lock, and withDefaultMappings. The toString method now returns human readable information about the prefix mapping.
AGResultSet now supports methods getResourceModel, getResultVars, and remove.
AGQuerySolution now supports methods contains, get, getLiteral, getResource, and varNames.
Python Client
Rfe8610 - Help finding pairs of UPIs
The 23rd example in the Python tutorial has been extended to demonstrate how to use inequalities to reduce the number of permutation matches in SPARQL and Prolog queries.
Rfe9311 - Expose revised FTI API in the Python client
Text-indexing is no longer configured by adding predicates, but by creating (and updating or deleting) multiple separate indices.
Rfe9318 - Move to session-based federation API.
To federate stores, one now calls AllegroGraphServer.openFederated, which opens a session that federates the given stores, rather than creating a persistent federated store and opening that.
Rfe9357 - Support wordFilter option to text-indices in Python client
The createFreeTextIndex and modifyFreeTextIndex methods of a connection now allow a wordFilters argument to be specified, which allows accent removal and word stemming to be configured.
Text Indexing
Rfe8775 - Support multiple text-indices per repository, add configuration parameters.
Multiple free-text indices can now be active in a repository, and there is an API to create, query, modify, and delete them. New parameters supported are: stop words, minimum word length, whether to index resources, which literals to index, indexing non-object parts, and the possibility to index all triples, regardless of predicate.
Rfe8775 - Add triples to free-text index as soon as they are added.
A commit used to be necessary to be able to find newly added triples through the free-text index. They can now be queried (by the current transaction) as soon as they are added.
Rfe9357 - Add word-filters to text-index configuration (accent removal, stemming)
It is now possible (both in the Lisp API and over HTTP) to specify a list of word filters when defining a freetext index. These are operations that are performed on every word that is indexed, and every word that is queried, allowing various 'normalisations' of words. Currently allows two filters, "drop-accents" to remove accents from letters, and "stem.english" to apply basic, English-language word stemming.
Changes to WebView
Rfe8775 - Allow free-text reconfiguration through WebView.
WebView now allows the creation, modification, and deletion of free-text indices, exposing a UI for all the new configuration parameters.
Rfe9408 - Expose free-form sessions in WebView
There is now, for users with the "session" permission flag, a control on the main WebView page for starting and entering a session over a federated, reasoning, filtered, and/or remote store.
Rfe9438 - Show server version in WebView
WebView now shows the exact version of the server in its header.
Rfe9453 - Support bulk-adding of namespaces in WebView
Clicking 'bulk input' when adding a namespace now pops up a new input dialog, where a set of namespaces can be typed or pasted all at once.
Bug19105 - Restore Safari-compatibility for WebView
WebView was broken on recent versions of the Safari browser. This has now been remedied.
AllegroGraph 4.0.1
Welcome to AllegroGraph 4.0.1. This version of AllegroGraph contains updates to the AllegroGraph 4.0 release, including:
- Cursor optimizations for increased performance
- Improved caching of RDFS++ reasoning results
- Improved SPARQL range-query efficiency
- Heuristic optimization of negation as failure in SPARQL queries
- SPARQL query performance tuning documentation added
- Hardware Profiles added to Performance-Tuning Guide
This release of AllegroGraph v4.0 provides overall product stability and efficiency improvements.
Server Changes
Rfe9137: Heuristic optimization for negation as failure in SPARQL queries.
SPARQL queries which use the common OPTIONAL/FILTER (!bound(?foo)) idiom to express negation as failure are now handled more efficiently, computing fewer unnecessary intermediate results.
This optimization can be disabled by setting the non-exported variable sparql.algebra::rewrite-naf-p to nil.
Rfe9320: Improved caching of RDFS++ reasoning results
Improved RDFS++ reasoner efficiency by selectively caching intermediate results. This makes many queries involving the reasoner between 2 and 10 times faster.
Rfe9352, Bug19067, Bug19068: Improve internal cursor operations
Optimizing how AllegroGraph treats empty cursors (i.e., cursors known to return no results) so that operations with them are faster.
Rfe9374: Improved SPARQL range-query efficiency in some cases
Improved the detection of range queries that will return no results which allows SPARQL to optimize other cursor operations.
Bug18978: Optimizing cursor creation and filtering
Improved the way AllegroGraph internally manages cursor creation and triple-filtering to increase performance.
Documentation
Hardware Profiles added to Performance-Tuning Guide.
The performance-tuning guide has a new section describing the kind of hardware Franz recommends for users who will be deploying applications on the scale of 3-4 billion triples.
Rfe9374: SPARQL query performance tuning documentation added.
The performance tuning guide has been extended to describe possible approaches to handling typed literals to improve SPARQL query performance for numeric and date FILTERs.
AllegroGraph 4.0
Welcome to the new AllegroGraph v4.0. This latest version of AllegroGraph is designed take full advantage of symmetric multi-processing hardware and brings enterprise database features to the world of Semantics and RDF. AllegroGraph, now incorporates:
- Full ACID database compliance.
- Full Recoverability.
- Online backup and restore.
- Automatic Indexing.
- HTTP-based client/server libraries supporting: Java, Sesame, Jena and Python.
This release of AllegroGraph v4.0 is focused on enhancements to database backup and restore, Java Jena client library compliance, and overall product stability improvements.
Server Changes
Add a Lisp function to list the catalogs on a server.
The function #'catalogs, exported from the db.agraph package, can now be used to find out which catalogs a server provides.
Added ego-group-layers function
ego-group-layers returns a nodes ego-group organized by the depth at which each node was found. [Lisp API]
Documentation: ego-group-layers (node depth generator)
Return a list of lists of the nodes in nodes ego group.
Each element of the result is a list of the nodes discovered at that list's depth.
These are the nodes in the graph that can be reached by following paths of at most length depth starting at node. The paths are determined using the generator to select neighbors of each node in the ego-group. See the description of SNA generators for additional details.
Added support for auto-commit to the bulk load functions
The commit parameter to load-ntriples, load-rdf/xml and load-trix can be one of
nil- do not committ- commit at the end of the load- a number N - commit every N-triples
Changed behavior of breadth-first-path-search when maximum-depth is given.
Previously, when the breadth-first-path-search functions were given a maximum depth, AllegroGraph would return the shortest path or paths found that were no longer than that depth. Now, AllegroGraph returns all paths found that are no longer than maximum depth.
This can be very expensive for graphs with high connectivity. To return only the shortest path, use a maximum depth of nil. [Lisp API].
Since the Prolog SNA functors use the underlying SNA functions, their behavior has also changed.
Change the keyword value used to identify typed literals
Previously, AllegroGraph used :typed-literal to specify a typed literal. To make things more consistent with :literal :literal-language, this has been changed to :literal-typed. [Lisp API]
Define an error type to signal failures to connect to the AllegroGraph server.
Any operation (opening, creating, deleting, probing) that has to talk to the server will now raise a condition of type db.agraph:could-not-connect-to-server-error if it fails to find a server on the specified port.
Fix mistake in instance-subprocess communication protocol.
This would cause the instance to go into an infinite loop and become unresponsive when one of its children died.
Make delete-duplicates-on-merge setting persistent.
The flag was not properly restored when opening a triple store.
Rfe7381 - Add a db keyword argument to several API commands Added the db argument to the following commands
- part->string
- part->value
- part->ntriples
- part->long
- part->concise
- part->terse
- print-triple
[Lisp API]
Rfe8765 - Improved Prolog query performance
Many Prolog queries execute more than 30% more quickly.
Rfe8860 - Geospatial functions now accept strings
The geospatial functions in AllegroGraph previously required a subtype object. They now accept strings which name a subtype object. This makes them easier to use. [Lisp API]
Rfe9146 - Allow 'merging' of store metadata on concurrent change.
Before, two sessions both adding a type-mapping and then committing would cause a conflict error. The metadata system is now clever enough to merge such changes.
Rfe9209 - Don't log process termination messages before cleanup.
Both instance processes and their children used to output '... terminated' before they cleaned up, which in some cases led to them hanging or crashing after they claimed to have terminated.
Rfe9212 - Periodically clean up deleted triple data structures
Previously, the data structures which track deleted triples could grow without bounds. Now, they are periodically examined and obsolete portions are purged.
Rfe9291 - Use XSD instead of internal type-codes for /mappings
The /mapping protocol now returns lists of
mapping-kind datatype/predicate XSD-type rather than
mapping-kind datatype/predicate numeric-type-code This makes both debugging and general use of the mappings much simpler for clients.
Rfe9294 - Added SNA path functors to bind only the first matching path
Added breadth-first-search-path and bidirectional-search-path functors to correspond with the existing depth-first-search-path functor. These all run SNA path finding algorithms and stop after finding the first matching path.
Rfe9315 - Do early probe to see if file system supports >4GB file offsets
Allegrograph requires that the underlying file systems support offsets longer than 32 bits. Some file systems (i.e. NFSv2) do not support long offsets, which caused file system related failures in varying situations. Allegrograph now probes each of the file systems specified in a configuration to make sure that long file offsets are supported.
Rfe9322 - Changes in encoded date and time UPIs.
Corrected a problem where the fractional seconds stored in an encoded UPI could be lost. The update fixes the stored representation and invalidates existing triple-stores that have any time, date or date-time encoded UPIs.
Rfe9329 - Improved loading of a required shared library.
Previously, AllegroGraph required loading of a particular shared library (librt) that did not always exist by the same name on all Linux distributions. This problem has been corrected.
Rfe9330: Set up selinux security context on shared library during installation
Previously, on systems with SELinux enabled, the agraph server would not start up properly unless the operator issued the following command first:
chcon -t textrel_shlib_t <agraph lib dir>/libacl*.so Now, this command is automatically performed during RPM package install or when using the install-agraph script that comes with the tarball package.
Rfe9340 - Disallow overlapping catalog directories.
Having, for example, one catalog in /tmp/foo, and another in /tmp/foo/bar, or two catalogs in the exact same directory, is no longer allowed, since this is bound to cause stores to be corrupted.
Rfe9362: Output a log message when deleting a triple-store.
Deleting a store wasn't visible in the log before. It now notes which store was deleted.
Spr36556 - Reduce number of duplicate rdf:type inferences
AllegroGraph's type inference was returning a triple for each possible inference path rather than filtering out the duplicates. I.e., if a rdf:type B could be inferred two ways, then the reasoner would return two triples. This no longer happens.
Spr 36565 - Standardize the printing of encoded types
AllegroGraph would sometimes print typed literals non-syntactically. This no longer occurs.
Bug18679 - Made part->value function more consistent
The part->value function now returns type-codes for both UPIs and future-parts. Previously, it returned a type-code for UPIs and a keyword (like :literal or :anon) for future-parts. [Lisp API].
Bug18679 - Removed the deprecated part-value
The deprecated part-value function has been removed. Use part->value in its place. [Lisp API]
Bug18738 - Make query-planner work on remote-triple-stores
Previously, select queries that needed to reorder their clauses would instead signal an error on remote-triple-stores. This patch corrects this problem and lets remote-triple-stores work correctly with the query-planner. [Lisp API]
Bug18840, Bug19003 - Inconsistent state after duplicate purging
Total triple counts were wrong after duplicate purging. By default, AllegroGraph purges duplicate triples from the index during merge operations. Previously, when duplicate triples were purged, the total count of triples in the store was not updated. Also, in most cases, one of the seven triple indices would not be purged of duplicates, resulting in wasted storage and potentially inconsistent query results. These problems have been corrected.
Bug18879 - Corrected add-triple so that it could accept large strings
Unlike the bulk loaders, the add-triple function was unable to accept strings longer than 8092 characters and would signal an error. Now add-triple can accept strings of arbitrary size.
Bug18800 - Unbound prolog variables could pollute select query results
Queries like
(select (?s ?t)
(q ?s !<http://franz.com/sna#marriage>
!<http://franz.com/sna#peruzzi>)
(optional (q ?s !rdf:type ?t))) could return unbound expessions like `{Unbound 14a170c9}` instead of `nil`. Now these queries correctly return `nil`.
Bug18880 - Corrected serialization of RDF/XML
The serializer sometimes added extra angle brackets around typed literals. Now it correctly outputs them as, e.g.,
"42"^^<http://www.w3.org/2001/XMLSchema#int> Bug18947 - Make service daemon responsible for deleting stores
delete-triple-store used to only check whether the store was open in the current image, and thus deleted stores even if another process had them open. This moves the responsibility for deleting stores to the service-daemon, which actually knows whether anyone has the store open.
Bug18949 - Fix geospatial type mapping in federated stores
AllegoGraph was not handling geospatial type-mappings in federated-triple-stores and could fail to return all of the triples from every leaf store. This is corrected.
Bug18972 - Correct printing of times with fractional seconds
When a time or dateTime with fractional seconds had leading zeros in the fractional seconds part, AllegroGraph could fail to print it. I.e., a time like 10:31:55.028192 might print as 10:31:55.281900. This printing problem has been corrected.
Bug19010 - Fix 'store in use' bug when accessing store over HTTP.
Fixes a bug where a store accessed over HTTP could not be deleted or replaced through the direct Lisp API until its handle had timed out.
Bug19018 - Make sure instance processes are cleaned up when breaking
Fixes a few cases where unexpected errors would cause the daemon to get confused about the stores it had open (instance crash, client dying while opening or creating).
Bug19024 - Show catalog name in store-status error messages
Errors like 'store exists' and 'store not found' used to only contain the store name, now they also have the catalog. Trying to perform an operation on a non-existent catalog now raises an error that clearly indicates the catalog doesn't exist.
Bug19045 - Use . as the fractional seconds separator in times and dateTimes
Previously AllegroGraph used the comma (,) as the separator in its printed representation of XSD times and dateTimes. This is acceptable ISO 8601 notation but not acceptable XSD notation. This bug fixes the problem so that fractional seconds are separated from whole seconds by a period (.).
Bug19055: Fix potential hang when committing
Under certain circumstances, commits could hang indefinitely while waiting for chunk merging to complete, but chunk merging would never commence due to a counter having an invalid value. This problem has been corrected.
Bug19061: upi->String table race condition fix
Previously, under high load conditions, errors could occur during string table operations. This problem has been fixed.
AllegroGraph Server: HTTP Interface
Allow nested lists in JSON-shaped Prolog-query output.
application/json results for Prolog queries can now contain nested arrays of results, to make it easier to use some of our SNA functionality.
Allow the creation of sessions with federated, graph-filtered, or reasoning stores.
It is now possible to spawn HTTP sessions on a store built up by applying an arbitrary combination reasoning, federation, and graph-filtering to concrete triples.
Cause output of --http-trace switch to actually get flushed to the file.
Before, it was buffered, and would only be visible after a large amount was written, or the server was shut down.
Clean up namespaces when a repository is deleted.
Before, deleting and re-creating a repository would leave the old namespaces intact.
Enable streaming of XML bodies when loading statements.
A problem in our SAX parser made this difficult before, but that has since been fixed. This makes it possible to load large files with less overhead.
Fix a bug that prevented session permissions to show up in WebView user-contol UI.
(No further text.)
Fix a bug with namespace caching.
Sometimes, back-ends wouldn't notice a user's namespaces had changed when they were updated twice within the same second.
Fix bug in remote geospatial queries.
There was a problem that broke remote-triple-stores when more than one geospatial subtype occurred in a query response. This is now fixed.
Fix user-access bug in HTTP protocol.
Fix a bug that prevented users with access to a store in the root catalog, but not the whole catalog, from accessing this store over HTTP.
Make user-access-configuration interface in WebView more obvious
It was easy to mistake the widget for adding access for an indication that access had already been granted.
Support changing user passwords through the HTTP interface
There is now /users/[NAME]/password, to which one can POST a new password.
Support resetting namespaces to the defaults.
There is now a reset argument to the namespace clearing interface, which, when true, will cause the default namespaces to be restored.
Rfe9131 - Add defaultGraphName parameter to SPARQL query interface
This is used to provide a way to refer to the default graph inside a SPARQL query using a URI.
Rfe8999 - Use a unified format for serializing blank nodes.
Before, each component had its own format. This moves the HTTP server to the unified format.
Rfe9183 - Silence AServe's socket-reset warnings
The log is no longer polluted with spurious connection-reset messages.
Rfe9185 - Support delete-duplicates-during-merge remotely
Remote-triple-stores now support the :delete-duplicates-during-merge initarg, and the db-delete-duplicates-during-merge accessor.
Rfe9217 - Add more help text to the WebView interface.
Potentially confusing input fields in WebView (such as those where resources or literals are entered) now have a question-mark icon that pops up an explanation.
Rfe9236 - Support SPARQL/Update over HTTP.
POST requests to a repository can now issue a SPARQL/Update query, if the user has write access to the store.
Rfe9247 - Expose server processes in HTTP interface
It is now possible to list, inspect, and kill the server's processes through the HTTP interface. Additionally, one can start up telnet servers in the processes, in order to directly debug them.
Rfe9247(2) - Allow process control from WebView
Superusers can now view the server processes, and open telnet servers in them, through the WebView UI. (See the 'Processes' link in the navigation bar at the top-level page.)
Rfe9295 - Add a commit argument to PUT/POST /statements.
This supports auto-committing every X triples, to make uploading huge files viable.
Rfe9310 - The PUT /repository/[name] now overwrites by default.
The PUT /repository/[name] service in the HTTP protocol now overwrites existing repositories by that name unless override=false is passed.
Bug19002 - Fix bug in HTTP protocol namespace caching.
Fix a bug that caused problems with namespaces after deleting and re-creating (over HTTP) a store that had namespaces defined.
Bug19005 - Corrected problem in client lookups of individual datatype mappings.
Previously, a request for an individual datatype mapping would return nil regardless of how the mapping was defined. Now it returns the correct value.
Documentation
Server Installation Page Updated to be Ubuntu-Friendly
The server-installation instructions have been updated to make them more friendly to people who don't know how to install Python, python-pycurl, and python-cjson packages. This included some specific suggestions for Ubuntu users for RPM installation and Python client installation.
Rfe9322 - Added client tutorials to the documentation
The documentation now contains tutorials on how to use the Python and Java clients. Please see the links off of the main web pages.
Java Client
Expanded Coverage of Literal Values in Java Tutorial.
The Java tutorial has been revised and expanded to document the behavior of literal typed values (in example5()). This includes the default behavior of untyped literals, as well as their use in getStatements queries as well as filtered and direct SPARQL queries. The section covers strings, floats, ints, untyped values, dates and times.
Java Tutorial Describes Managing User Accounts
The Java tutorial has a new section on "Creating AllegroGraph Users with WebView." It describes how to create a manage new AllegroGraph user accounts.
Rfe8812 - Jena compliance improvements
Added compliance tests for Jena GraphMaker and Model interfaces and corrected several non-compliances.
Properly handle the case of getting a namespace uri for a prefix that does not exist, and add support for expanding a QName.
Add GraphMaker support for creating anonymous graphs, listing graphs, checking existence of a graph, and returning existing graph objects rather than creating new ones when possible.
Throw AlreadyExistsException when attempting to create an existing graph in strict mode, and throw a DoesNotExist exception when attempting to remove or open a non-existent graph in strict mode.
Rfe8812 - Jena Graph compliance improvements
Added compliance tests for the Jena Graph interface and corrected several non-compliances.
AGBulkUpdateHandler's removeAll method properly clears a graph.
Added a custom AGCapabilities class declaring handlesLiteralTyping to be false, as AGGraph does not currently support D-entailment.
Add support for the executeInTransaction method in AGTransactionHandler, though using begin/commit/rollback explicitly is preferred.
AGTripleIterators are now tied to AGGraphs, calling remove on the iterator will delete from the graph.
Rfe9359 - Jena Prolog support.
Added support for Prolog select queries over a Jena model. Queries must be written to select for variables that are bound to RDF parts, rather than to arbitrary Lisp objects.
Python Client
Removed Common Logic
Removed the experimental and undocumented client side Common Logic query language. Please contact [email protected] for information if you were using this language.
Removed JDBC result sets
Removed the ability to get JDBC result sets. Please contact [email protected] if you need ways to get the result data without instantiating the corresponding Literal, URI, or BNode objects.
Namespace implementation is now server-side
Removed the client-side namespace implementation in favor of a server-side namespace management.
"Duplicate Triples" added to Python Tutorial.
The Python Tutorial has a new section on "Duplicate Triples", based on example23(). It describes the sources of duplicate triples and repetitive query results, and prescribes techniques for bringing them under control.
Python Tutorial Describes Managing User Accounts.
The Python tutorial has a new section on "Creating AllegroGraph Users with WebView." It describes how to create a manage new AllegroGraph user accounts.
Expanded Coverage of Literal Values in Python Tutorial.
The Python tutorial has been revised and expanded to document the behavior of literal typed values (in example5()). This includes the default behavior of untyped literals, as well as their use in getStatements queries as well as filtered and direct SPARQL queries. The section covers strings, floats, ints, untyped values, dates and times.
Python Examples from Command Line
The Python tutorial example file (tutorialexamples40.py) can now be run from the command line.
$ python tutorial_examples_40.py runs all tests.
$ python tutorial_examples_40.py all runs all tests.
$ python tutorial_examples_40.py 1 5 22 runs tests 1, 5, and 22 Rfe9295 - Changes addStatements and addFile
addStatements and addFile can now auto-commit triples based on a count for large loads of triples. Set the add commit size to a positive integer by calling setAddCommitSize or setting the addcommitsize property on the RepositoryConnection for size-based commits, or set to None or 0 for regular commit semantic on the add methods.
Using this feature can decrease the server's memory usage during loading.
Rfe9316 - RepositoryConnection.setRuleLanguage removed
RepositoryConnection.setRuleLanguage has been removed. Both addRules and loadRules already take an language parameter which defaults to QueryLanguage.PROLOG as the only language currently supported for rules.
Bug19001 - Changes to RepositoryConnection.clearNamespaces API
RepositoryConnect.clearNamespaces now supports a reset parameter (which defaults to True). clearNamespaces deletes all namespaces in the repository for the current user. If the reset argument is True, the user's namespaces are reset to the default set of namespaces, otherwise all namespaces are cleared.
Bug19004 - Make sure Python clients ping their session.
Previously, the thread to keep a dedicated session alive was created but not started, making it possible for Python clients to have their session time out on them.
AllegroGraph 4.0m2
General
The M2a release includes the following major functionality:
Full database backups and restores.
Text Indexer Enhancement for databases with more than 1 Billion text elements and strings with international characters.
Infiniband testing and performance document.
Deleted Triples Data structure improvement that removes the previous hard-coded limit on the number of triples that can be deleted per transaction.
Cluster Manager Design Document.
Jena functionality that was outlined as being highest priority.
Federation of AllegroGraph 4 databases
Support for Social Network Analysis
Support for two dimensional geospatial analysis.
Support for 48-bit triple IDs.
User definable placement of the transaction log.
There were many minor enhancements and bug fixes.
Note that AllegroGraph databases created with release m1 or m2 will have to be converted to a new format before use with version m2a. See Repository Upgrading.
Performance and Robustness
Improve logging by making some messages more concise and adding additional messages to improve diagnostics.
Improve and make string tables more robust
better consistency checks
compress large strings (>= 256 chars) in the string table
Added conversion tools for moving m1 triple-stores to m2
Support SIGUSR1-based reload in agraph-control script
After merging UPI tables, schedule a checkpoint to delete files which are no longer needed.
Safer file deletion for UPI tables to prevent problems with untunded databases.
Optimize UPI table merging, do not merge table portions that can't have common entries.
Improve client handling of instance process death
Improve upgrade checkpoint handling
Improved database versioning framework
Reference count indices so that they do not disappear out from under cursors
Rmprove transaction log upgrade handling
Writes to transaction log files are now aligned to the filesystem block size to allow for optimal performance when used with certain storage subsystems.
Control placement of transaction log (with TransactionLogDir)
The maximum number of triples that can be added to a triple store is now 2^48 (formerly 2^32).
Added TransactionLogDir config file parameter. This specifies where transaction log files should go. If not specified, defaults to the value of Main.
The agraph server process now responds to SIGUSR1 by reloading the server configuration file. The agraph-control script and /etc/init.d/agraph scripts now accept the 'reload' command and will send SIGUSR1 to the agraph server process.
Enhanced in-the-field debugging support. When requested by Franz tech support, you can send SIGUSR2 an AllegroGraph subprocess to get a backtrace for all threads.
Other Enhancements
configure-agraph now turns off terminal echo before prompting for the super user password. Also, since the password is no longer echoed, configure-agraph prompts for the password twice to make sure it has been entered properly.
support fractional seconds in time and dateTime
improve support for duplicate deletion during merge operations
improve and unify the printing of blank nodes
make the opening and re-opening of federated-triple-stores more robust
document group centrality
add API to auto-commit (on a per store basis)
add depth-first-search-path functor
add :plain encoding
improve handling of polygons in the geospatial sub-system
cleanup part printing
Remove line breaks in server log files to make automated processing and filtering easier.
Crash instance process if one of its helper processes crashes.
Detect death of instance process in client. Generate clear error message instead of hanging the client.
Add database versioning framework. Implement stepwise upgrade paths for databases that have been created with different software versions.
Fixed bugs and other corrections
Fix for bug18831: Agraph.cfg "overwrite" option appends to previous file instead of overwriting it as it should.
Fixed bug18774: Running "agraph-control start" twice would overwrite a valid pid file. This has been fixed.
SPARQL
Significant SPARQL performance improvements throughout, particularly in Left Joins (OPTIONAL), DISTINCT and REDUCED, and certain numeric FILTERs.
Massive performance increase for queries with both ORDER BY and a small LIMIT.
Imposed bindings (:with-variables) now apply to the output portion of SPARQL CONSTRUCT and DESCRIBE queries.
Some queries that use the SPOGI cache as an optimization could return duplicates. Fixed.
Similarly, the free-text index now returns distinct results in all cases, which avoids some unnecessary duplicate results.
Add the ability to use a graph URI, on both input and output, as a signifier for the default graph UPI of the current store.
Complete xsd:string/simple literal entailment. This allows you to seamlessly switch between using xsd:strings and plain literals without query changes. Plain literals are more efficient for the storage layer, and encouraged in all circumstances.
Improvements (in both efficiency and correctness) for SPARQL comparisons and XQuery functions when handling encoded values.
RDF/XML serializer can now handle on-the-fly namespace abbreviation, at the cost of more verbose output.
SPARQL extensions (e.g., geospatial forms) are now permitted by default.
URIs can now be used to specify geospatial subtypes, as well as the more verbose and less portable UUID form.
Java Client
Added Social Network Analysis capability.
Updated the Java Tutorial and tutorial examples to include Social Network Analysis.
Added Geospatial analysis capability.
Updated the Java Tutorial and tutorial examples to include Geospatial Analysis.
Established a Jena semantic framework API.
Created the Jean Tutorial, and a file of Jena tutorial examples.
Python Client
Added back support for SPOGI cache enabling/disabling
Enabled returning SPARQL-results+json or SPARQL-results+xml result from a query rather than a binding set
Introduced a boolean argument to openSession and session API for whether or not to load the initfile in the process handling the session
Federation support.
Introduced several options and better command-line option parsing for the load stress script
Minor updates to Python Tutorial and examples.
Changes to earlier releases
Significant changes in AllegroGraph 4.0 compared to earlier releases makes changes to earlier releases irrelevant to release 4.0 and later. Contact [email protected] if you have questions about releases prior to 4.0.