Questions about unaggregated variables and the usage of variables in GROUP BY

[nkaralis]

Version

6.0.0

Question

Hello.

I would like to ask some questions about the handling of unaggregated variables.

For the examples provided below, I have used Jena v6.0.0 (Fuseki and ARQ) and the following graph.

<http://www.example.org/s2> <http://www.example.org/p1> <http://www.example.org/o3>.
<http://www.example.org/s2> <http://www.example.org/p2> "test literal".
<http://www.example.org/s2> <http://www.example.org/p2> <http://www.example.org/o2>.
<http://www.example.org/s1> <http://www.example.org/p1> <http://www.example.org/o1>.
<http://www.example.org/s3> <http://www.example.org/p1> <http://www.example.org/o5>.
<http://www.example.org/s3> <http://www.example.org/p2> <http://www.example.org/o6>.

In the paragraph 11.4 of the specification, it is stated that "In a query level which uses aggregates, only expressions consisting of aggregates and constants may be projected, with one exception. When GROUP BY is given with one or more simple expressions consisting of just a variable, those variables may be projected from the level."

Based on the above statement, the queries

Q1. SELECT ?s (COUNT(?o) AS ?c) WHERE { ?s ?p ?o } and

Q2. SELECT (?s AS ?s2) (COUNT(?o) AS ?c) WHERE { ?s ?p ?o }

raise the errors "Non-group key variable in SELECT: ?s" and "Non-group key variable in SELECT: ?s in expression ?s", respectively.

However, in the paragraph 18.2.4.1, the provided algorithm does not follow the statement of paragraph 11.4, as it replaces an unaggregated V variable with SAMPLE(V)

First question: Which statement should be followed?

Second question: Based on the algorithm of paragraph 18.2.4.1, should Q2 be rejected?

Third question: Why does the algorithm treat (?X AS VAR) and ?X differently?

Fourth question: When it comes to the cases of ORDER BY(X) and HAVING(X), shouldn't the queries Q3 and Q4 (provided below) return the same results?

Q3. SELECT ?s (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY ?s HAVING(ISIRI(?p)) (empty result set)

Q4. SELECT ?s (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY ?s HAVING(ISIRI(SAMPLE(?p))) (explicit SAMPLE, three solutions)

---

I also have a few questions about the variables appearing in GROUP BY.

In the table provided in paragraph 18.2.1 of the specification, it is stated that V is in-scope given an expression GROUP BY (expr AS v).

Fifth Question: Should the query Q5 (provided below) throw a syntax error?

Q5. SELECT (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY (1 AS ?c) # empty result set

Sixth Question: Since the GROUP BY is processed before the SELECT clause, is ?c always assigned the UNDEF value in the Group in Q6 (provided below)?

Q6. SELECT (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY ?c # one solution

[afs]

Questions about the spec should go to https://github.com/w3c/sparql-query. Jena is just one implementation.

These are all static issues (parse, algebra build) about the query.

In the paragraph 11.4 of the specification, it is stated that "In a query level which uses aggregates, only expressions consisting of aggregates and constants may be projected, with one exception. When GROUP BY is given with one or more simple expressions consisting of just a variable, those variables may be projected from the level."

Based on the above statement, the queries Q1. SELECT ?s (COUNT(?o) AS ?c) WHERE { ?s ?p ?o } and Q2. SELECT (?s AS ?s2) (COUNT(?o) AS ?c) WHERE { ?s ?p ?o } raise the errors "Non-group key variable in SELECT: ?s" and "Non-group key variable in SELECT: ?s in expression ?s", respectively. However, in the paragraph 18.2.4.1, the provided algorithm does not follow the statement of paragraph 11.4, as it replaces an unaggregated V variable with SAMPLE(V)

First question: Which statement should be followed?

The 11.4 text.

The SAMPLE text is a known error in SPARQL Query which will be addressed in SPARQL 1.2.
w3c/sparql-query#165

Second question: Based on the algorithm of paragraph 18.2.4.1, should Q2 be rejected?

Not applicable. Implicit sample is not intended.

Third question: Why does the algorithm treat (?X AS VAR) and ?X differently?

It (as in 11.4) doesn't.
(?s AS ?s2) is rejected because of the expression part, ?s, not because of the assigned variable ?s2.

Fourth question: When it comes to the cases of ORDER BY(X) and HAVING(X), shouldn't the queries Q3 and Q4 (provided below) return the same results?

That's two questions 😄

Q3. SELECT ?s (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY ?s HAVING(ISIRI(?p)) (empty result set)

HAVING is a filter after the aggregate step and ?p not passed out of the GROUP BY/aggregation step. ISIRI(?p) is an error hence the filter is false.

Q4. SELECT ?s (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY ?s HAVING(ISIRI(SAMPLE(?p))) (explicit SAMPLE, three solutions)

An expression in HAVING and GROUP BY can use aggregates (just like in the SELECT clause) which adds to the aggregates over the GROUP BY. In HAVING, that's not very useful but the spec is the spec.

SAMPLE(?p) adds an aggregation to the grouping so SAMPLE(?p) is available after aggregation as a value and the ?p is from within the groups.

I also have a few questions about the variables appearing in GROUP BY.

In the table provided in paragraph 18.2.1 of the specification, it is stated that V is in-scope given an expression GROUP BY (expr AS v).

Fifth Question: Should the query Q5 (provided below) throw a syntax error?
Q5. SELECT (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY (1 AS ?c) # empty result set

No - it is legal.

Aggregates can involve any of group-by variables - the variable will be same across all rows within the group.

Sixth Question: Since the GROUP BY is processed before the SELECT clause, is ?c always assigned the UNDEF value in the Group in Q6 (provided below)?
Q6. SELECT (COUNT(DISTINCT ?o) AS ?c) WHERE { ?s ?p ?o } GROUP BY ?c # one solution

UNDEF is never assigned. Variables are "not in the solution mapping".

"Since the GROUP BY is processed before the SELECT clause," -- not really, the SELECT clause has several different roles; the aggregates get applied to the group (Jena uses hidden variables) then these can be used in expressions.

GROUP BY ?c is an expression use of ?c, not a declaration (no AS). There is no ?c in the grouping data unlike (1 AS ?c).

[nkaralis]

Thank you very much for the detailed response!

HAVING is a filter after the aggregate step and ?p not passed out of the GROUP BY/aggregation step. ISIRI(?p) is an error hence the filter is false.

Does this mean that a parsing error should be thrown only if there is a non-aggregated variable in the SELECT clause?

Regarding Q5, my question was about the double assignment to variable ?c. Is it legal? Section 19.8 states: "12. Variables introduced by AS in a SELECT clause must not already be in-scope."

[afs]

I think Q5 is a Jena bug. ?c should be in-scope at the start of the SELECT expressions. It look like group keys are not checked properly.

A simpler case of the bug is

SELECT (3 AS ?z) WHERE {} GROUP BY ?z

[nkaralis]

I agree that SELECT (3 AS ?z) WHERE {} GROUP BY ?z should be rejected, but, according to the standard it is valid, right? Note that only (expr AS v) is considered for the scope of GROUP BY.

[afs]

That part of the spec may well allow it but it leads to reassignment which isn't valid.