A quick intro to XQuery

XQuery is a highly malleable amalgam of an SQL-like syntax grafted on to XPath.

At its simplest, an XQuery expression that will likely look familiar is this:

      //book

This, of course, is standard XPath and is a complete, valid, self-contained XQuery query. It says to return a list of all book elements existing in whatever XML environment the particular query engine is set up to query against. Each query engine vendor determines the scope of the universe it knows about, which might be a single document, or several thousand. In the case of XQEngine, the environment is the sum total of all documents that have been previously indexed using XmlEngine.setDocument().

If the above example is querying against the small sample bibliography cited in the W3C's XML Query Use Cases document, we can recast it to point explicitly at that document using XQuery's built-in document() function:

      document("bib.xml")//book

This query returns exactly the same results as the first.

How the results get returned is again vendor-specific, but one possibility (the likeliest?) is that you'll be getting back a traditional, serialized XML text stream. You might get back DOM nodes, if you've requested that and the feature's supported by the engine you're using. If we assume for the moment we're getting back serialized XML, the above queries pull <book> elements out of the document and return something that looks like this:

   <book year="1994">
      <title>TCP/IP Illustrated</title>
      <author><last>Stevens</last><first>W.</first></author>
        ...

   <book year="1992">
      <title>Advanced Programming in the Unix environment</title>
        ...
              
   <book year="2000">
      <title>Data on the Web</title>
        ...
   etc.

We're returning a serialized nodeset. Whether that nodeset gets wrapped by a single enclosing element to form the root of a well-formed document tree -- it doesn't in this example -- is also up to the particular implementation. You might get back a series of unwrapped XML fragments as in the above -- and depending on the query, possibly something even less XML-ish than that. (The query "1" is a perfectly valid XQuery query, although it doesn't return anything that looks at all like XML.)

XPath expressions in XQuery look almost identical to those defined in the November, 1999 W3C XML Path Language (XPath) specification and employed gainfully in several different domains within the W3C world, most noticeably in XSLT. XQuery extends XPath's native syntax is several minor ways, most noticeably to introduce a RANGE form that brackets the "from" and "to" ends of a subscript range in predicates.

FLWR expressions

XQuery becomes most interesting when you go beyond simple XPath expressions and introduce FLWR expressions into the mix. FLWR (pronounced flower) is an acronym that stands for the four possible XQuery subexpressions this expression type can contain: FOR, LET, WHERE, and RETURN.

(These keywords, by the way, are case-insensitive. I generally uppercase them so that they stand out to better delineate a query's structure, but that's strictly a matter of personal preference.)

The productions in the XQuery grammar that formally define a FLWR expression look like this:

   FlwrExpr     ::= (ForClause | letClause)+ whereClause? returnClause
   ForClause    ::= 'FOR' Variable 'IN Expr (',' Variable IN Expr)*
   LetClause    ::= 'LET' Variable ':=' Expr (',' Variable := Expr)*
   WhereClause  ::= 'WHERE' Expr
   ReturnClause ::= 'RETURN' Expr

This, at least, is the formal definition in the original XQuery Working Draft of February, 2001. This might change, although it's unlikely to change greatly.

Reading this fragment of the grammar shouldn't cause any great difficulties if you're accustomed to the mysteries of DTD's. It says, in more colloquial terms, "A FLWR statement consists of a series of one or more FOR and/or LET clauses, followed by an optional WHERE and terminated by a mandatory RETURN". Note by the way that I tend to use the terms "clause," "statement," and "expression" somewhat indiscriminately in the following. This may cause some discomfort among the ranks of the theoretically correct.

Because the production labelled FlwrExpr says that a FOR or a LET will be followed eventually by a WHERE, and the WhereClause says that a WHERE is followed by an Expr -- an XQuery expression -- anything that's a valid expression, including another FlwrExpr, can be plugged in at the tail end of the chain, like an ever-lengthening sequence of Lego blocks. The definition of ReturnClause makes it just another snap-in Lego piece that can also be -- and by the grammar rules above, must be -- attached to the end of the chain.

This makes the FLWR production very malleable, highly recursive, and capable of generating a large number of possible query instances, including just about any combination of FOR, LET, WHERE, and RETURN statements imaginable. Some of them make more sense than others, in terms of their actually being useful programming constructs that model some real-world question. Some of them don't make much sense at all. We'll examine a few of the more useful permutations.

Before we do, let's look first at the individual LET and FOR subexpressions a FLWR expression can contain.

LET expressions

Remember our original query: //books? The same query, recast as a simple FLWR expression, might be

    LET $books := //book
    RETURN 
        $books

This expression says: "Evaluate the XPath expression //book, assign the resulting nodeset to the newly created variable $books, and then return that entire list to the user." The earlier XPath form is obviously more compact than this; there are other reasons, discussed below, for using the more verbose XQuery variant.

Indentation, like casing on keywords, is a matter of personal style.

We've just introduced the concept of a variable, an identifier starting with a dollar sign ('$'). In this case the variable $books is assigned an entire nodeset as its value. Once assigned -- bound in programming language parlance -- the value of a variable is fixed and can't be changed: variable usage in XQuery is, like Near Beer and other non-alcoholic look-alikes, side effect-free.

Although it doesn't make much sense in terms of good programming practise, for the sake of showing equivalence, we could just as easily rewrite the above expression as

    LET $books := //book
    RETURN 
        //books

In this case we've assigned the nodeset resulting from evaluating the XPath expression //book to the nodeset variable $books and then promptly disregarded the assignment, re-evaluating the original expression all over again and returning the resulting nodeset. Now that you know you can do that, don't.

FOR expressions

We can change our LET statement to a FOR:

    FOR $book IN //book
    RETURN
        $book

This is yet another variation on the same theme; we get back the same exact nodeset of books. How does this work? And why?

The FOR statement says to evaluate the XPath expression on the right-hand side of the IN keyword to produce a nodeset, and to then iterate through that nodeset one node at a time, assigning each node in turn to the variable on the left side of the IN. $book gets bound multiple times -- sounds like we're in an antiquarian book shop, doesn't it? -- once for each node in the nodeset.

Subsequent clauses are then executed once for each iteration, typically doing something useful with each new $book node as it's made available. The variable $book is visible to all clauses subordinate to the FOR statement where it's been defined.

The semantics of the RETURN statement terminating the FOR might be a little confusing at first glance. Like other clauses subordinate to a FOR, the RETURN statement is executed once for each iteration through the loop containing it, but the results it returns are returned only once, on final termination of the loop. Is that confusing? It works like this.

During each iteration, each successive node in the original nodeset is isolated in turn and assigned to the variable $book. The RETURN statement then evaluates its contents as a nodeset -- which in this case turns out to be the same exact node -- and adds that node to the end of a growing list of nodes that have already been evaluated. In more general terms, the contents of the RETURN will be evaluated on each new iteration as a multiple-node nodeset that then gets appended to the end of the resulting nodeset under construction. I know that's a mouthful.

Once the FOR has finished iterating, and only then, does the resulting nodeset that's been built up in the RETURN statement get serialized and returned in one fell swoop.

The result set that's constructed in the RETURN doesn't have to correspond one-on-one to the original nodeset you started with. In fact it would be pretty boring if that was always the case, although it worked out that way in this example. For example, referring back to bib.xml, the expression

    FOR $book IN //book
    RETURN
        $book/editor

is going to find only one book out of the four that has an editor, so that's the only node that will be returned. Here are the relevant parts of the file being queried against for reference:

   <book year="1994">
      <author><last>Stevens</last><first>W.</first></author>
		...
   </book>

   <book year="1992">
      <author><last>Stevens</last><first>W.</first></author>
		...
   </book>

   <book year="2000">
      <author><last>Abiteboul</last><first>Serge</first></author>
      <author><last>Buneman</last><first>Peter</first></author>		
      <author><last>Suciu</last><first>Dan</first></author>
		...
   </book>

   <book year="1999">
      <editor><last>Gerbarg</last><first>Darcy</first></editor>
		...
   </book>

In the above query, $book will be assigned to four times, once for each <book> in the nodeset that resulted from evaluating //book, but out of all four subsequent evaluations of the expression $book/editor in the RETURN statement, only one is going to produce a non-null, single node result. That single node ends up comprising the entire nodeset that gets constructed and eventually passed back up and out of the RETURN. The end result of all that work (and explanation!) is:

     <editor><last>Gerbarg</last><first>Darcy</first></editor>

FOR's can be nested, which shouldn't surprise us, given their Lego-like snap-together capability. We can say:

   FOR $book in //book
       FOR $author IN $book/author
       RETURN
           $author/last

The RETURN statement ends up getting executed five times. Let's count. $book gets assigned to four times in the top-level FOR. For each assignment, the variable $author in the second FOR gets assigned to a number of times which depends on how many authors each book contains. Enumerating, $author gets assigned, respectively, one <author> node, one <author> node, three <author> nodes, and no <author> nodes. Concatenate these all together in the RETURN statement and you get five author last names.

Again exploring a minor pedagogic variation, we can introduce an intermediate variable using a LET clause. We could say, for example:

   FOR $book in //book
   LET $authors := $book/author
       FOR $author IN $authors
       RETURN
           $author/last

This doesn't buy us much in this case, but in some circumstances it might help simplify the code.

Predicates and WHERE's

Coming soon to retail outlets everywhere.

Decorating the result-set

ditto.

created 25 apr 2001
last updated 10 may 2001