Skip to main content

TOKENS()

The TOKENS() function is the only function that you can use freely in the query without a SEARCH statement. A wrapping ANALYZER() call in a search expression does not affect the analyzer argument nor allow you to omit it.

Syntax

TOKENS(input, analyzer) → tokenArray

Split the input string with the help of the specified analyzer into an array. You can use the resulting array in FILTER or SEARCH statements with the IN operator.

KeyTypeDescription
inputstringText to tokenize.
analyzerstringName of an analyzer.

Example 1

Example query showcasing the "text_de" analyzer, which features tokenization with stemming, case conversion, and accent removal for German text:

RETURN TOKENS("Lörem ipsüm, DOLOR SIT Ämet.", "text_de")
[
[
"lor",
"ipsum",
"dolor",
"sit",
"amet"
]
]

Example 2

This example searches for documents where the text attribute contains certain tokens in any order:

FOR doc IN viewName
SEARCH ANALYZER(doc.text IN TOKENS("dolor amet lorem", "text_en"), "text_en")
RETURN doc

Alternatively, if you want to search for tokens in a particular order, use PHRASE() instead.

Example 3

When calling a TOKENS() function, you must always specify the analyzer name:

FOR doc IN viewName
SEARCH ANALYZER(doc.text IN TOKENS("foo", "text_en"), "text_en")
RETURN doc