00_Intro.asciidoc

Full text search

Now that we have covered the simple case of searching for structured data, it is time to explore full text search — how to search within full text fields in order to find the most relevant documents.

The two most important aspects of full text search are:

Relevance

The ability to rank results by how relevant they are to the given query, whether relevance is calculated using TF/IDF (see [relevance-intro]), proximity to a geo-location, fuzzy similarity or some other algorithm.

Analysis

The process of converting a block of text into distinct, normalized tokens (see [analysis-intro]) in order to (a) create an inverted index and (b) query the inverted index.

As soon as we talk about either relevance or analysis, we are in the territory of queries, rather than filters.

Term-based vs. Full-text

While all queries perform some sort of relevance calculation, not all queries have an analysis phase. Besides specialized queries like the bool or function_score queries which don’t operate on text at all, textual queries can be broken down into two families:

Term-based queries

Queries like the term or fuzzy queries are low-level queries which have no analysis phase. They operate on a single term. A term query for the term "Foo" looks for that exact term in the inverted index and calculates the TF/IDF relevance _score for each document that contains the term.

It is important to remember that the term query looks in the inverted index for the exact term only — it won’t match any variants like "foo" or "FOO". It doesn’t matter how the term came to be in the index, just that it is. If you were to index ["Foo","Bar"] into an exact value not_analyzed field, or "Foo Bar" into an analyzed field with the whitespace analyzer, both would result in having the two terms "Foo" and "Bar" in the inverted index.

Full-text queries

Queries like the match or query_string queries are high-level queries which understand the mapping of a field:

• If you use them to query a date or integer field, they will treat the query string as a date or integer respectively.

• If you query an exact value (not_analyzed) string field, they will treat the whole query string as a single term.

• But if you query a full-text (analyzed) field, they will first pass the query string through the appropriate analyzer to produce the list of terms to be queried.

Once the query has assembled a list of terms, it executes the appropriate low-level query for each of these terms, then combines their results to produce the final relevance score for each document.

We will discuss this process in more detail in the following chapters.

You seldom need to use the term-based queries directly. Usually you want to query full text, not individual terms, and this is easier to do with the high-level full-text queries (which end up using term-based queries internally).

If you do find yourself wanting to use a query on an exact value not_analyzed field, think about whether you really want a query or a filter.

Single-term queries usually represent binary yes|no questions and are almost always better expressed as a filter, so that they can benefit from filter caching:

GET /_search
{
    "query": {
        "filtered": {
            "filter": {
                "term": { "gender": "female" }
            }
        }
    }
}