BM25 ranked full-text search using the pg_textsearch extension.

BM25 is a modern ranking function that generally provides better relevance than traditional TF-IDF (used by full_text/5). It's particularly effective for top-k queries with LIMIT clauses due to Block-Max WAND optimization.

For detailed usage examples, performance tips, and migration guide, see the BM25 Search guide.

defmodule YourApp.Repo.Migrations.CreatePgTextsearchExtension do
  use Ecto.Migration

  def change do
    execute "CREATE EXTENSION IF NOT EXISTS pg_textsearch", "DROP EXTENSION IF EXISTS pg_textsearch"

    # Create BM25 index with language configuration
    execute """
    CREATE INDEX posts_body_bm25_idx ON posts
    USING bm25(body) WITH (text_config='english')
    """, "DROP INDEX posts_body_bm25_idx"
  end
end

Options

• :order - Ordering of results. Note that BM25 returns negative scores (lower is better):
  • :asc (default) - orders by score ascending (best matches first)
  • :desc - orders by score descending (worst matches first)
  • :none - no ordering applied
• :index_name - Explicit index name. Required when using score_threshold.
• :score_key - Atom key to select the BM25 score into the result map. The score is merged via select_merge/3, so the query needs to select a map before calling bm25/5 (e.g. select([p], %{body: p.body})).
  • :none (default) - score is not selected
  • atom - selects score as this key
• :score_threshold - Post-filter results by BM25 score (applied after ORDER BY). Since scores are negative and lower is better, use negative thresholds (e.g., -3.0 keeps only results with score < -3.0, i.e., scores like -4.0, -5.0 which are better matches). May return fewer results than LIMIT.
• :pre_filter - Whether to exclude non-matching rows.
  • false (default) - no pre-filtering
  • true - adds a WHERE score < 0 clause to exclude non-matches
• :highlight - a keyword list of result keys to columns to highlight the term's matches in, e.g. highlight: [body: p.body]. See highlight/3.

Examples

Basic search - returns top 10 most relevant posts:

Post
|> Torus.bm25([p], p.body, "database search")
|> limit(10)
|> select([p], p.body)
|> Repo.all()

With score selection (select a map before calling, so the score has somewhere to merge into):

Post
|> select([p], %{body: p.body})
|> Torus.bm25([p], p.body, "database", score_key: :relevance)
|> limit(5)
|> Repo.all()
# => [%{body: "...", relevance: -2.5}, ...]

With WHERE clause pre-filtering:

Post
|> where([p], p.category_id == 123)
|> Torus.bm25([p], p.body, "database")
|> limit(10)
|> Repo.all()

With score threshold (post-filtering, may return fewer than LIMIT, index_name is required):

Post
|> Torus.bm25([p], p.body, "database", score_threshold: -5.0, index_name: "posts_body_idx")
|> limit(10)
|> Repo.all()

When to use bm25/5 vs full_text/5

Use bm25/5 when:

• You need better relevance ranking than TF-IDF
• You need faster search with large datasets
• You have large result sets with LIMIT (top-k queries)
• Single column search is sufficient
• You're on PostgreSQL 17+

Use full_text/5 when:

• You need multi-column search with different weights per column
• You want to use stored tsvector columns
• You're on PostgreSQL < 17
• You need the concat filter type

Index options

BM25 indexes support these parameters in the WITH clause:

• text_config - PostgreSQL text search configuration (required). This determines the language/stemming rules. Available configs: 'english', 'french', 'german', 'simple' (no stemming), etc. Run SELECT cfgname FROM pg_ts_config; to list all.
• k1 - Term frequency saturation (default: 1.2, range: 0.1-10.0)
• b - Length normalization (default: 0.75, range: 0.0-1.0)

CREATE INDEX custom_idx ON documents
USING bm25(content)
WITH (text_config='english', k1=1.5, b=0.8);

Performance tips

• BM25 is most efficient with ORDER BY + LIMIT (enables Block-Max WAND optimization)
• For filtered searches, create a separate B-tree index on the filter column
• Pre-filtering works best when the filter is selective (<10% of rows)
• Post-filtering with score_threshold may return fewer results than LIMIT

Full text search with rank ordering. Accepts a list of columns to search in. A list of columns can either be a text or tsvector type. If tsvectors are passed make sure to set stored: true.

Cleans the term, so it can be input directly by the user. The default preset of settings is optimal for most cases.

Full Text Searching (or just text search) provides the capability to identify natural-language documents that satisfy a query, and optionally to sort them by relevance to the query. The most common type of search is to find all documents containing given query terms and return them in order of their similarity to the query. Notions of query and similarity are very flexible and depend on the specific application. The simplest search considers query as a set of words and similarity as the frequency of query words in the document. Read more in PostgreSQL Full Text Search docs.

Options

• :prefix_search - whether to apply prefix search.
  • true (default) - the term is treated as a prefix
  • false - only counts full-word matches
• :stored - whether to use stored tsvector or not.
  • false (default) - columns (or expressions) passed as qualifiers are of type text
  • true - columns (or expressions) passed as qualifiers are tsvectors
• :language - language used for the search. Defaults to "english".
• :term_function - function used to convert the term to ts_query. Can be one of:
  • :websearch_to_tsquery (default) - converts term to a tsquery, normalizing words according to the specified or default configuration. Quoted word sequences are converted to phrase tests. The word “or” is understood as producing an OR operator, and a dash produces a NOT operator; other punctuation is ignored. This approximates the behavior of some common web search tools.
  • :plainto_tsquery - converts term to a tsquery, normalizing words according to the specified or default configuration. Any punctuation in the string is ignored (it does not determine query operators). The resulting query matches documents containing all non-stopwords in the term.
  • :phraseto_tsquery - converts term to a tsquery, normalizing words according to the specified or default configuration. Any punctuation in the string is ignored (it does not determine query operators). The resulting query matches phrases containing all non-stopwords in the text.
• :rank_function - function used to rank the results.
  • :ts_rank_cd (default) - computes a score showing how well the vector matches the query, using a cover density algorithm. See Ranking Search Results for more details.
  • :ts_rank - computes a score showing how well the vector matches the query.
• :rank_weights - a list of weights for each column. Defaults to [:A, :B, :C, :D], padded with :D when there are more than four columns. If provided, it should have at least as many weights as there are columns. A single weight can be either a string or an atom. Possible values are:
  • :A - 1.0
  • :B - 0.4
  • :C - 0.2
  • :D - 0.1
• :rank_normalization - an integer that specifies whether and how a document's length should impact its rank. The integer option controls several behaviors, so it is a bit mask: you can specify one or more behaviors using | (for example, 2|4).
  • 0 - ignores the document length
  • 1 (default for ts_rank) - divides the rank by 1 + the logarithm of the document length
  • 2 - divides the rank by the document length
  • 4 (default for ts_rank_cd) - divides the rank by the mean harmonic distance between extents (this is implemented only by ts_rank_cd)
  • 8 - divides the rank by the number of unique words in document
  • 16 - divides the rank by 1 + the logarithm of the number of unique words in document
  • 32 - divides the rank by itself + 1
• :order - describes the ordering of the results. Possible values are
  • :desc (default) - orders the results by similarity rank in descending order.
  • :asc - orders the results by similarity rank in ascending order.
  • :none - doesn't apply ordering at all.
• :filter_type - filter type
  • :or (default) - uses OR operator to combine different column matches. Selecting this option means that the search term won't match across columns.
  • :concat - joins the columns into a single tsvector and searches for the term in the concatenated string containing all columns.
  • :none - doesn't apply any filtering and returns all results.
• :empty_return - whether to return all results when the search term is empty.
  • true (default) - returns all results when the search term is empty.
  • false - returns an empty list when the search term is empty.
• :highlight - a keyword list of result keys to columns to highlight the term's matches in, e.g. highlight: [title: p.title]. See highlight/3.
• :coalesce - when joining multiple columns via filter_type: :concat, wraps each column in a COALESCE function to handle NULL values. Only applies with filter_type: :concat and more than one column - it is ignored otherwise.
  • true (default) - adds COALESCE
  • false - doesn't add COALESCE to the query. Choose this when you can guarantee that all columns are non-null.

Example usage

iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...>  Post
...> |> Torus.full_text([p], [p.title, p.body], "uncov hogwar")
...> |> select([p], p.title)
...> |> Repo.all()
["Diagon Bombshell"]

Optimizations

• Store precomputed tsvector in a separate column, add a GIN index to it, and use stored: true.

• Add a GIN tsvector index on the column(s) you search in. Use Torus.QueryInspector.tap_sql/2 on your query (with all the options passed) to see the exact search string and add an index to it. For example for nullable title, the GIN index could look like:

  CREATE INDEX index_gin_posts_title
  ON posts USING GIN (to_tsvector('english', COALESCE(title, '')));

Highlights matches of term in qualifier using PostgreSQL ts_headline.

Use it in select/select_merge alongside any search macro. Two highlighting types are supported via the :type option:

• :word (default) - word-based, uses the same term parsing as full_text/5, so it pairs naturally with full_text/5, bm25/5, and hybrid/4. It also highlights the exact-word matches of similarity/5 (though not its fuzzy matches).
• :substring - highlights every occurrence of the term as a plain substring (implemented with regexp_replace, the term is regex-escaped). Pairs with ilike/5 and like/5, whose %term% patterns match inside words where word-based highlighting finds nothing.

semantic/5 matches aren't lexical, so there is nothing to highlight there.

Highlighting from the search macros

Instead of repeating the term, pass highlight: [result_key: column] directly to full_text/5, bm25/5, similarity/5, ilike/5, like/5, or a hybrid/4 branch's options - the search's own term and options are reused. ilike/5/like/5 highlight as substrings with the macro's case sensitivity, stripping %/_ wildcards from the term; the rest highlight word matches. Only leading/trailing wildcards translate cleanly: a wildcard in the middle of the term (or an escaped \%) is stripped too, so the remaining substring may no longer occur in the text and nothing gets highlighted - for such patterns call highlight/3 yourself with the substring you want marked. The highlighted value is merged via select_merge/3, so either use a key that exists on the selected struct (its value is replaced with the highlighted text) or select a map before the search macro.

iex> insert_post!(title: "Hogwarts Shocker")
...> Post
...> |> Torus.full_text([p], [p.title], "shocker", highlight: [title: p.title])
...> |> Repo.all()
...> |> Enum.map(& &1.title)
["Hogwarts <b>Shocker</b>"]

iex> insert_post!(title: "Hogwarts Shocker")
...> Post
...> |> Torus.ilike([p], [p.title], "%ogwart%", highlight: [title: p.title])
...> |> Repo.all()
...> |> Enum.map(& &1.title)
["H<b>ogwart</b>s Shocker"]

Options

• :type - :word (default) or :substring, see above.
• :start_sel, :stop_sel - strings the matches are wrapped in. Default to "<b>" and "</b>".

Options for type: :substring:

• :case_sensitive - defaults to false (matching ilike/5); set to true to only highlight exact-case occurrences (matching like/5).

Options for type: :word:

• :language - language used for the search. Defaults to "english".
• :term_function - function used to convert the term to ts_query. Same options as in full_text/5. Defaults to :websearch_to_tsquery.
• :prefix_search - whether to also highlight words the term matches as a prefix. Defaults to true (same as full_text/5).
• :highlight_all - whether to return the whole document.
  • true (default) - returns the full text with all matches highlighted.
  • false - returns a fragment (snippet) around the matches, controlled by the options below.
• :max_words, :min_words - fragment size when highlight_all: false. Default to PostgreSQL's 35 and 15.
• :short_word - words of this length or less are dropped at fragment start/end. Defaults to 3.
• :max_fragments - maximum number of fragments to return. Defaults to 0.
• :fragment_delimiter - string used to join fragments. Defaults to " ... ".

Examples

iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> Post
...> |> Torus.full_text([p], [p.title, p.body], "shocker")
...> |> select([p], Torus.highlight(p.title, "shocker"))
...> |> Repo.all()
["Hogwarts <b>Shocker</b>"]

With a snippet instead of the full text:

iex> insert_post!(body: "A magic spell disrupts the Quidditch Cup final at Hogwarts.")
...> Post
...> |> Torus.full_text([p], [p.body], "quidditch")
...> |> select([p], Torus.highlight(p.body, "quidditch", highlight_all: false, max_words: 5, min_words: 2))
...> |> Repo.all()
["<b>Quidditch</b> Cup final"]

Substring highlighting alongside ilike/5:

iex> insert_post!(title: "Hogwarts Shocker")
...> Post
...> |> Torus.ilike([p], [p.title], "%ogwart%")
...> |> select([p], Torus.highlight(p.title, "ogwart", type: :substring))
...> |> Repo.all()
["H<b>ogwart</b>s Shocker"]