tokenizers

For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt

<!-- Tab options on this page. Append to the .md URL to filter: ?tabs=<id,...> select specific tabs (e.g. ?tabs=nodejs,shell) ?allTabs=true include every tab (no param) default: one tab per tabset

Available tabs: other tabs: vib, jsoneditor, visual-editor, json-editor -->

Tokenizers for Custom Search Analyzers

A custom analyzer's tokenizer determines how MongoDB Search splits up text into discrete chunks for indexing. Tokenizers require a type field, and some take additional options as well.

"tokenizer": {
  "type": "<tokenizer-type>",
  "<additional-option>": "<value>"
}

The sample index definitions and queries on this page use the [sample collection](https://www.mongodb.com/docs/search/indexes/analyzers/custom.md#std-label-custom-analyzers-eg-coll named minutes. To try these examples, load the minutes collection on your cluster and navigate to the Create a Search Index page in the Atlas UI following the steps in the [Create a MongoDB Search Index](https://www.mongodb.com/docs/search/tutorial.md#std-label-create-fts-index tutorial. Then, select the minutes collection as your data source, and follow the example procedure to create an index from the Atlas UI or using [mongosh.](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh

edgeGram

The edgeGram tokenizer tokenizes input from the left side, or “edge”, of a text input into n-grams of given sizes. You can't use a custom analyzer with [edgeGram](https://www.mongodb.com/docs/search/indexes/analyzers/tokenizers.md#std-label-edgeGram-tokenizer-ref tokenizer in the analyzer field for [synonym](https://www.mongodb.com/docs/search/indexes/synonyms.md#std-label-synonyms-ref or [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete [field mapping definitions.](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-fts-field-mappings

Attributes

The edgeGram tokenizer has the following attributes:

Note:

The edgeGram tokenizer yields multiple output tokens per word and across words in input text, producing token graphs.

Because autocomplete.

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named edgegramExample. It uses the edgeGram tokenizer to create tokens (searchable terms) between 2 and 7 characters long starting from the first character on the left side of words in the message field.

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type edgegramExample in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select edgeGram from the dropdown and type the value for the following fields:

1. Click Add to add the custom analyzer to your index.

2. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

3. Select message from the Field Name dropdown and String from the Data Type dropdown.

4. In the properties section for the data type, select edgegramExample from the Index Analyzer and Search Analyzer dropdowns.

5. Click Add, then Save Changes.

db.minutes.createSearchIndex(
  "default",
  {
    "mappings": {
      "dynamic": true,
      "fields": {
        "message": {
          "analyzer": "edgegramExample",
          "type": "string"
        }
      }
    },
    "analyzers": [
      {
        "charFilters": [],
        "name": "edgegramExample",
        "tokenFilters": [],
        "tokenizer": {
          "maxGram": 7,
          "minGram": 2,
          "type": "edgeGram"
        }
      }
    ]
  }
)

The following query searches the message field in the minutes collection for text that begin with tr.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "text": { "query": "tr", "path": "message" } } }

Output:

```js SCORE: 0.3150668740272522 _id: "1" message: "try to siGn-In" page_updated_by: Object last_name: "AUERBACH" first_name: "Siân" email: "auerbach@example.com" phone: "(123)-456-7890" text: Object en_US: " This page deals with department meetings." sv_FI: "Den här sidan behandlar avdelningsmöten" fr_CA: "Cette page traite des réunions de département"

SCORE: 0.3150668740272522 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "We'll head out to the conference room by noon." ```

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "tr",
        "path": "message"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "message": 1
    }
  }
])

Output:

{ _id: 1, message: 'try to siGn-In' },
{ _id: 3, message: 'try to sign-in' }

MongoDB Search returns documents with _id: 1 and _id: 3 in the results because MongoDB Search created a token with the value tr using the edgeGram tokenizer for the documents, which matches the search term. If you index the message field using the standard tokenizer, MongoDB Search would not return any results for the search term tr.

The following table shows the tokens that the edgeGram tokenizer and by comparison, the standard tokenizer, create for the documents in the results:

Use the edgeGram tokenizer when your search terms begin at the start of a word, such as a search for tr matching try. Use the [nGram](https://www.mongodb.com/docs/search/indexes/analyzers/tokenizers.md#std-label-nGram-tokenizer-ref tokenizer when your search terms can begin in the middle of a word, such as a search for eek matching weekly.

keyword

The keyword tokenizer tokenizes the entire input as a single token. MongoDB Search doesn't index string fields that exceed 32766 characters using the keyword tokenizer.

Attributes

The keyword tokenizer has the following attributes:

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named keywordExample. It uses the keyword tokenizer to create a token (searchable terms) on the entire field as a single term.

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type keywordExample in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select keyword from the dropdown.

6. Click Add to add the custom analyzer to your index.

7. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

8. Select message from the Field Name dropdown and String from the Data Type dropdown.

9. In the properties section for the data type, select keywordExample from the Index Analyzer and Search Analyzer dropdowns.

10. Click Add, then Save Changes.

db.minutes.createSearchIndex(
  "default",
  {
    "mappings": {
      "dynamic": true,
      "fields": {
        "message": {
          "analyzer": "keywordExample",
          "type": "string"
        }
      }
    },
    "analyzers": [
      {
        "charFilters": [],
        "name": "keywordExample",
        "tokenFilters": [],
        "tokenizer": {
          "type": "keyword"
        }
      }
    ]
  }
)

The following query searches the message field in the minutes collection for the phrase try to sign-in.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "text": { "query": "try to sign-in", "path": "message" } } }

Output:

js SCORE: 0.5472603440284729 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "<body>We'll head out to the conference room by noon.</body>"

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "try to sign-in",
        "path": "message"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "message": 1
    }
  }
])

Output:

{ _id: 3, message: 'try to sign-in' }

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value try to sign-in using the keyword tokenizer for the documents, which matches the search term. If you index the message field using the standard tokenizer, MongoDB Search returns documents with _id: 1, _id: 2 and _id: 3 for the search term try to sign-in because each document contains some of the tokens the standard tokenizer creates.

The following table shows the tokens that the keyword tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

nGram

The nGram tokenizer tokenizes into text chunks, or “n-grams”, of given sizes. You can't use a custom analyzer with [nGram](https://www.mongodb.com/docs/search/indexes/analyzers/tokenizers.md#std-label-nGram-tokenizer-ref tokenizer in the analyzer field for [synonym](https://www.mongodb.com/docs/search/indexes/synonyms.md#std-label-synonyms-ref or [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete [field mapping definitions.](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-fts-field-mappings

Attributes

The nGram tokenizer has the following attributes:

Example

The following index definition indexes the title field in the minutes collection using a custom analyzer named ngramExample. It uses the nGram tokenizer to create tokens (searchable terms) between 4 and 6 characters long in the title field.

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type ngramAnalyzer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select nGram from the dropdown and type the value for the following fields:

1. Click Add to add the custom analyzer to your index.

2. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

3. Select message from the Field Name dropdown and String from the Data Type dropdown.

4. In the properties section for the data type, select ngramAnalyzer from the Index Analyzer and Search Analyzer dropdowns.

5. Click Add, then Save Changes.

db.minutes.createSearchIndex("default", {
  "mappings": {
    "dynamic": true,
    "fields": {
      "title": {
        "analyzer": "ngramExample",
        "type": "string"
      }
    }
  },
  "analyzers": [
    {
      "charFilters": [],
      "name": "ngramExample",
      "tokenFilters": [],
      "tokenizer": {
        "maxGram": 6,
        "minGram": 4,
        "type": "nGram"
      }
    }
  ]
})

The following query searches the title field in the minutes collection for the term ekly.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "index": "default", "text": { "query": "ekly", "path": "title" } } }

Output:

js SCORE: 0.5895273089408875 _id: "1" message: "try to siGn-In" page_updated_by: Object last_name: "AUERBACH" first_name: "Siân" email: "auerbach@example.com" phone: "(123)-456-7890" text: Object en_US: "<head> This page deals with department meetings.</head>" sv_FI: "Den här sidan behandlar avdelningsmöten" fr_CA: "Cette page traite des réunions de département"

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "ekly",
        "path": "title"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "title": 1
    }
  }
])

Output:

{ _id: 1, title: "The team's weekly meeting" }

MongoDB Search returns the document with _id: 1 in the results because MongoDB Search created a token with the value ekly using the nGram tokenizer for the documents, which matches the search term. If you index the title field using the standard or edgeGram tokenizer, MongoDB Search would not return any results for the search term ekly.

The following table shows the tokens that the nGram tokenizer and by comparison, the standard and edgeGram tokenizer create for the document with _id: 1:

regexCaptureGroup

The regexCaptureGroup tokenizer matches a Java regular expression pattern to extract tokens.

Tip:

To learn more about Java regular expression syntax, see the Pattern class in the Java documentation.

Attributes

The regexCaptureGroup tokenizer has the following attributes:

Example

The following index definition indexes the page_updated_by.phone field in the minutes collection using a custom analyzer named phoneNumberExtractor. It uses the following:

• mappings character filter to remove parenthesis around the first three digits and replace all spaces and periods with dashes

• regexCaptureGroup tokenizer to create a single token from the first US-formatted phone number present in the text input

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type phoneNumberExtractor in the Analyzer Name field.

4. Expand Character Filters and click Add character filter.

5. Select mapping from the dropdown and click Add mapping.

6. Enter the following characters in the Original field, one at a time, and leave the corresponding Replacement field empty.

1. Click Add character filter.

2. Expand Tokenizer if it's collapsed.

3. Select regexCaptureGroup from the dropdown and type the value for the following fields:

1. Click Add to add the custom analyzer to your index.

2. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.phone field.

3. Select page_updated_by.phone from the Field Name dropdown and String from the Data Type dropdown.

4. In the properties section for the data type, select phoneNumberExtractor from the Index Analyzer and Search Analyzer dropdowns.

5. Click Add, then Save Changes.

db.minutes.createSearchIndex("default", {
  "mappings": {
    "dynamic": true,
    "fields": {
      "page_updated_by": {
        "fields": {
          "phone": {
            "analyzer": "phoneNumberExtractor",
            "type": "string"
          }
        },
        "type": "document"
      }
    }
  },
  "analyzers": [
    {
      "charFilters": [
        {
          "mappings": {
            " ": "-",
            "(": "",
            ")": "",
            ".": "-"
          },
          "type": "mapping"
        }
      ],
      "name": "phoneNumberExtractor",
      "tokenFilters": [],
      "tokenizer": {
        "group": 0,
        "pattern": "^\\b\\d{3}[-]?\\d{3}[-]?\\d{4}\\b$",
        "type": "regexCaptureGroup"
        }
    }
  ]
})

The following query searches the page_updated_by.phone field in the minutes collection for the phone number 123-456-9870.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "index": "default", "text": { "query": "123-456-9870", "path": "page_updated_by.phone" } } }

Output:

js SCORE: 0.5472603440284729 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "<body>We'll head out to the conference room by noon.</body>"

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "123-456-9870",
        "path": "page_updated_by.phone"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "page_updated_by.phone": 1
    }
  }
])

Output:

{ _id: 3, page_updated_by: { phone: '(123).456.9870' }

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value 123-456-7890 using the regexCaptureGroup tokenizer for the documents, which matches the search term. If you index the page_updated_by.phone field using the standard tokenizer, MongoDB Search returns all of the documents for the search term 123-456-7890.

The following table shows the tokens that the regexCaptureGroup tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

regexSplit

The regexSplit tokenizer splits tokens with a Java regular-expression based delimiter.

Tip:

To learn more about Java regular expression syntax, see the Pattern class in the Java documentation.

Attributes

The regexSplit tokenizer has the following attributes:

Example

The following index definition indexes the page_updated_by.phone field in the minutes collection using a custom analyzer named dashDotSpaceSplitter. It uses the regexSplit tokenizer to create tokens (searchable terms) from one or more hyphens, periods and spaces on the page_updated_by.phone field.

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type dashDotSpaceSplitter in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select regexSplit from the dropdown and type the value for the following field:

1. Click Add to add the custom analyzer to your index.

2. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.phone field.

3. Select page_updated_by.phone from the Field Name dropdown and String from the Data Type dropdown.

4. In the properties section for the data type, select dashDotSpaceSplitter from the Index Analyzer and Search Analyzer dropdowns.

5. Click Add, then Save Changes.

db.minutes.createSearchIndex("default", {
  "mappings": {
    "dynamic": true,
    "fields": {
      "page_updated_by": {
        "fields": {
          "phone": {
            "analyzer": "dashDotSpaceSplitter",
            "type": "string"
          }
        },
        "type": "document"
      }
    }
  },
  "analyzers": [
    {
      "charFilters": [],
      "name": "dashDotSpaceSplitter",
      "tokenFilters": [],
      "tokenizer": {
        "pattern": "[-. ]+",
        "type": "regexSplit"
      }
    }
  ]
})

The following query searches the page_updated_by.phone field in the minutes collection for the digits 9870.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "index": "default", "text": { "query": "9870", "path": "page_updated_by.phone" } } }

Output:

js SCORE: 0.5472603440284729 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "<body>We'll head out to the conference room by noon.</body>"

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "9870",
        "path": "page_updated_by.phone"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "page_updated_by.phone": 1
    }
  }
])

Output:

{ _id: 3, page_updated_by: { phone: '(123).456.9870' }

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value 9870 using the regexSplit tokenizer for the documents, which matches the search term. If you index the page_updated_by.phone field using the standard tokenizer, MongoDB Search would not return any results for the search term 9870.

The following table shows the tokens that the regexCaptureGroup tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

standard

The standard tokenizer tokenizes based on word break rules from the Unicode Text Segmentation algorithm.

Attributes

The standard tokenizer has the following attributes:

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named standardExample. It uses the standard tokenizer and the [stopword token filter.](https://www.mongodb.com/docs/search/indexes/analyzers/token-filters.md#std-label-stopword-tf-ref

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type standardExample in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select standard from the dropdown.

6. Click Add to add the custom analyzer to your index.

7. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

8. Select message from the Field Name dropdown and String from the Data Type dropdown.

9. In the properties section for the data type, select standardExample from the Index Analyzer and Search Analyzer dropdowns.

10. Click Add, then Save Changes.

db.minutes.createSearchIndex(
  "default",
  {
    "mappings": {
      "dynamic": true,
      "fields": {
        "message": {
          "analyzer": "standardExample",
          "type": "string"
        }
      }
    },
    "analyzers": [
      {
        "charFilters": [],
        "name": "standardExample",
        "tokenFilters": [],
        "tokenizer": {
          "type": "standard"
        }
      }
    ]
  }
)

The following query searches the message field in the minutes collection for the term signature.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "text": { "query": "signature", "path": "message" } } }

Output:

js SCORE: 0.5376965999603271 _id: "4" message: "write down your signature or phone №" page_updated_by: Object last_name: "LEVINSKI" first_name: "François" email: "levinski@example.com" phone: "123-456-8907" text: Object en_US: "<body>This page has been updated with the items on the agenda.</body>" es_MX: "La página ha sido actualizada con los puntos de la agenda." pl_PL: "Strona została zaktualizowana o punkty porządku obrad."

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "signature",
        "path": "message"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "message": 1
    }
  }
])

Output:

{ _id: 4, message: 'write down your signature or phone №' }

MongoDB Search returns the document with _id: 4 because MongoDB Search created a token with the value signature using the standard tokenizer for the documents, which matches the search term. If you index the message field using the keyword tokenizer, MongoDB Search would not return any results for the search term signature.

The following table shows the tokens that the standard tokenizer and by comparison, the keyword analyzer, create for the document with _id: 4:

uaxUrlEmail

The uaxUrlEmail tokenizer tokenizes URL (Uniform Resource Locator)s and email addresses. Although uaxUrlEmail tokenizer tokenizes based on word break rules from the Unicode Text Segmentation algorithm, we recommend using uaxUrlEmail tokenizer only when the indexed field value includes URL (Uniform Resource Locator)s and email addresses. For fields that don't include URL (Uniform Resource Locator)s or email addresses, use the [standard](https://www.mongodb.com/docs/search/indexes/analyzers/tokenizers.md#std-label-standard-tokenizer-ref tokenizer to create tokens based on word break rules.

Attributes

The uaxUrlEmail tokenizer has the following attributes:

Example

Basic Example

Create a simple index using the uaxUrlEmail tokenizer.

The following index definition indexes the page_updated_by.email field in the minutes collection using a custom analyzer named basicEmailAddressAnalyzer. It uses the uaxUrlEmail tokenizer to create tokens (searchable terms) from URL (Uniform Resource Locator)s and email addresses in the page_updated_by.email field.

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type basicEmailAddressAnalyzer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select uaxUrlEmail from the dropdown.

6. Click Add to add the custom analyzer to your index.

7. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.email field.

8. Select page_updated_by.email from the Field Name dropdown and String from the Data Type dropdown.

9. In the properties section for the data type, select basicEmailAddressAnalyzer from the Index Analyzer and Search Analyzer dropdowns.

10. Click Add, then Save Changes.

db.minutes.createSearchIndex("default", {
  "mappings": {
    "fields": {
      "page_updated_by": {
        "fields": {
          "email": {
            "analyzer": "basicEmailAddressAnalyzer",
            "type": "string"
          }
        },
        "type": "document"
      }
    }
  },
  "analyzers": [
    {
      "name": "basicEmailAddressAnalyzer",
      "tokenizer": {
        "type": "uaxUrlEmail"
      }
    }
  ]
})

The following query searches the page_updated_by.email field in the minutes collection for the email lewinsky@example.com.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "text": { "query": "lewinsky@example.com", "path": "page_updated_by.email" } } }

Output:

js SCORE: 0.5472603440284729 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "<body>We'll head out to the conference room by noon.</body>"

db.minutes.aggregate([
  {
    "$search": {
      "autocomplete": {
        "query": "lewinsky@example.com",
        "path": "page_updated_by.email"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "page_updated_by.email": 1
    }
  }
])

Output:

{ _id: 3, page_updated_by: { email: 'lewinsky@example.com' } }

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value lewinsky@example.com using the uaxUrlEmail tokenizer for the documents, which matches the search term. If you index the page_updated_by.email field using the standard tokenizer, MongoDB Search returns all the documents for the search term lewinsky@example.com.

The following table shows the tokens that the uaxUrlEmail tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Advanced Example

Create an index using the uaxUrlEmail tokenizer with autocomplete tokenization strategy.

The following index definition indexes the page_updated_by.email field in the minutes collection using a custom analyzer named emailAddressAnalyzer. It uses the following:

• The [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref type with an edgeGram [tokenization strategy](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete

• The uaxUrlEmail tokenizer to create tokens (searchable terms) from URL (Uniform Resource Locator)s and email addresses

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type emailAddressAnalyzer in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select uaxUrlEmail from the dropdown.

6. Click Add to add the custom analyzer to your index.

7. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the page_updated_by.email field.

8. Select page_updated_by.email from the Field Name dropdown and Autocomplete from the Data Type dropdown.

9. In the properties section for the data type, select emailAddressAnalyzer from the Index Analyzer and Search Analyzer dropdowns.

10. Click Add, then Save Changes.

db.minutes.createSearchIndex("default", {
  "mappings": {
    "fields": {
      "page_updated_by": {
        "fields": {
          "email": {
            "analyzer": "emailAddressAnalyzer",
            "tokenization": "edgeGram",
            "type": "autocomplete"
          }
        },
        "type": "document"
      }
    }
  },
  "analyzers": [
    {
      "name": "emailAddressAnalyzer",
      "tokenizer": {
        "type": "uaxUrlEmail"
      }
    }
  ]
})

The following query searches the page_updated_by.email field in the minutes collection for the term exam.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "autocomplete": { "query": "lewinsky@example.com", "path": "page_updated_by.email" } } }

Output:

js SCORE: 1.0203158855438232 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "<body>We'll head out to the conference room by noon.</body>"

db.minutes.aggregate([
  {
    "$search": {
      "autocomplete": {
        "query": "lewinsky@example.com",
        "path": "page_updated_by.email"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "page_updated_by.email": 1
    }
  }
])

Output:

{ _id: 3, page_updated_by: { email: 'lewinsky@example.com' } }

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value lewinsky@example.com using the uaxUrlEmail tokenizer for the documents, which matches the search term. If you index the page_updated_by.email field using the standard tokenizer, MongoDB Search returns all the documents for the search term lewinsky@example.com.

The following table shows the tokens that the uaxUrlEmail tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

whitespace

The whitespace tokenizer tokenizes based on occurrences of whitespace between words.

Attributes

The whitespace tokenizer has the following attributes:

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named whitespaceExample. It uses the whitespace tokenizer to create tokens (searchable terms) from any whitespaces in the message field.

Visual Editor

1. In the Custom Analyzers section, click Add Custom Analyzer.

2. Select the Create Your Own radio button and click Next.

3. Type whitespaceExample in the Analyzer Name field.

4. Expand Tokenizer if it's collapsed.

5. Select whitespace from the dropdown.

6. Click Add to add the custom analyzer to your index.

7. In the Field Mappings section, click Add Field Mapping to apply the custom analyzer on the message field.

8. Select message from the Field Name dropdown and String from the Data Type dropdown.

9. In the properties section for the data type, select whitespaceExample from the Index Analyzer and Search Analyzer dropdowns.

10. Click Add, then Save Changes.

db.minutes.createSearchIndex("default", {
  "mappings": {
    "dynamic": true,
    "fields": {
      "message": {
        "analyzer": "whitespaceExample",
        "type": "string"
      }
    }
  },
  "analyzers": [
    {
      "charFilters": [],
      "name": "whitespaceExample",
      "tokenFilters": [],
      "tokenizer": {
        "type": "whitespace"
      }
    }
  ]
})

The following query searches the message field in the minutes collection for the term SIGN-IN.

1. Click the Query button for your index.

2. Click Edit Query to edit the query.

3. Click on the query bar and select the database and collection.

4. Replace the default query with the following and click Find:

json { "$search": { "text": { "query": "sign-in", "path": "message" } } }

Output:

js SCORE: 0.6722691059112549 _id: "3" message: "try to sign-in" page_updated_by: Object last_name: "LEWINSKY" first_name: "Brièle" email: "lewinsky@example.com" phone: "(123).456.9870" text: Object en_US: "<body>We'll head out to the conference room by noon.</body>"

db.minutes.aggregate([
  {
    "$search": {
      "text": {
        "query": "sign-in",
        "path": "message"
      }
    }
  },
  {
    "$project": {
      "_id": 1,
      "message": 1
    }
  }
])

Output:

[ { _id: 3, message: 'try to sign-in' } ]

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value sign-in using the whitespace tokenizer for the documents, which matches the search term. If you index the message field using the standard tokenizer, MongoDB Search returns documents with _id: 1, _id: 2 and _id: 3 for the search term sign-in.

The following table shows the tokens that the whitespace tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3: