Skip to content
Nappai Documentation

Natural Language Text Splitter

Natural Language Text Splitter cuts texts respecting where sentences end in the text’s language, so no piece leaves a sentence half-finished.

Long documents are cut into pieces before being stored in a vector database: that way each search returns only the piece that answers, not the whole document.

When to use it

  • Your texts are prose, such as articles, reports or emails.
  • You want each piece to contain complete sentences.

Configuration

FieldWhat it does
Chunk SizeMaximum size of each piece, in characters. Default: 1000.
Chunk OverlapHow many characters two consecutive pieces share, so an idea is not cut in half. Default: 200.
InputThe texts to cut, usually the output of a component that loads files.
SeparatorThe characters to cut on. Empty, paragraph breaks.
LanguageThe text’s language, in English, for example spanish. Empty, English.

Output

The Data output holds the pieces, each with the data of the document it comes from. Connect it to Ingest Data of a vector database, such as Chroma DB.

Example

A newspaper prepares its Spanish archive for a search engine.

  1. Connect the articles to Input and type spanish in Language.
  2. Connect Data to a vector database.

Good practices

  • Start with the defaults and adjust Chunk Size if search results fall short or bring too much text.