CharacterTextSplitter
CharacterTextSplitter cuts texts at the separator you choose, for example paragraph breaks, and groups the parts into pieces of a maximum size.
Long documents are cut into pieces before being stored in a vector database: that way each search returns only the piece that answers, not the whole document.
When to use it
- Your texts have a clear separator, such as a blank line between records.
- You want to control exactly where the cuts happen.
Configuration
| Field | What it does |
|---|---|
| Chunk Size | Maximum size of each piece, in characters. Default: 1000. |
| Chunk Overlap | How many characters two consecutive pieces share, so an idea is not cut in half. Default: 200. |
| Input | The texts to cut, usually the output of a component that loads files. |
| Separator | The characters to cut on. Empty, paragraph breaks. |
Output
The Data output holds the pieces, each with the data of the document it comes from. Connect it to Ingest Data of a vector database, such as Chroma DB.
Example
A shop exports its catalogue with a blank line between products.
- Connect the catalogue to Input.
- Leave Separator empty to cut at the blank lines.
- Connect Data to a vector database.
Good practices
- Start with the defaults and adjust Chunk Size if search results fall short or bring too much text.
Related components
- Recursive Character Text Splitter: the recommended splitter for ordinary texts.
- Chroma DB: where to store the pieces.