Split Text
Split Text cuts long texts into chunks. You decide the maximum size of each chunk, how much text is repeated between one chunk and the next, and which character the text is cut on.
It’s a common step before storing documents in a knowledge base or sending long texts to an AI model, which works better with manageable pieces.
When to use it
- Prepare manuals, contracts or internal policies to store them in a knowledge base and query them later.
- Split the transcript of a long meeting to summarize it in parts.
- Separate a document into paragraphs to analyze each one separately.
- Break a very long text into pieces an AI model can process.
Configuration
| Field | What it does |
|---|---|
| Data Inputs | The texts you want to split. Connect it from the component that reads the document. |
| Chunk Overlap | How many characters are repeated between one chunk and the next, so context isn’t lost at the cuts. The default is 200. |
| Chunk Size | Maximum size of each chunk, in characters. The default is 1000. If you set it to 0, the text is cut at every occurrence of the separator, with no size limit, and empty chunks are dropped. |
| Separator | The character the text is cut on. The default is a line break. |
Output
The Chunks output returns one record per chunk, with the chunk’s text and the information the original record carried (for example, the file name). You can connect it to a knowledge base component, to a Loop or to any component that takes records.
Example
A company wants its internal assistant to answer questions about the employee handbook. The flow reads the handbook, passes it through Split Text with Chunk Size 1000 and Chunk Overlap 200, and stores the chunks in a knowledge base. When someone asks about holidays, the assistant finds the chunk that covers it.
Good practices
- Keep Chunk Overlap much smaller than Chunk Size.
- To split a text into exact paragraphs, set Chunk Size to 0 and use the line break as separator.