Skip to content
Nappai Documentation

Natural Language Text Splitter

This component in Nappai helps you break down large amounts of text into smaller pieces. This is useful when you need to process text in stages or when working with systems that have limits on the amount of text they can handle at once. It intelligently splits the text based on natural language patterns, so the chunks make sense.

Relationship with NLTK

This component uses the NLTK (Natural Language Toolkit) library, a powerful tool for working with human language. NLTK helps the component understand sentence structure and paragraph breaks, resulting in more meaningful text chunks.

Inputs

  • Chunk Size: The maximum number of characters allowed in each chunk. Think of this as the length of each of your “shorter ropes”. The default is 1000 characters.
  • Chunk Overlap: The number of characters that overlap between consecutive chunks. This helps ensure a smooth transition between chunks and prevents information loss at the chunk boundaries. The default is 200 characters.
  • Input: The text you want to split. This can be text from a document or other data sources within Nappai.
  • Separator: Characters used to separate chunks. If left blank, the component will use a double line break (“\n\n”). You can use this to specify a different separator if needed.
  • Language: The language of the text. This helps the component understand the text’s structure better and split it more accurately. The default is “English”.

Outputs

The component produces a list of text chunks. These chunks can then be fed into other components in your Nappai workflow for further processing, such as analysis, summarization, or translation.

Usage Example

Imagine you have a long article about cats. You want to summarize it using Nappai’s summarizer. The summarizer might only accept text up to a certain length. The Natural Language Text Splitter can break the article into smaller chunks, each of which can then be summarized individually by the summarizer. Finally, you can combine the summaries to get a summary of the whole article.

Templates

[List of templates where the component can be seen and its configuration - This section needs to be populated with actual template information]

  • Summarizer: Use this component to summarize the text chunks created by the Natural Language Text Splitter.
  • Categorizer: Categorize the content of the text chunks.
  • Entities extraction: Extract key information (entities) from the text chunks.
  • Semantic Text Splitter: An alternative text splitter that uses semantic meaning to divide the text. This might be better for some applications.
  • CharacterTextSplitter: A simpler text splitter that divides text based solely on character count.

Tips and Best Practices

  • Start with the default values for Chunk Size and Chunk Overlap. Adjust them only if necessary.
  • Experiment with different separators to optimize the splitting for your specific text.
  • Choose the correct language to ensure accurate splitting.
  • For very large texts, consider using the component in conjunction with other components to manage the workflow efficiently.

Security Considerations

No specific security considerations apply to this component. However, ensure that the input text does not contain sensitive information that should not be processed by the system.