Semantic Chunking Processor

[urvishp80]

You can set SEMANTIC_CHUNKING_PARAMS env variable with a path to a json file with some params.
Example params (default) (semantic_chunker_params.json):

{
            # Semantic chunking params (for SemanticChunker, from documentation)
            "breakpoint_threshold_type": "standard_deviation",
            "buffer_size": 4,
            "breakpoint_threshold_amount": 3,
            "sentence_split_regex": r'(?<=[\.\!\?\n])(\n?)\s+',
            # Openai models
            "oai_embedding_model": "text-embedding-3-small",  # Model for embedding
            "oai_gen_model": "gpt-4o-mini",  # Model used for generating titles
            # Other params
            "chunk_threshold": 6000,  # Chunk only if number of tokens larger than this
            "use_markdown_headers": True,  # Use markdown headers for splitting
            "max_chunk_size": 6000,  # Max number of tokens in a chunk
            "generate_titles": False,  # Use LLM to generate titles for a chunk
}