Chunking that respects structure — don't shred your own documents — step 5 of 9
The recursive splitter takes a list of separators in PRIORITY order — try the most semantic one first, fall back through less semantic ones, end with a hard char cut. Fill in the list for general English text: paragraph break, then sentence end, then word, then character.
Expected output:
['\n\n', '. ', ' ', '']
⌘↵ runs the editor.read, then continue.
Chunking that respects structure — don't shred your own documents — step 5 of 9
The recursive splitter takes a list of separators in PRIORITY order — try the most semantic one first, fall back through less semantic ones, end with a hard char cut. Fill in the list for general English text: paragraph break, then sentence end, then word, then character.
Expected output:
['\n\n', '. ', ' ', '']
⌘↵ runs the editor.read, then continue.