If you've spent any time building AI-powered search or Q&A systems, you've hit the same wall I did: your documents are too long for embedding models, and naively splitting them by character count produces garbage results. The retrieval step falls apart because the chunks don't respect sentence boundaries, paragraph structure, or document hierarchy. The fix isn't complicated, but it needs to be done right. That's why I built the RAG Content Chunker.
This post covers what RAG actually is, why chunking matters so much, and how my actor on Apify handles it.
What is RAG?
RAG stands for Retrieval-Augmented Generation. It's a pattern where you give a large language model access to external knowledge by retrieving relevant documents at query time, rather than relying solely on what the model memorized during training.
The typical pipeline looks like this:
Crawl/Collect content
-> Chunk it into smaller pieces
-> Embed each chunk (turn it into a vector)
-> Store vectors in a database (Pinecone, Qdrant, Weaviate, etc.)
-> At query time, search for relevant chunks
-> Feed those chunks to the LLM as context
Every step matters, but chunking is the one that silently ruins everything if done poorly. If your chunks are too large, the embedding model can't capture the meaning well. Too small, and you lose context. Split in the wrong place, and you cut a sentence in half -- making retrieval unreliable and the LLM's answer incoherent.
Chunking is the foundation. Get it wrong, and no amount of prompt engineering or model upgrades will save you.
What the RAG Content Chunker Does
The RAG Content Chunker is an Apify Actor that takes raw text or Markdown and splits it into token-counted, metadata-rich chunks that are ready to be embedded and stored in a vector database. No external API calls, no API keys required -- it's pure local computation.
The stack: Python 3.11, tiktoken (cl100k_base encoding), Apify Actor framework.
It supports two input modes:
- Direct text -- paste in your content and get chunks back immediately
- Dataset chaining -- point it at the output of any Apify crawler (like Website Content Crawler) and it processes every item automatically
The second mode is where it gets powerful. You can crawl an entire documentation site with an existing Apify actor, then chain this chunker directly onto that output. No glue code, no intermediate scripts.
Three Chunking Strategies
Not all content is the same, so the actor offers three strategies:
Recursive (default)
The general-purpose option. It tries to split on paragraph boundaries first. If a paragraph is still too large, it falls back to sentence boundaries. If a single sentence exceeds the token limit, it splits on word boundaries. As a last resort, it does a hard cut by token count.
This cascading approach means the chunker always prefers the most natural break point available.
Markdown
For structured documents -- documentation pages, READMEs, crawled web content with headers. It splits on Markdown headers (h1 through h6), preserving the document hierarchy. Each chunk knows which section heading it belongs to, which is valuable metadata for retrieval.
If a single section is still too large, it falls back to recursive splitting within that section.
Sentence
Best for conversational content, Q&A pairs, or prose. It splits strictly on sentence boundaries, keeping each sentence intact. Good for content where paragraph structure isn't meaningful but sentence integrity is.
Here's what the strategy selection looks like in practice:
| Strategy | Best For | How It Splits |
|---|---|---|
recursive |
General text, mixed content | Paragraphs, then sentences, then words |
markdown |
Documentation, web pages | Markdown headers, preserves section structure |
sentence |
Conversational content, Q&A | Sentence boundaries only |
Token-Aware Splitting
This is the part that matters most. The chunker doesn't split by character count -- it splits by token count using tiktoken's cl100k_base encoding. This is the same tokenizer used by OpenAI's text-embedding-3-small, text-embedding-3-large, and GPT-4.
That means when you set chunk_size: 512, you're getting chunks that are genuinely 512 tokens or fewer as measured by the embedding model you're likely using. No guessing, no "roughly 4 characters per token" approximations.
import tiktoken
_ENCODING = tiktoken.get_encoding("cl100k_base")
def _count_tokens(text: str) -> int:
return len(_ENCODING.encode(text, disallowed_special=()))
You can configure chunk size anywhere from 64 to 8,192 tokens, and overlap from 0 to 2,048 tokens. Overlap is useful for preserving context across chunk boundaries -- the trailing sentences of one chunk appear at the start of the next.
Deterministic Chunk IDs
Every chunk gets a deterministic ID based on a SHA-256 hash of the source URL, chunk index, and the first 200 characters of the text:
def _chunk_id(text: str, index: int, source_url: str = "") -> str:
payload = f"{source_url}|{index}|{text[:200]}"
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]
Why does this matter? Because if you re-run the chunker on the same content, you get the same IDs. This enables incremental updates in your vector database -- you can detect which chunks changed and only re-embed those, instead of wiping and rebuilding the entire index.
What the Output Looks Like
Each chunk is pushed as a separate dataset item, making the output directly consumable by downstream actors or any tool that reads Apify datasets:
{
"chunk_id": "a1b2c3d4e5f67890",
"chunk_index": 0,
"text": "This is the actual chunk content...",
"token_count": 127,
"source_url": "https://example.com/page",
"page_title": "Example Page",
"section_heading": "Introduction"
}
A summary item is appended at the end with totals and billing:
{
"_summary": true,
"total_chunks": 42,
"total_tokens": 18340,
"strategy": "recursive",
"chunk_size": 512,
"chunk_overlap": 64,
"processing_time": 0.342,
"billing": {
"total_chunks": 42,
"amount": 0.021,
"rate_per_chunk": 0.0005
}
}
Pricing
The actor uses Apify's Pay-Per-Event model at $0.0005 per chunk -- that's $0.50 per 1,000 chunks.
To put that in perspective:
| Content Size | Approx. Chunks | Cost |
|---|---|---|
| Single blog post | 10-20 | $0.005-$0.01 |
| 10-page website | 50-100 | $0.025-$0.05 |
| 100-page docs site | 500-1,000 | $0.25-$0.50 |
The billing logic is transparent and auditable -- it's a single multiplication in the source code, nothing hidden.
Security and Validation
Even though this actor doesn't involve an LLM (so prompt injection isn't a concern), I built in a proper validation layer:
- Size limits -- 500K characters max for text input, 10K max dataset items
- Sanitization -- strips null bytes and control characters while preserving newlines and tabs for Markdown structure
- Injection prevention -- dataset IDs and field names are validated against strict regex patterns to prevent path traversal or injection
- No secrets -- the actor requires no API keys or credentials to run
- No network calls -- all processing is local computation on Apify's infrastructure
Where to Find It
The actor is live and ready to use:
- Apify Store: https://apify.com/labrat011/rag-content-chunker
- Source Code: https://github.com/labrat-0/rag-content-chunker
What I Learned
- Token-aware splitting is non-negotiable for RAG. Character-based splitting is a trap that produces inconsistent chunk sizes and breaks mid-word.
- Deterministic IDs are a small detail that saves enormous headaches when you're doing incremental updates against a vector database.
- Dataset chaining on Apify is a clean pattern. Building actors that consume the output of other actors means you can assemble entire pipelines without writing integration code.
- Input validation matters even when there's no LLM involved. Any public-facing actor should assume adversarial input and handle it gracefully.
What's Next
This is my second skill on the Apify Store (the first being the Multilingual Translation Agent). I'm continuing to build focused, composable tools that do one thing well and make themselves useful in larger pipelines.
If you're building a RAG pipeline and need a chunking step that just works, give it a try.
Research Resources
- RAG Content Chunker - Apify Store - Live actor listing
- RAG Content Chunker - GitHub - Source code and documentation
- tiktoken - OpenAI's fast BPE tokenizer
- Apify Platform - Cloud platform for web scraping and automation actors
- Retrieval-Augmented Generation (RAG) - Background on the RAG pattern