Why my chunking still failed despite tokenization

Contents

• The error persisted despite chunking
• Why 512 tokens are not the whole story
• What I take from this
• All articles in the series

The error persisted despite chunking

I had already changed the ingestion worker to split long texts into multiple token chunks. The log showed something that initially looked promising:

// Old worker log showing tight chunk sizes with no safety margin
[IngestWorker] captureId abc | Original tokens: 828 | Chunks: 2 | ChunkSize: 512 | Overlap: 64

Yet the embedding provider kept returning the dreaded context overflow error. That was where I had to stop trusting the rough chunk count in the code. A theoretical number in a log is not proof that the vector database provider or local embedding model will actually accept the final input without errors.

Why 512 tokens are not the whole story

My first serious mistake with mxbai-embed-large was simple: I used the hard model limit of 512 tokens directly as the maximum chunk size in the code. That looks logical at first, but in practice leaves far too little room. During processing, a model often adds special tokens internally, such as <s> and </s> at the beginning and end of the text. A simple external token counter also often counts words differently from the embedding model’s internal tokenization pipeline.

If you cut text at exactly 512 tokens, those differences and additional special tokens inevitably push the actual API request beyond the context window’s hard limit. The result: the worker formally segments the input correctly, but leaves no practical safety margin.

That was the decisive diagnosis. Chunking itself was not the problem. The naive assumption that the maximum model limit was also a safe, usable working size for ingestion was.

What I take from this

We must never target the maximum model limit. We need to treat it as an absolute upper bound. Stable embeddings require a deliberate buffer. Only then does the worker remain stable when the model’s tokenizer adds special tokens or minor differences arise during re-encoding.

This diagnosis prompted two fundamental architectural decisions in our Memory Engine monorepo:

1. Upgrade the model to bge-m3: A more capable multilingual model with a much larger context window of 8192 tokens.
2. Use a conservative token budget: Set a controlled chunk size of 1500 tokens, well below the maximum, to maximize semantic focus and eliminate any risk of overflow.

All articles in the series

1. Vision and system overview: Chrome Extension, RAG architecture and project background: Read article
2. Building a RAG system: Qdrant, embeddings and cosine similarity in TypeScript: Read article
3. AI provider abstraction: Ollama vs. OpenAI, interface design and no vendor lock-in: Read article
4. Chrome Extension MV3: Three isolated runtime contexts, message passing and Strategy Pattern: Read article
5. Docker Compose strategy: The override pattern, from local development to Azure: Read article
6. Ollama locally vs. Docker: The decision and its consequences: Read article
7. Ollama auto-pull entrypoint: Automated model setup at container startup: Read article
8. tsconfig and Vite: Node16 vs. bundler, and why Vite has its own rules: Read article
9. Load complete Instagram captions with MutationObserver: Read article
10. Chrome Extension foundation with health indicator and retry queue: Read article
11. Phase 2 features: Shadow DOM overlay, Tailwind v4 and duplicate detection: Read article
12. Race condition in platform detection: How a UI event breaks Instagram detection: Read article
13. Extracting PostId in two Instagram layouts: querySelector vs. ancestor traversal: Read article
14. Capture a complete Instagram carousel with MutationObserver: Lazy loading, observer-before-click and timeout fallback: Read article
15. Notes and tags when saving screenshots: Read article
16. Instagram keyboard shortcuts block Chrome Extension input: Read article
17. Lowercase normalization and duplicate detection in tag input: Read article
18. Zitadel Login V2 in Docker Compose with three hidden errors: Read article
19. PKCE OAuth in a Chrome MV3 extension: Read article
20. React frontend with react-oidc-context and Zitadel: Read article
21. Vite build-time environment variables in Docker: Read article
22. Event-driven ingestion with BullMQ and Redis: Read article
23. MinIO instead of Azurite: S3-compatible object storage locally and on Hetzner: Read article
24. access_token, id_token and the Userinfo endpoint: Which belongs where: Read article
25. Qdrant multi-tenancy with one collection per user: Read article
26. When backend and frontend know different types: Read article
27. Removing Zitadel bootstrap: Host header bug and manual setup: Read article
28. Backend code review with six problems fixed before launch: Read article
29. Traefik instead of NGINX: Reverse proxy for a growing Docker Compose stack: Read article
30. Two-layer rate limiting: Traefik and express-rate-limit with Redis: Read article
31. Implementing GDPR Article 17 correctly: Promise.allSettled and export batching: Read article
32. Embedding model lock-in: Why mxbai-embed-large is a permanent production decision: Read article
33. Docker volumes in production: Named volumes, bind mounts and the Hetzner volume approach: Read article
34. Two vulnerabilities before launch: Redis without authentication and an exposed Qdrant admin port: Read article
35. Traefik as the single entry point in the Docker Compose stack: Read article
36. Connecting Zitadel behind Traefik with issuer, JWKS and Login V2: Read article
37. Fixing the frontend when the nginx healthcheck fails on localhost: Read article
38. Observability for my Docker Compose stack with Bull Board and Dozzle: Read article
39. Open the Qdrant dashboard securely with local Traefik and an SSH tunnel: Read article
40. Diagnosis of chunking failures despite tokenization: (this article)
41. Why I set the chunk size to 1500 tokens: Read article
42. Splitting the embedding workflow into clear steps: Read article
43. Bringing chunking, storage and search back together: Read article
44. Capture Intake as a separate module, less orchestration in the Express endpoint: Read article

Working on a similar RAG system and want to evaluate the same structure for your project? Let us assess it together.