Chunking, Qdrant and retrieval without duplicate RAG search results

Contents

• What I want to see during verification
• The retrieval challenge
• The deduplication strategy in the query path
• What the results show
• GDPR Article 17 compliance at the press of a button
• All articles in the series

What I want to see during verification

To examine a split workflow thoroughly, we need to keep three clearly separated layers in view:

1. The worker must reliably split a long source text into overlapping token chunks.
2. Qdrant must be able to store multiple points per capture record.
3. The query path must still return at most one highly relevant result per capture.

Only when these three layers work together seamlessly do we have a robust RAG system. We had already diagnosed the problem and chosen 1500 tokens. Only then did the ingestion process have enough stability to validate the interaction between storage and search systematically.

The retrieval challenge

Once a capture is divided into chunks, Qdrant contains multiple vectors referring to the same physical capture through captureId. A similarity search is likely to return several chunks from the same post.

Without countermeasures, the search results would be flooded with duplicates or fragments of one post. This drastically reduces result diversity and fills the downstream LLM’s valuable context window with redundant text.

The deduplication strategy in the query path

We solve this directly in the backend retrieval logic, retrieval.ts:

// Query deduplication logic based on captureId
const deduplicatedResults = results.reduce<StoredQueryResult[]>((acc, current) => {
  const exists = acc.some((item) => item.payload.captureId === current.payload.captureId);
  if (!exists) {
    acc.push(current);
  }
  return acc;
}, []);

First, we run the Qdrant similarity search, isolated to the requesting user’s userId. Then we deduplicate the returned matches programmatically by captureId. Only the chunk with the highest cosine similarity score for the user’s query remains for each capture.

What the results show

The end-to-end validation confirms success:

• In the worker log, we see the exact input token count and resulting chunks, for example tokens=2145 chunks=2.
• In Qdrant, points are stored cleanly under random RFC 4122 UUIDs with all necessary minimal metadata.
• In the search API response, POST /query returns a highly precise, deduplicated result array. Each match represents a unique capture, enriched with the semantically strongest text excerpt for the LLM.

GDPR Article 17 compliance at the press of a button

Choosing a single shared collection with a keyword index on userId also greatly benefits the right-to-erasure process. We no longer need to laboriously delete per-user collections to remove all of a user’s vector data without residue. A simple Qdrant filter-delete request immediately deletes all associated points:

// Instant tenant erasure from shared collection
await client.delete(COLLECTION_NAME, {
  filter: {
    must: [{ key: 'userId', match: { value: userId } }],
  },
});

This delivers GDPR compliance at high speed without operational overhead.

All articles in the series

1. Vision and system overview: Chrome Extension, RAG architecture and project background: Read article
2. Building a RAG system: Qdrant, embeddings and cosine similarity in TypeScript: Read article
3. AI provider abstraction: Ollama vs. OpenAI, interface design and no vendor lock-in: Read article
4. Chrome Extension MV3: Three isolated runtime contexts, message passing and Strategy Pattern: Read article
5. Docker Compose strategy: The override pattern, from local development to Azure: Read article
6. Ollama locally vs. Docker: The decision and its consequences: Read article
7. Ollama auto-pull entrypoint: Automated model setup at container startup: Read article
8. tsconfig and Vite: Node16 vs. bundler, and why Vite has its own rules: Read article
9. Load complete Instagram captions with MutationObserver: Read article
10. Chrome Extension foundation with health indicator and retry queue: Read article
11. Phase 2 features: Shadow DOM overlay, Tailwind v4 and duplicate detection: Read article
12. Race condition in platform detection: How a UI event breaks Instagram detection: Read article
13. Extracting PostId in two Instagram layouts: querySelector vs. ancestor traversal: Read article
14. Capture a complete Instagram carousel with MutationObserver: Lazy loading, observer-before-click and timeout fallback: Read article
15. Notes and tags when saving screenshots: Read article
16. Instagram keyboard shortcuts block Chrome Extension input: Read article
17. Lowercase normalization and duplicate detection in tag input: Read article
18. Zitadel Login V2 in Docker Compose with three hidden errors: Read article
19. PKCE OAuth in a Chrome MV3 extension: Read article
20. React frontend with react-oidc-context and Zitadel: Read article
21. Vite build-time environment variables in Docker: Read article
22. Event-driven ingestion with BullMQ and Redis: Read article
23. MinIO instead of Azurite: S3-compatible object storage locally and on Hetzner: Read article
24. access_token, id_token and the Userinfo endpoint: Which belongs where: Read article
25. Qdrant multi-tenancy with one collection per user: Read article
26. When backend and frontend know different types: Read article
27. Removing Zitadel bootstrap: Host header bug and manual setup: Read article
28. Backend code review with six problems fixed before launch: Read article
29. Traefik instead of NGINX: Reverse proxy for a growing Docker Compose stack: Read article
30. Two-layer rate limiting: Traefik and express-rate-limit with Redis: Read article
31. Implementing GDPR Article 17 correctly: Promise.allSettled and export batching: Read article
32. Embedding model lock-in: Why mxbai-embed-large is a permanent production decision: Read article
33. Docker volumes in production: Named volumes, bind mounts and the Hetzner volume approach: Read article
34. Two vulnerabilities before launch: Redis without authentication and an exposed Qdrant admin port: Read article
35. Traefik as the single entry point in the Docker Compose stack: Read article
36. Connecting Zitadel behind Traefik with issuer, JWKS and Login V2: Read article
37. Fixing the frontend when the nginx healthcheck fails on localhost: Read article
38. Observability for my Docker Compose stack with Bull Board and Dozzle: Read article
39. Open the Qdrant dashboard securely with local Traefik and an SSH tunnel: Read article
40. Diagnosis of chunking failures despite tokenization: Read article
41. Why I set the chunk size to 1500 tokens: Read article
42. Splitting the embedding workflow into clear steps: Read article
43. Bringing chunking, storage and search back together: (this article)
44. Capture Intake as a separate module, less orchestration in the Express endpoint: Read article

Are you working on a similar RAG system and want to assess the same structure for your project? Let us assess it together.