• Home
  • Blog
  • AI
  • Build a Private RAG Assistant with Ollama and Qdrant
Build a Private RAG Assistant with Ollama and Qdrant

Build a Private RAG Assistant with Ollama and Qdrant

Many teams want an AI assistant that answers questions about their own documents, but cannot send those documents to a third-party API. Retrieval-augmented generation (RAG) with a local model solves this. I built one for a multi-tenant SaaS platform using Ollama and the Qdrant vector database, and this post explains the architecture and the pieces that matter.

Key takeaways

  • RAG means: find the most relevant text first, then ask the model to answer using only that text.
  • Ollama runs open models locally, and Qdrant stores and searches the embeddings.
  • Chunking quality and tenant isolation matter more than the model you pick.
  • Index on a schedule, and always return sources with the answer.

How RAG works

A plain language model only knows what it was trained on. RAG adds a retrieval step:

  1. Index. Split your documents into chunks, turn each chunk into an embedding (a list of numbers that captures meaning) and store it in a vector database.
  2. Retrieve. When a user asks a question, embed the question and find the closest chunks.
  3. Generate. Send the question and those chunks to the model with an instruction to answer from them.

The stack

  • Ollama serves models over a local HTTP API. I used Qwen 2.5 for answers and an embedding model such as nomic-embed-text for vectors.
  • Qdrant stores vectors and supports filtering by metadata, which is how you keep tenants apart.
  • A Node.js worker extracts text from files, chunks it and writes to Qdrant.

Step 1: create a collection

The vector size must match your embedding model. For nomic-embed-text it is 768.

PUT http://localhost:6333/collections/knowledge
{
  "vectors": { "size": 768, "distance": "Cosine" }
}

Step 2: embed and store chunks

async function embed(text) {
  const res = await fetch("http://localhost:11434/api/embed", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model: "nomic-embed-text", input: text }),
  });
  const data = await res.json();
  return data.embeddings[0];
}

async function indexChunk({ id, text, tenantId, source }) {
  await fetch("http://localhost:6333/collections/knowledge/points", {
    method: "PUT",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      points: [{ id, vector: await embed(text), payload: { text, tenantId, source } }],
    }),
  });
}

Step 3: retrieve and answer

async function ask(question, tenantId) {
  const vector = await embed(question);
  const search = await fetch("http://localhost:6333/collections/knowledge/points/search", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      vector,
      limit: 5,
      with_payload: true,
      filter: { must: [{ key: "tenantId", match: { value: tenantId } }] },
    }),
  });
  const { result } = await search.json();
  const context = result.map((hit) => hit.payload.text).join("\n---\n");

  const chat = await fetch("http://localhost:11434/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "qwen2.5",
      stream: false,
      messages: [
        {
          role: "system",
          content:
            "Answer only from the provided context. If it is not there, say you do not know.",
        },
        { role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
      ],
    }),
  });
  const data = await chat.json();
  return { answer: data.message.content, sources: result.map((hit) => hit.payload.source) };
}

The filter on tenantId is the important line. It guarantees one customer's question can never retrieve another customer's documents.

What makes it work in production

  • Chunk by meaning, not by length. Split on headings and paragraphs, keep chunks to a few hundred words and overlap them slightly so answers are not cut in half.
  • Extract text properly. PDF and DOCX files need real text extraction before chunking. Re-extract when a file changes.
  • Index on a schedule. A nightly job over files, knowledge-base articles and announcements keeps the assistant current without slowing users down.
  • Always return sources. Showing which documents an answer came from builds trust and makes mistakes easy to spot.
  • Tell the model to admit gaps. An instruction to answer only from context reduces made-up answers.

When to use a hosted model instead

Local models trade some answer quality and speed for privacy and cost control. If your data is not sensitive, a hosted model may be simpler. If it is, the same pipeline works with either, because only the chat call changes.

I use this pattern in Dashify, and I offer AI and LLM integration for products that need it.

  • RAG
  • Ollama
  • Qdrant
  • LLM
  • AI assistant

Let's Work Together.

Tell me about your project. I'll reply with questions, a first approach and honest advice.