Build a RAG Chatbot in C# with Semantic Kernel (2026)

Learn how to build a RAG chatbot in C# using Semantic Kernel. Step-by-step tutorial with embeddings, vector search & code. Start building today!

Building a RAG chatbot in C# is one of the most in-demand AI skills for .NET developers in 2026 — and for good reason. Retrieval-Augmented Generation (RAG) lets you build an AI chatbot that answers questions using your own documents — internal wikis, PDFs, product manuals, support tickets — instead of relying on whatever the model memorized during training. In this tutorial, you'll build a complete RAG chatbot using C# and Microsoft's Semantic Kernel, the official .NET SDK for orchestrating large language models. We'll cover embeddings, vector search, chunking, prompt grounding, and the pitfalls that trip up most first-time builders.

By the end, you'll have a working console chatbot that ingests documents, retrieves the most relevant passages for each question, and generates grounded, citation-friendly answers — all in idiomatic, modern C#.

What Is Retrieval-Augmented Generation (RAG)?

Large language models have two fundamental limitations: their knowledge is frozen at training time, and they know nothing about your private data. RAG solves both problems with a simple but powerful pattern:

• Ingest: Split your documents into chunks and convert each chunk into an embedding — a numeric vector that captures its meaning.
• Retrieve: When a user asks a question, embed the question the same way and find the chunks whose vectors are most similar.
• Augment: Inject those chunks into the prompt as context.
• Generate: Ask the LLM to answer using only that context.

Why does this matter? Because the alternative — fine-tuning a model on your data — is expensive, slow to update, and prone to hallucination. RAG is cheaper, updates instantly (just re-index the document), and lets you trace every answer back to a source. That auditability is why RAG dominates enterprise AI chatbot architecture today.

Why Use Semantic Kernel for a RAG Chatbot in C#?

Semantic Kernel (SK) is Microsoft's open-source AI orchestration SDK for .NET. Compared to hand-rolling HTTP calls, it gives you:

• Provider abstraction: Swap between OpenAI, Azure OpenAI, Anthropic, or local models (via Ollama) without rewriting your app.
• First-class vector store connectors: In-memory, Qdrant, Azure AI Search, Redis, Postgres/pgvector, and more — behind one consistent API.
• Dependency-injection friendly design: SK plugs straight into IServiceCollection, so it feels like any other .NET library, not a bolted-on framework.
• Chat history and prompt management: Built-in types for multi-turn conversations, which every real chatbot needs.

If you're a .NET developer, Semantic Kernel is the shortest path from "I have documents" to "I have a working RAG chatbot."

Project Setup

Create a new console app and install the packages:

dotnet new console -n RagChatbot
cd RagChatbot
dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Connectors.InMemory --prerelease
dotnet add package Microsoft.Extensions.VectorData.Abstractions --prerelease

We'll use the in-memory vector store for this tutorial so you can run everything locally with zero infrastructure. In production you'd swap in Qdrant, Azure AI Search, or pgvector — the code barely changes, which is exactly the point of SK's abstractions.

Set your API key as an environment variable rather than hardcoding it (a common and dangerous mistake):

// PowerShell:  $env:OPENAI_API_KEY = "sk-..."
// bash:        export OPENAI_API_KEY="sk-..."

Step 1: Define the Document Chunk Model

Every chunk we store needs the text itself, its embedding vector, and metadata so we can cite sources later. Semantic Kernel's vector data attributes describe how each property maps into the vector store:

using Microsoft.Extensions.VectorData;

public sealed class DocChunk
{
    [VectorStoreKey]
    public string Id { get; set; } = Guid.NewGuid().ToString();

    [VectorStoreData]
    public string SourceFile { get; set; } = string.Empty;

    [VectorStoreData]
    public string Text { get; set; } = string.Empty;

    // 1536 dimensions matches OpenAI's text-embedding-3-small model
    [VectorStoreVector(Dimensions: 1536)]
    public ReadOnlyMemory<float> Embedding { get; set; }
}

Why the metadata matters: a chatbot that answers "according to onboarding-guide.md..." is dramatically more trustworthy than one that answers from nowhere. Users can verify; you can debug. Never store bare text without provenance.

Step 2: Chunk Your Documents Properly

Chunking is where most RAG chatbots quietly fail. Chunks that are too large dilute relevance and waste context tokens; chunks that are too small lose the surrounding meaning ("it increased by 40%" — what did?). A good starting point is 200–500 tokens with a 10–15% overlap so sentences straddling a boundary aren't orphaned:

public static class Chunker
{
    public static IEnumerable<string> Chunk(string text, int maxChars = 1500, int overlap = 200)
    {
        // Split on paragraphs first — semantic boundaries beat arbitrary cuts
        var paragraphs = text.Split("\n\n", StringSplitOptions.RemoveEmptyEntries);
        var current = new System.Text.StringBuilder();

        foreach (var para in paragraphs)
        {
            if (current.Length + para.Length > maxChars && current.Length > 0)
            {
                var chunk = current.ToString();
                yield return chunk;
                // Carry the tail of the previous chunk forward as overlap
                current.Clear();
                current.Append(chunk[Math.Max(0, chunk.Length - overlap)..]);
            }
            current.AppendLine(para).AppendLine();
        }

        if (current.Length > 0)
            yield return current.ToString();
    }
}

Why paragraph-first splitting? Embeddings capture meaning per chunk. A chunk that cuts a sentence in half embeds a corrupted meaning, which produces corrupted retrieval. Respecting natural document structure (paragraphs, headings, sections) is the cheapest retrieval-quality win available.

Step 3: Ingest Documents into the Vector Store

Now we embed each chunk and upsert it. Note how the embedding generator and the vector store are separate concerns — you can swap either independently:

using Microsoft.Extensions.AI;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.InMemory;

public sealed class Ingestor
{
    private readonly IEmbeddingGenerator<string, Embedding<float>> _embedder;
    private readonly InMemoryCollection<string, DocChunk> _collection;

    public Ingestor(
        IEmbeddingGenerator<string, Embedding<float>> embedder,
        InMemoryCollection<string, DocChunk> collection)
    {
        _embedder = embedder;
        _collection = collection;
    }

    public async Task IngestFileAsync(string path)
    {
        var text = await File.ReadAllTextAsync(path);
        var chunks = Chunker.Chunk(text).ToList();

        // Batch the embedding call — one API round-trip, not one per chunk
        var embeddings = await _embedder.GenerateAsync(chunks);

        var records = chunks.Zip(embeddings, (chunk, emb) => new DocChunk
        {
            SourceFile = Path.GetFileName(path),
            Text = chunk,
            Embedding = emb.Vector
        });

        await _collection.UpsertAsync(records);
        Console.WriteLine($"Ingested {chunks.Count} chunks from {path}");
    }
}

Why batch embeddings? Embedding APIs accept arrays. Ingesting a 300-chunk document with one request instead of 300 is roughly 100× faster and far less likely to hit rate limits. This single change is the difference between a 2-second ingest and a 3-minute one.

Step 4: Retrieve and Generate — the Heart of the RAG Chatbot

Here's where retrieval-augmented generation actually happens. We embed the user's question, run a vector similarity search, and build a grounded prompt:

using Microsoft.SemanticKernel.ChatCompletion;

public sealed class RagChat
{
    private readonly IChatCompletionService _chat;
    private readonly IEmbeddingGenerator<string, Embedding<float>> _embedder;
    private readonly InMemoryCollection<string, DocChunk> _collection;
    private readonly ChatHistory _history;

    public RagChat(
        IChatCompletionService chat,
        IEmbeddingGenerator<string, Embedding<float>> embedder,
        InMemoryCollection<string, DocChunk> collection)
    {
        _chat = chat;
        _embedder = embedder;
        _collection = collection;
        _history = new ChatHistory(
            """
            You are a helpful assistant that answers questions using ONLY the
            provided context. If the context does not contain the answer, say
            "I don't have that information in the documents." Always mention
            which source file your answer came from.
            """);
    }

    public async Task<string> AskAsync(string question)
    {
        // 1. Embed the question
        var queryEmbedding = (await _embedder.GenerateAsync(question)).Vector;

        // 2. Vector search: top 3 most similar chunks
        var results = _collection.SearchAsync(queryEmbedding, top: 3);

        var context = new System.Text.StringBuilder();
        await foreach (var result in results)
        {
            context.AppendLine($"[Source: {result.Record.SourceFile}]");
            context.AppendLine(result.Record.Text);
            context.AppendLine("---");
        }

        // 3. Ground the question in retrieved context
        _history.AddUserMessage(
            $"""
            Context:
            {context}

            Question: {question}
            """);

        // 4. Generate the answer
        var response = await _chat.GetChatMessageContentAsync(_history);
        _history.AddAssistantMessage(response.Content ?? string.Empty);
        return response.Content ?? string.Empty;
    }
}

Why the strict system prompt? Without the instruction to answer only from context, the model happily blends retrieved facts with its training data — and you can no longer tell which parts of an answer are grounded. The explicit "I don't have that information" escape hatch is equally important: it converts hallucinations into honest refusals.

Step 5: Wire It All Together

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.InMemory;

var apiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY")
    ?? throw new InvalidOperationException("Set OPENAI_API_KEY first.");

var builder = Kernel.CreateBuilder();
builder.AddOpenAIChatCompletion("gpt-4o-mini", apiKey);
builder.AddOpenAIEmbeddingGenerator("text-embedding-3-small", apiKey);
var kernel = builder.Build();

var embedder = kernel.GetRequiredService<IEmbeddingGenerator<string, Embedding<float>>>();
var chat = kernel.GetRequiredService<IChatCompletionService>();

var store = new InMemoryVectorStore();
var collection = store.GetCollection<string, DocChunk>("docs");
await collection.EnsureCollectionExistsAsync();

// Ingest every markdown file in ./docs
var ingestor = new Ingestor(embedder, collection);
foreach (var file in Directory.GetFiles("docs", "*.md"))
    await ingestor.IngestFileAsync(file);

// Chat loop
var rag = new RagChat(chat, embedder, collection);
Console.WriteLine("RAG chatbot ready. Ask a question (or 'exit'):\n");

while (true)
{
    Console.Write("You: ");
    var question = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(question) || question == "exit") break;

    var answer = await rag.AskAsync(question);
    Console.WriteLine($"\nBot: {answer}\n");
}

Drop a few markdown files into a docs folder, run dotnet run, and you have a working chatbot over your own documents. Anthropic's Claude models work equally well here via SK's connector packages — the RAG pattern is entirely provider-agnostic.

Best Practices for Production RAG Chatbots

• Use a real vector database. The in-memory store vanishes on restart and doesn't scale. Qdrant (self-hosted, free) or Azure AI Search (managed) are the most popular choices for .NET teams — and with SK's abstractions, migrating is mostly a one-line change.
• Cache embeddings by content hash. Re-embedding unchanged documents on every deploy burns money. Hash each chunk; only embed chunks whose hash changed.
• Tune top-k empirically. Three chunks is a starting point, not gospel. Too few and the answer lacks context; too many and you drown the model in noise. Build a small test set of question–answer pairs and measure.
• Add a relevance threshold. Vector search always returns something, even for off-topic questions. Discard results below a similarity score cutoff so "what's the weather?" doesn't retrieve your HR policy.
• Consider hybrid search. Pure vector search struggles with exact identifiers (error codes, part numbers, acronyms). Combining keyword (BM25) and vector search — which Azure AI Search and Qdrant both support — noticeably improves retrieval quality.
• Log the retrieved chunks with every answer. When a user reports a wrong answer, the first question is always "what did retrieval return?" If you didn't log it, you can't debug it.

Common Pitfalls to Avoid

• Mismatched embedding models. The model that embeds your query must be the same one that embedded your documents. Mixing them produces vectors in incompatible spaces — retrieval silently returns garbage. This is the #1 "my RAG doesn't work" bug.
• Wrong vector dimensions. The Dimensions value in your record model must match your embedding model (1536 for text-embedding-3-small, 3072 for text-embedding-3-large). A mismatch throws at ingest time — the lucky failure mode — or truncates silently in some stores.
• Unbounded chat history. Every turn appends the full retrieved context to history. After twenty questions, you're sending enormous prompts and paying for it. Trim old context messages, or store only the question and answer after each turn.
• Ingesting PDFs as raw text. PDF extraction mangles tables, headers, and multi-column layouts. Use a proper extraction library (PdfPig works well in .NET) and inspect the output before trusting your index.
• Skipping evaluation. A RAG chatbot that "seems fine" in a five-minute demo can be wrong 30% of the time. Build a test set early — even 20 known question–answer pairs catches regressions when you change chunk sizes or models.

Conclusion: Your RAG Chatbot in C# Is Just the Beginning

You've now built a complete RAG chatbot in C# with Semantic Kernel: chunking documents intelligently, batch-generating embeddings, running vector similarity search, and grounding LLM answers in retrieved context with source citations. Key takeaways:

• RAG beats fine-tuning for private-document Q&A: it's cheaper, updates instantly, and every answer is traceable to a source.
• Retrieval quality is everything. Chunking strategy, embedding-model consistency, and top-k tuning matter more than which LLM you pick.
• Semantic Kernel's abstractions pay off — swapping vector stores or model providers is a configuration change, not a rewrite.
• Ground strictly and refuse honestly. A chatbot that says "I don't know" builds more trust than one that guesses confidently.

From here, natural next steps are swapping in Qdrant for persistence, adding hybrid search, exposing the chatbot through an ASP.NET Core minimal API, and building an evaluation harness. The pattern you've learned today is the same one powering enterprise document assistants everywhere — you now have it running in pure C#.

Tags: #RAG chatbot C# #Semantic Kernel tutorial #C# AI chatbot #retrieval augmented generation #vector database C# #chat with your documents #Semantic Kernel RAG #C# embeddings

About csharp-coder.com
Your go-to resource for C#, .NET, and modern software development. Follow along for daily tutorials, tips, and real-world examples.