Answer up front: A RAG pipeline architecture is a set of connected services that ingest raw documents, turn them into embeddings, store them in a vector database, retrieve the most relevant chunks, and finally feed those chunks to a language model for generation. In practice you need a modular design, solid chunking, a fast vector store with hybrid search, and observability that lets you spot bottlenecks before they break your service.
Below I walk through each piece of that puzzle, share the code I run in production, and point out the trade-offs that kept me up at night.
What are the core components of a RAG pipeline?
A RAG pipeline architecture typically consists of:
- Ingestion Layer – pulls data from PDFs, web pages, DB rows, or streaming APIs.
- Chunking & Pre-processing – splits text into manageable pieces, normalizes, and optionally adds metadata.
-
Embedding Service – calls an encoder (e.g., OpenAI
text-embedding-ada-002or a local BERT) and produces dense vectors. - Vector Store – persists embeddings, supports similarity search, and often offers hybrid (vector + keyword) capabilities.
- Retrieval Engine – given a user query, fetches the top-k chunks using the vector store.
- LLM Generation – injects retrieved context into a prompt and calls the LLM.
- API & Orchestration – FastAPI endpoints, background workers, and a message bus to glue everything together.
In my last project I ran each component as a small Docker container behind a Cloud Run service. The biggest pain point was the session leak in the SQLAlchemy layer that silently ate DB connections after a few thousand requests. I fixed it by scoping sessions to the request lifecycle – see my write-up on FastAPI SQLAlchemy Session Leak Detection for the exact steps.