Building "Chat with your Data" Apps using Embeddings, ChatGPT, and Cosmos DB for Mongo DB vCore
Key takeaways
- To answer over your own documents you use Retrieval-Augmented Generation (RAG), not fine-tuning or stuffing everything into the prompt.
- The pipeline: turn document text into OpenAI embeddings, store them in a vector database, run a vector similarity search on the user’s query, then pass the top matches plus the question to the chat model.
- A model’s token limit is the core reason for the embeddings-plus-vector-DB approach; embeddings capture semantic meaning so related content clusters together.
- Document chunking is a real design tradeoff: chunks too large hit token limits, too small lose context; the demo slices a PDF into records with an overlap option.
- The approach generalises to other vector databases (Milvus, Pinecone, Chroma) by adapting the insert-embedding and similarity-search functions, and secrets are best kept in a vault rather than in code.
Retrieval-augmented generation (RAG) combines a language model with selected information from an external knowledge base. This updated tutorial demonstrates chunking, embeddings, retrieval, grounded answering, and visualization with a deterministic local fixture. Optional OpenAI and Cosmos DB adapters remain disabled by default.
Retrieve relevant context
The offline example uses HashingVectorizer and cosine similarity. It provides a transparent stand-in for hosted embeddings and lets us test retrieval before connecting cloud services.
doc_vectors = vectorizer.transform([item["text"] for item in documents]).toarray()
query_vector = vectorizer.transform([question]).toarray()
scores = cosine_similarity(query_vector, doc_vectors)[0]Only the top chunks are supplied to the answer step. The deterministic answer cites the fixture document ID so grounding can be inspected.
Optional live adapters
Live generation uses OpenAI().responses.create(...). The Cosmos DB for MongoDB vCore adapter reads COSMOS_MONGODB_CONNECTION_STRING from the environment and verifies the connection with ping; no secrets are stored in the notebook.
Production RAG also requires authorization filters, robust chunking, hybrid retrieval or reranking, prompt-injection defenses, citation validation, evaluation datasets, and telemetry. Retrieval quality and answer quality must be measured separately.
Frequently asked questions
- Why can’t you just paste your whole knowledge base into the prompt?
- Models have a token limit, so large documents will not fit in a single prompt. Instead you convert your data into embeddings, store them in a vector database, and retrieve only the most relevant slices to augment each query — Retrieval-Augmented Generation.
- What is Retrieval-Augmented Generation (RAG)?
- RAG combines a language model with external knowledge. Both your source data and the user’s query are converted into embeddings; the query is matched against the knowledge base to find similar content, and that content is added as context to the prompt, producing more informed, domain-specific answers.
- Do you need to fine-tune the model for this?
- No. The model answers purely from the context supplied at query time. Fine-tuning is a separate technique; this "chat with your data" pattern relies on retrieval, not on changing the model’s weights.
- Can you use a vector database other than Cosmos DB?
- Yes. The same approach works with alternatives such as Milvus, Pinecone, and Chroma — you adapt the insert-embedding and similarity-search functions to each database’s API.
3 Commentsarchived from the original site