​
Vector Indexing with Vedaya

This guide explains how to use Vedaya’s Vector Indexing API to manage document embeddings, retrieve statistics, and perform vector operations.

​
Overview

Vedaya’s Vector Indexing system allows you to:

  • Generate and store vector embeddings for your documents
  • Retrieve statistics about indexed data
  • Get embeddings for specific files
  • Generate vectors locally
  • Download original documents

​
Getting Statistics

To get statistics about your indexed data:

import requests

url = "https://vedaya-backend.fly.dev/api/vector-indexing/stats"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers)
print(response.json())

The response includes information such as:

{
  "total_documents": 42,
  "total_chunks": 1250,
  "embedding_dimensions": 768,
  "storage_usage_mb": 128.5,
  "indexed_document_types": {
    "pdf": 28,
    "docx": 10,
    "txt": 4
  }
}

​
Listing Indexed Documents

To retrieve a list of all your indexed documents:

import requests

url = "https://vedaya-backend.fly.dev/api/vector-indexing/documents"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers)
print(response.json())

The response includes metadata about each document:

{
  "documents": [
    {
      "document_id": "d12345678",
      "filename": "research-paper.pdf",
      "file_size_kb": 1024,
      "chunk_count": 30,
      "creation_date": "2023-06-15T10:30:45Z",
      "status": "indexed"
    },
    {
      "document_id": "d87654321",
      "filename": "product-spec.docx",
      "file_size_kb": 512,
      "chunk_count": 15,
      "creation_date": "2023-06-16T14:22:10Z",
      "status": "indexed"
    }
  ]
}

​
Getting Sample Chunks

To retrieve sample chunks with their vector embeddings:

import requests

url = "https://vedaya-backend.fly.dev/api/vector-indexing/chunks"

params = {
  "limit": 5  # Number of chunks to retrieve
}

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers, params=params)
print(response.json())

​
Getting File Embeddings

To retrieve embeddings for a specific file:

import requests

file_id = "f12345678"  # ID of the file
url = f"https://vedaya-backend.fly.dev/api/vector-indexing/embeddings/{file_id}"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers)
print(response.json())

The response includes chunked text and corresponding vector embeddings:

{
  "file_id": "f12345678",
  "filename": "research-paper.pdf",
  "chunks": [
    {
      "chunk_id": "c12345-1",
      "text": "This research focuses on the application of...",
      "embedding": [0.1, 0.2, 0.3, ...],
      "position": 1
    },
    {
      "chunk_id": "c12345-2",
      "text": "The results demonstrate a significant improvement...",
      "embedding": [0.2, 0.3, 0.4, ...],
      "position": 2
    }
  ]
}

​
Generating Local Vectors

For simple use cases or testing, you can generate vectors locally:

import requests

url = "https://vedaya-backend.fly.dev/api/vector-indexing/generate-local-vectors"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.post(url, headers=headers)
print(response.json())

This starts an asynchronous process to generate vectors locally. You can check the status of this process:

import requests

url = "https://vedaya-backend.fly.dev/api/vector-indexing/local-indexing-status"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers)
print(response.json())

​
Downloading Documents

To download the original document:

import requests

document_id = "d12345678"  # ID of the document
url = f"https://vedaya-backend.fly.dev/api/vector-indexing/documents/{document_id}/download"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers)

# Save the file
with open("downloaded_document.pdf", "wb") as file:
    file.write(response.content)

​
Understanding Embeddings

Vector embeddings are numerical representations of text that capture semantic meaning. In Vedaya:

  • Each document is split into chunks
  • Each chunk is converted to a vector using an embedding model
  • The default embedding dimension is typically 768 or 1536, depending on the model
  • Similar text chunks have similar vector representations
  • These vectors enable semantic search, clustering, and other NLP operations

​
Working with Vector Similarity

When you query the knowledge base, Vedaya uses vector similarity to find relevant chunks:

  • The query text is converted to a vector using the same embedding model
  • Vector similarity (usually cosine similarity) is computed between query and chunks
  • Chunks with the highest similarity scores are returned
  • This enables semantic search rather than just keyword matching

​
Best Practices

  1. Embedding Models: Use consistent embedding models across your workflow
  2. Chunk Size: Optimal chunk size depends on your use case; experiment to find the best setting
  3. Monitoring: Regularly check the vector indexing stats to understand your data growth
  4. Local vs. Cloud: Use local vector generation for testing, but prefer cloud-based embeddings for production
  5. Performance: For large document collections, consider batch processing to optimize performance

For more details on available endpoints and parameters, see the Vector Indexing API Reference.