​

Data Ingestion with Vedaya

This guide walks you through the process of uploading and processing documents using Vedaya’s Data Ingestion API.

​

Overview

Vedaya’s Data Ingestion system allows you to:

• Upload files directly from your device
• Connect to cloud storage providers (Google Drive, Dropbox, OneDrive, etc.)
• Ingest content from URLs
• Configure processing parameters like chunking size and embedding models

​

Upload Files

​

Direct Upload

The simplest way to get started is by uploading files directly:

import requests

url = "https://vedaya-backend.fly.dev/api/data-ingestion/upload-files"

# Prepare the files and parameters
files = [
    ('files', ('document.pdf', open('document.pdf', 'rb'), 'application/pdf'))
]
data = {
    'chunk_size': 500,
    'overlap_pct': 10,
    'embedding_model': 'default-embedding',
    'pdf_extractor': 'pypdf2'
}

# Send the request with your API key
headers = {
    'Authorization': 'Bearer YOUR_API_KEY'
}
response = requests.post(url, headers=headers, files=files, data=data)
print(response.json())

This will return a response with the status of your upload and processing:

{
  "status": "success",
  "message": "Files uploaded successfully",
  "file_ids": ["f12345678"],
  "processing_status": "queued"
}

​

Cloud Integration

Connect to cloud storage providers to ingest files:

import requests
import json

url = "https://vedaya-backend.fly.dev/api/data-ingestion/connect-cloud"

payload = json.dumps({
  "provider": "gdrive",
  "access_token": "YOUR_ACCESS_TOKEN",
  "refresh_token": "YOUR_REFRESH_TOKEN",
  "folder_path": "My Documents/Data",
  "chunk_size": 500,
  "overlap_pct": 10
})

headers = {
  'Authorization': 'Bearer YOUR_API_KEY',
  'Content-Type': 'application/json'
}

response = requests.post(url, headers=headers, data=payload)
print(response.json())

For OAuth-based providers, we provide endpoints to handle the authentication flow. See the Cloud Integration API documentation for details.

​

Ingesting from URLs

To fetch and process content from a URL:

import requests
import json

url = "https://vedaya-backend.fly.dev/api/data-ingestion/ingest-url"

payload = json.dumps({
  "url": "https://example.com/document.pdf",
  "filename": "example-document",
  "chunk_size": 500,
  "overlap_pct": 10
})

headers = {
  'Authorization': 'Bearer YOUR_API_KEY',
  'Content-Type': 'application/json'
}

response = requests.post(url, headers=headers, data=payload)
print(response.json())

​

Monitoring Ingestion Status

After uploading files, you can check their processing status:

import requests

file_id = "f12345678"  # The ID returned from upload
url = f"https://vedaya-backend.fly.dev/api/data-ingestion/status/{file_id}"

headers = {
  'Authorization': 'Bearer YOUR_API_KEY'
}

response = requests.get(url, headers=headers)
print(response.json())

​

Customizing Ingestion Parameters

​

Chunking Strategy

Chunking controls how documents are split into smaller pieces for processing and retrieval:

• chunk_size: Number of characters in each chunk (default: 500)
• overlap_pct: Percentage of overlap between chunks (default: 10%)

Larger chunks contain more context but can reduce precision during retrieval. Smaller chunks improve precision but may lose context.

​

PDF Extraction Options

Vedaya supports multiple PDF extraction methods:

• pypdf2: Fast and lightweight (default)
• pdfplumber: Better for complex layouts
• pymupdf: High performance with advanced features
• pymupdf4llm: Optimized for language model processing

Choose based on your document complexity and extraction needs.

​

Best Practices

1. File Size: Keep individual files under 20MB for optimal processing
2. File Types: Supported formats include PDF, DOCX, TXT, CSV, PPTX, and HTML
3. Batch Processing: For large collections, use multiple requests in parallel
4. Error Handling: Monitor the status endpoint to catch and retry failed ingestions
5. Embedding Models: Start with the default model and experiment with others as needed

For more details on available endpoints and parameters, see the Data Ingestion API Reference.