Indexing

Indexing is the process of taking data from the Object Store and making it available for search.

Features

1. Automatic updates - updates to objects in the object store are automatically reflected in indexes
2. Near real time - updates are reflected in the index in seconds
3. Embedding generation - embeddings, and other data representations, are automatically computed based on your data
4. Fully managed - we handle keeping all the data sync'd and up to date in the indexes, so you never have to worry

How indexing works

Indexing comprises of several steps:

1. Monitoring the Object Store for object updates
2. Processing the objects. This includes crawling images, chunking, etc..
3. Building embeddings for search. These can be dense or sparse embeddings
4. Storing the results in the index, making them available for search

FAQs

If I update an object that already exists, is there any time where the object isn't available for search?

Answer: No. When you update an existing object, the existing object is still searchable while the update is being processed. Once the update completes the new data will be reflected in search results.

Limitations

1. Limit of 971KiB per Object (995000 bytes)
2. Limit on the number of images indexed per Object. See Pricing