Multimodal Search: Now Your App Can See Inside Your Products

Grab your favorite Python file & warm up your CLI, because today we’re adding Multimodal Search to self-service Beta! If you already have an account, you can get building now at app.objective.inc. If you don’t have an account yet, grab one today!

One Index that sees inside all of your datatypes.

An Objective Multimodal Index is a new index type that sees inside all of the datatypes in your Objects. There’s a ton of heavy lifting that goes on in a Multimodal Index type to make it just work. No glue code, no figuring out how to combine scores, no juggling separate datatype result sets — your Objective Multimodal Index sees inside all the datatypes in your Object and understands the human meaning, holistically.

Just like your Text Indexes and Image Indexes, Multimodal Indexes take the natural language query your user searches and turns it into meaning that it can use to find relevant results. And just like all Objective Search Indexes, as the data in your Object Store changes, your Multimodal Index stays up to date automatically, and in real-time.

Building product search that just feels magical.

Let’s say we’re building search for your online clothing store, and each Object in your Object Store has a product_photo attribute:

{
	"name": "Winter Wonderland Pea Coat",
	"color": "forest green",
	"product_photo": "https:// ... 3723598.jpg",
}

And let’s say the image at the URL referenced in product_photo looks like this:

And let’s say our user searches “forest green winter coat with big buttons”. Notice that “big buttons” is never mentioned anywhere in the text, but it’s visible in the image. And while they’re big by coat standards, but still a relatively small part of the visual real-estate of this product photo.

Your Multimodal Index pays attention to different parts of the query, and different parts of the Object, to understand the meaning & relevancy of each of the Objects in your Object Store. It knows to look out for big buttons, and it knows to look for darker green hues.

Real Multimodal Search packed into a few lines of Python.

One of the best parts of your Multimodal Index is how much power you get from a few lines of code. Once you’ve loaded some Objects into an Object Store, it’s as simple as creating a Multimodal Index connected to that Object Store, and you’re minutes away from wicked-powerful search.

from objective import Client

# Initialize superpowers
client = Client(api_key="YOUR_API_KEY")

# Now let's build those Objects with images for your Object Store!
objects = [
	{
  	"id": "1",
  	"object": {
			"name": "Winter Wonderland Pea Coat",
			"color": "forest green",
			"product_photo": "https:// ... 3723598.jpg",
		}
  },
	...
]

client.object_store.upsert_objects(objects)

# Create a Multimodal Index

index = client.indexes.create_index(
	index_type='multimodal',
	fields={
		"searchable": ["name", "color"],
		"crawlable": ["product_photo"]
	}
)

# Search your index!

index.search(query="forest green winter coat with big buttons")

Notice the fields properties — you decide which properties your Index should be searchable as text. And we make crawling your images as easy as searching text. Just add that property to crawlable and we’ll handle all the hard vision and intent understanding work for you.

We’ll handle the AI heavy lifting. You go invent the future.

We think the possibilities really are endless when every developer has access to real, powerful multimodal search in just a few lines of code.

Ecommerce search experiences get a whole lot more intuitive when the burden isn’t put on the user to creatively combine keywords and instead they can just say what they mean. And then add semantic context to narrow their results.

Customer support instantly gets more meaningful when your support team has search that sees inside all of the datatypes in a ticket. Letting your support team have an even bigger impact on the business.

And searching long-form content with mixed datatypes becomes possible, powerful, and really simple.

We really can’t wait to see what you build with Multimodal Search!

‍