Objective Search Evaluator: An Open-Source Tool for Search Quality Assessment

Last month, we launched Anton. Anton is an API designed and optimized to judge the quality of search results. Search evaluations are crucial for the success of search systems, as they help search teams assess and improve the quality of results.

Today, we are open sourcing a Python library that we are actively developing to help Anton users hit the ground running with their own search evaluations.

The Objective Search Evaluator, is designed to be extended to any search engine. It currently supports running evaluations on Objective indexes and OpenSearch indexes. Adding support for additional search engines is supported through an easy to use API (see example “scrapers”).

Evaluation results are loaded to Pandas DataFrames and the library includes several built-in reports (with more to come!).

The repository includes 2 quickstarts:

1. Running an evaluation on an Objective search index and viewing results.
2. Running evaluations on an Objective search index and an OpenSearch index at the same time and comparing results.

Example usage (from the Quickstart):

# Set up your scraper
scraper = ObjectiveScraper(
    ObjectiveScrapeParams(
        limit=10,
        scrape_id="my-search-engine",
        api_key="your_objective_api_key",
        index_id="your_index_id",
        object_fields="*"
    )
)

# Create an evaluator
evaluator = ObjectiveEvaluator(
    scrapers=[scraper],
    api_key="your_anton_api_key",
    work_dir="work/"
)

# Run the evaluation
queries = ["red dress", "graphic t-shirt", "jeans for men"]
evaluator.run(queries)

# Generate a summary
evaluator.summary()

Key Features

• Evaluate search quality using Anton’s AI-powered judgments
• Compare results across multiple search engines simultaneously
• Extend functionality with custom scrapers for any search system
• Generate data visualizations and reports for analysis

The Objective Search Evaluator is available on GitHub at https://github.com/objective-inc/search-evaluator. We welcome contributions and feedback from the community.