Like this shoe, but for running

One of the test queries in AI Commerce Agent pairs a photo of a black Lee Cooper formal shoe with “find something like this but better for running.” A visual match is only part of the request. The result also needs to fit the intended use.

Why a formal shoe ranks first

The first result in the debug log is product 2823, the exact shoe in the photo, with an image score of 1. Second place goes to another formal shoe. The image search has found a close match, but the request for running shoes hasn’t affected the ranking yet.

Keep each ranking stage

I combine the text, image, and keyword results, then apply rules based on what the user is asking for. Multimodal queries can go through another round of model reranking. Here’s what ranks first at each stage:

1. Initial retrieval: Lee Cooper black formal shoe, product 2823.
2. Rule-based reranking: Nike Air Max +2009 sports shoe, product 22693.
3. Final result after model reranking: Nike Lunarfly+2 Shield sports shoe, product 15735.

The top result is now a sports shoe. That’s progress, though it doesn’t prove the shoe is right for a particular runner. The categories and some attributes come from a prepared catalog; the actual product details still matter. The debug log keeps the candidates, scores, match reasons, and each ranking so I can trace what changed.

One query still misses

The report covers 18 hand-written test queries. Strict Hit@3 is 0.944: 17 of them return at least one labeled relevant product in the top three. The one miss is a sandals query: it gets the subcategory right but misses the target product IDs.

That’s why I track exact-product hits separately from category matches. This small test set helps catch regressions; it doesn’t measure how well the system would do with real users.

Sources: the query’s full debug record, the 18-query evaluation report.