Engineering NexusRec AI: Multimodal Visual-Semantic Retail Discovery with PyTorch Two-Tower Retrieval, Qdrant Hybrid RRF, and Thompson Sampling Bandits

Consider an enterprise commerce or industrial MRO (Maintenance, Repair, and Operations) catalog serving 2.5 million SKUs. When a customer uploads a photo of a damaged high-pressure Viton hydraulic flange seal or searches in natural language for "a waterproof GORE-TEX alpine shell under $220 that pairs with crampon mountaineering boots", traditional recommendation systems fracture along two fault lines ^{[1] [2]}:

1. Pure Collaborative Filtering Fails on Cold-Start & Visual Compatibility: Matrix factorization and item-to-item co-visitation graphs assign zero weight to newly launched SKUs (zero historical clicks) and are completely blind to visual texture, chromaticity, and structural part geometry (OpenCV / CNN visual descriptors).
2. Bi-Encoder Vector Search Ignores Fine-Grained Margin & Cross-Feature Interactions: While Approximate Nearest Neighbor (ANN) vector search retrieves Top-100 candidates in <10ms, inner-product embeddings compress user and item representations into separate vectors—losing non-linear cross-features such as user price elasticity vs. SKU gross margin or category saturation (returning 10 nearly identical black jackets) ^{[3] [4]}.

In NexusRec AI (Live Interactive Demo), we solve both failure modes via a 4-Stage Multimodal Recommendation Funnel: uniting OpenCV + PyTorch 64-D Two-Tower Contrastive Retrieval, Qdrant Universal Hybrid Search (Dense + BM25 Reciprocal Rank Fusion with hard payload filters), Scikit-learn Gradient Boosted Cross-Encoder Reranking, Thompson Sampling Contextual Bandits, and Maximal Marginal Relevance (MMR) diversity optimization ^{[5] [6]}.

1. The Architectural Bottleneck: Single-Stage Vector Lookup vs. 4-Stage Multimodal Funnel

Relying on a single-stage vector similarity lookup or a raw LLM prompt over catalog JSON creates three production failures:

1. Inventory & Budget Constraint Leakage: Post-filtering ANN top-$K$ results after graph traversal causes "empty result collapse" when 95% of nearest neighbors are out of stock or exceed the buyer's price ceiling. Hard constraints (in_stock == True, unit_price <= max_price, domain == "industrial") must be pushed down into the Qdrant HNSW payload filter during graph traversal ^[4].
2. Cold-Start Starvation (Rich-Get-Richer Bias): Deterministic greedy ranking permanently buries new high-margin arrivals that lack click history. Without a probabilistic exploration policy (Thompson Sampling over $\text{Beta}(\alpha_i, \beta_i)$ posteriors), the recommender never gathers conversion feedback on cold-start items ^[5].
3. Category Monoculture: Ranking purely by pointwise relevance populates the Top-10 viewport with redundant variants of a single category, depressing multi-item basket attachment.

The diagram below contrasts Naive Single-Stage Vector Retrieval (Partition A) against the NexusRec 4-Stage Multimodal Funnel (Partition B):

2. PyTorch Two-Tower InfoNCE Training & Qdrant Hybrid Reciprocal Rank Fusion

In Stage 1, the Item Tower encodes each SKU's 32-D OpenCV visual signature (12-bin HSV chromaticity, 10-bin Sobel edge orientation, and 10-bin Gabor surface texture) concatenated with its 32-D semantic/commercial vector into an L2-normalized 64-D embedding $\mathbf{v}_i \in \mathbb{S}^{63}$. The User Tower encodes the user's exponentially decayed session clickstream and live query intent into $\mathbf{u}_b \in \mathbb{S}^{63}$, trained via In-Batch Negative InfoNCE (Sampled Softmax) Loss with temperature $\tau = 0.08$ ^{[1] [2]}:

$$ \mathcal{L}{\text{InfoNCE}} = -\frac{1}{B} \sum{b=1}^{B} \log \frac{\exp\left(\langle \mathbf{u}_b, \mathbf{v}b^+ \rangle / \tau\right)}{\sum{j=1}^{B} \exp\left(\langle \mathbf{u}_b, \mathbf{v}_j \rangle / \tau\right)} $$

In Stage 2, Qdrant executes Universal Hybrid Search, fusing the dense Two-Tower + visual similarity rank $r_{\text{dense}}(d)$ with the sparse BM25/SPLADE lexical rank $r_{\text{sparse}}(d)$ via Reciprocal Rank Fusion (RRF, $k=60$) ^[4]:

$$ \text{RRF}(d) = \frac{\alpha}{k + r_{\text{dense}}(d)} + \frac{1 - \alpha}{k + r_{\text{sparse}}(d)} $$

# src/core/reranker.py
from typing import Any, Dict, List
import numpy as np

def apply_mmr_diversity(
    candidates: List[Dict[str, Any]],
    item_embeddings: Dict[str, np.ndarray],
    top_k: int = 10,
    lambda_diversity: float = 0.30,
    max_per_category: int = 2,
) -> List[Dict[str, Any]]:
    """
    Stage 4: Maximal Marginal Relevance (MMR) greedy selection with hard category caps.
    Balances Cross-Encoder + Thompson Bandit relevance against intra-list cosine similarity.
    """
    selected: List[Dict[str, Any]] = []
    remaining = list(candidates)
    category_counts: Dict[str, int] = {}

    while remaining and len(selected) < top_k:
        best_idx = 0
        best_mmr_score = -1e9

        for idx, cand in enumerate(remaining):
            cat = cand["category"]
            cat_penalty = 0.35 if category_counts.get(cat, 0) >= max_per_category else 0.0
            rel_score = float(cand["rerank_score"])

            if not selected:
                max_sim = 0.0
            else:
                vec_c = item_embeddings[cand["sku_id"]]
                max_sim = max(
                    float(np.dot(vec_c, item_embeddings[s["sku_id"]]))
                    for s in selected
                )

            mmr_score = (1.0 - lambda_diversity) * rel_score - lambda_diversity * max_sim - cat_penalty
            if mmr_score > best_mmr_score:
                best_mmr_score = mmr_score
                best_idx = idx

        chosen = remaining.pop(best_idx)
        chosen["mmr_score"] = round(float(best_mmr_score), 4)
        category_counts[chosen["category"]] = category_counts.get(chosen["category"], 0) + 1
        selected.append(chosen)

    return selected

3. Thompson Sampling Contextual Bandits for Cold-Start Exploration

To prevent cold-start starvation in Stage 3, every SKU maintains a Bayesian conversion posterior $\theta_i \sim \text{Beta}(\alpha_i, \beta_i)$ updated in real time as users click (+1 α) or skip (+1 β) recommended cards ^[5]. For cold-start items, the Cross-Encoder score $s_{\text{CE}}(u, i)$ is augmented with a Thompson sample $\tilde{\theta}_i$ scaled by uncertainty bonus $\gamma$:

$$ s_{\text{final}}(u, i) = 0.78 \cdot s_{\text{CE}}(u, i) + 0.22 \cdot \tilde{\theta}_i, \quad \tilde{\theta}_i \sim \text{Beta}(\alpha_i, \beta_i) $$

4. Reproducible Evaluation: NDCG@10, Recall@10, MRR, and Intra-List Diversity

We evaluated NexusRec AI across 4 benchmark buyer cohorts (Technical Alpine Gear, Industrial Hydraulic MRO, Urban Minimalist Apparel, and Cold-Start Visitor) against Baseline Popularity and Pure Collaborative Filtering:

5. Antithesis: When NOT to Use a 4-Stage Neural Recommendation Funnel

1. Small Static Catalogs (<100 SKUs): If a SaaS pricing page or boutique store sells fewer than 100 items, maintaining a PyTorch Two-Tower retriever and Qdrant HNSW index is unnecessary overhead; a deterministic SQL query with rule-based merchandising weights will execute in <1ms with zero ML maintenance.
2. Strict Regulatory Sequential Workflows: In clinical drug dosing or aviation checklists where item ordering is governed by regulatory protocol rather than user preference, probabilistic Thompson Sampling bandits must never be used.

References & Further Reading

1. Covington, P., Adams, J., & Sargin, E. (2016). Deep Neural Networks for YouTube Recommendations. Proceedings of the 10th ACM Conference on Recommender Systems (RecSys). ↩︎
2. Yi, X., et al. (2019). Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations. Google Research, ACM RecSys. ↩︎
3. Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML. ↩︎
4. Cormack, G. V., Clarke, C. L., & Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. ACM SIGIR. ↩︎
5. Chapelle, O., & Li, L. (2011). An Empirical Evaluation of Thompson Sampling. Advances in Neural Information Processing Systems (NeurIPS). ↩︎
6. Carbonell, J., & Goldstein, J. (1998). The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. ACM SIGIR. ↩︎