Elasticsearch：当混合搜索真正发挥作用时

作者：来自 Elastic Gustavo Llermaly

展示混合搜索何时优于单独的词汇或语义搜索。

在本文中，我们将通过示例探讨混合搜索，并展示它与单独使用词汇或语义搜索技术相比的真正优势。

什么是混合搜索？

混合搜索是一种结合了不同搜索方法（如传统词汇匹配和语义搜索）的技术。

当用户知道确切的单词时，词汇搜索非常有用。这种方法将找到相关文档，并使用 TF-IDF 以合理的方式对其进行排序，这意味着：你搜索的术语在数据集中越常见，它对分数的贡献就越小；在某个文档中越常见，它对分数的贡献就越大。

更多有关 TF-IDF 的阅读，请参阅文章 “Elasticsearch：分布式计分”。

但是，如果查询中的单词不在文档中，该怎么办？有时用户不是在寻找具体的东西，而是在寻找一个概念。他们可能不是在寻找特定的餐厅，而是在寻找 “与家人一起吃饭的好地方”。对于这种查询，语义搜索很有用，因为它会考虑搜索查询的上下文并带来类似的文档。与以前的方法相比，你可以期望获得更多的相关文档，但作为回报，这种方法在精度方面存在困难，尤其是数字方面。

混合搜索将术语匹配的精确度与语义搜索的上下文感知匹配相结合，为我们提供了两全其美的优势。

你可以在这篇文章中深入了解混合搜索，并在此文章中了解有关词汇和语义搜索差异的更多信息。

让我们使用房地产单位创建一个示例。

查询将是：quiet place in Pinewood with 2 rooms，其中 “quiet place” 是查询的语义部分，而 “Pinewood with 2 rooms” 是查询的文本或词汇部分。

配置 ELSER

我们将使用 ELSER 作为我们的模型提供者。

首先创建推理端点：

PUT _inference/sparse_embedding/my-elser-model 
{
  "service": "elser", 
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1
  }
}

如果这是你第一次使用 ELSER，你可能会在后台加载模型时遇到 502 Bad Gateway 错误。你可以在 Kibana 中的 Machine Learning > Trained Models 中检查模型的状态。部署后，你可以继续下一步。

配置索引

对于索引，我们将使用文本字段，并使用 semantic_text 作为语义字段。我们将复制描述，因为我们想将它们用于 match 和 semantic 查询。

PUT properties-hybrid
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "english"
      },
      "description": {
        "type": "text",
        "analyzer": "english", 
        "copy_to": "semantic_field"
      },
      "neighborhood": {
        "type": "keyword"
      },
      "bedrooms": {
        "type": "integer"
      },
      "bathrooms": {
        "type": "integer"
      },
      "semantic_field": {
        "type": "semantic_text",
        "inference_id": "my-elser-model"
      }
    }
  }
}

索引数据


POST _bulk
{ "index" : { "_index" : "properties-hybrid" , "_id": "1"} }
{ "title": "2 Bed 2 Bath in Sunnydale", "description": "Spacious apartment with modern amenities, perfect for urban dwellers.", "neighborhood": "Sunnydale", "bedrooms": 2, "bathrooms": 2 }
{ "index" : { "_index" : "properties-hybrid", "_id": "2" } }
{ "title": "1 Bed 1 Bath in Sunnydale", "description": "Compact apartment with easy access to downtown, ideal for singles or couples.", "neighborhood": "Sunnydale", "bedrooms": 1, "bathrooms": 1 }
{ "index" : { "_index" : "properties-hybrid", "_id": "3" } }
{ "title": "2 Bed 2 Bath in Pinewood", "description": "New apartment with modern bedrooms, located in a restaurant and bar area. Suitable for active people who enjoy nightlife.", "neighborhood": "Pinewood", "bedrooms": 2, "bathrooms": 2 }
{ "index" : { "_index" : "properties-hybrid", "_id": "4" } }
{ "title": "3 Bed 2 Bath in Pinewood", "description": "Secluded and private family unit with a practical layout with three total rooms. Near schools and shops. Perfect for raising kids.", "neighborhood": "Pinewood", "bedrooms": 3, "bathrooms": 2 }
{ "index" : { "_index" : "properties-hybrid", "_id": "5" } }
{ "title": "2 Bed 2 Bath in Pinewood", "description": "Retired apartment in a serene neighborhood, perfect for those seeking a retreat. This well-maintained residence offers two bedrooms with abundant natural light and silence.", "neighborhood": "Pinewood", "bedrooms": 2, "bathrooms": 2 }
{ "index" : { "_index" : "properties-hybrid", "_id": "6" } }
{ "title": "1 Bed 1 Bath in Pinewood", "description": "Apartment with a scenic view, ideal for those seeking an energetic environment.", "neighborhood": "Pinewood", "bedrooms": 1, "bathrooms": 1 }
{ "index" : { "_index" : "properties-hybrid", "_id": "7" } }
{ "title": "2 Bed 2 Bath in Maplewood", "description": "Nice apartment with a large balcony, offering a relaxed and comfortable living experience.", "neighborhood": "Maplewood", "bedrooms": 2, "bathrooms": 2 }
{ "index" : { "_index" : "properties-hybrid", "_id": "8" } }
{ "title": "1 Bed 1 Bath in Maplewood", "description": "Charming apartment with modern interiors, situated in a peaceful neighborhood.", "neighborhood": "Maplewood", "bedrooms": 1, "bathrooms": 1 }

查询数据

让我们从经典的匹配查询开始，它将根据标题和描述的内容进行搜索：

GET properties-hybrid/_search
{
  "query": {
    "multi_match": {
      "query": "quiet home 2 bedroom in Pinewood",
      "fields": ["title", "description"]
    }
  }
}

这是第一个结果：

{
    "description": "New apartment with modern bedrooms, located in a restaurant and bar area. Suitable for active people who enjoy nightlife.",
    "title": "2 Bed 2 Bath in Pinewood"
}

还不错。它成功地吸引了附近的 Pinewood 和 2 间卧室的需求，然而，这根本不是一个 quiet place。

现在，一个纯粹的语义查询：

GET properties-hybrid/_search
{
  "query": {
    "semantic": {
      "field": "semantic_field",
      "query": "quiet home in Pinewood with 2 rooms"
    }
  }
}

这是第一个结果：

{
    "description": "Secluded and private family unit with a practical layout with three total rooms. Near schools and shops. Perfect for raising kids.",
    "title": "3 Bed 2 Bath in Pinewood"
}

现在，搜索结果考虑了 quiet home 部分，将其与 “secluded and private” 等内容联系起来，但这个是 3 间卧室，我们正在寻找 2 间卧室。

现在让我们运行混合搜索。我们将使用 RRF（Reciprocal rank fusion - 倒数排名融合）来实现此目的，并结合前两个查询。RRF 算法将为我们混合两个查询的分数。

GET properties-hybrid/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "semantic": {
                "field": "semantic_field",
                "query": "quiet home 2 bedroom in Pinewood"
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "quiet home 2 bedroom in Pinewood",
                "fields": ["title", "description"]
              }
            }
          }
        }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  }
}

这是第一个结果：

{
    "description": ""Retired apartment in a serene neighborhood, perfect for those seeking a retreat. This well-maintained residence offers two bedrooms with abundant natural light and silence.",
    "title": "2 Bed 2 Bath in Pinewood"
}

现在的结果考虑既是一个安静的地方，而且有 2 间卧室。

评估结果

对于评估，我们将使用排名评估 API（Ranking Evaluation API ），它允许我们自动执行运行查询的过程，然后检查相关结果的位置。你可以选择不同的评估指标。在这个例子中，我将选择平均倒数排名 MRR (Mean reciprocal ranking)，它考虑了结果位置，并随着位置降低 1/位置# 而降低分数。

对于这个场景，我们将针对初始问题测试我们的 3 个查询（multi_match、semantic、hybrid）：

quiet home 2 bedroom in Pinewood

预计以下公寓将排在第一位，因为它满足所有标准。

Retired apartment in a serene neighborhood, perfect for those seeking a retreat. This well-maintained residence offers two bedrooms with abundant natural light and silence."

我们可以根据需要配置任意数量的查询，并在评级中添加我们期望排在第一位的文档的 ID：

GET /properties-hybrid/_rank_eval
{
  "requests": [
    {
      "id": "hybrid",
      "request": {
        "retriever": {
          "rrf": {
            "retrievers": [
              {
                "standard": {
                  "query": {
                    "semantic": {
                      "field": "semantic_field",
                      "query": "quiet home 2 bedroom in Pinewood"
                    }
                  }
                }
              },
              {
                "standard": {
                  "query": {
                    "multi_match": {
                      "query": "quiet home 2 bedroom in Pinewood",
                      "fields": [
                        "title",
                        "description"
                      ]
                    }
                  }
                }
              }
            ],
            "rank_window_size": 50,
            "rank_constant": 20
          }
        }
      },
      "ratings": [
        {
          "_index": "properties-hybrid",
          "_id": "5",
          "rating": 1
        }
      ]
    },
    {
      "id": "lexical",
      "request": {
        "query": {
          "multi_match": {
            "query": "quiet home 2 bedroom in Pinewood",
            "fields": [
              "title",
              "description"
            ]
          }
        }
      },
      "ratings": [
        {
          "_index": "properties-hybrid",
          "_id": "5",
          "rating": 1
        }
      ]
    },
    {
      "id": "semantic",
      "request": {
        "query": {
          "semantic": {
            "field": "semantic_field",
            "query": "quiet place in Pinewood with 2 rooms"
          }
        }
      },
      "ratings": [
        {
          "_index": "properties-hybrid",
          "_id": "5",
          "rating": 1
        }
      ]
    }
  ],
  "metric": {
    "mean_reciprocal_rank": {
      "k": 20,
      "relevant_rating_threshold": 1
    }
  }
}

从图中可以看出，该查询在混合搜索（第一位）中获得了 1 分，在其他搜索中获得了 0.5 分，这意味着在第二位返回了预期结果。

结论

全文搜索技术（查找术语并按术语频率对结果进行排序）和语义搜索（按语义接近度进行搜索）在不同情况下非常有用。一方面，当用户明确他们想要搜索的内容时，文本搜索会大放异彩，例如提供文章的确切 SKU 或技术手册中的单词。另一方面，当用户正在寻找文档中未明确定义的概念或想法时，语义搜索很有用。将这两种方法与混合搜索相结合，可以为你提供全文搜索功能以及添加语义相关文档，这在需要关键字匹配和上下文理解的特定场景中非常有用。这种双重方法提高了搜索准确性和相关性，使其成为复杂查询和多样化内容类型的理想选择。

想要获得 Elastic 认证？了解下一次 Elasticsearch 工程师培训何时开始！

Elasticsearch 包含新功能，可帮助你为你的用例构建最佳搜索解决方案。深入了解我们的示例笔记本以了解更多信息，开始免费云试用，或立即在你的本地机器上试用 Elastic。

原文：When hybrid search truly shines - Elasticsearch Labs