Elasticsearch：使用 ELSER v2 文本扩展进行语义搜索

Elastic 提供了一个强大的 ELSER 供我们进行语义搜索。ELSER 是一种稀疏向量的搜索方法。我们无需对它做任何的微调及训练。它是一种 out-of-domain 的模型。目前它仅对英文进行支持。希望将来它能对其它的语言支持的更好。更多关于 ELSER 的知识，请参阅文章 “Elasticsearch：使用 ELSER 释放语义搜索的力量：Elastic Learned Sparse EncoderR”。在本文中，我们将使用第二版的 ELSER 来进行语义搜索。我将使用 Jupyter notebook 演示如何使用 ELSER 模型 .elser_model_2 模型，该模型提供了更高的检索精度。

如果你已使用 ELSER 模型 .elser_model_1 设置索引，并且想要升级到 ELSER v2 模型 - .elser_model_2，请按照文章升级索引以使用 elser 模型的说明进行操作来进行升级。

安装

如果你还没有安装好自己的 Elasticsearch 及 Kibana，请参考文章：

安装 Elasticsearch 及 Kibana

如果你还没有安装好自己的 Elasticsearch 及 Kibana，那么请参考一下的文章来进行安装：

如何在 Linux，MacOS 及 Windows 上进行安装 Elasticsearch
Kibana：如何在 Linux，MacOS 及 Windows 上安装 Elastic 栈中的 Kibana

在安装的时候，请选择 Elastic Stack 8.x 进行安装。在安装的时候，我们可以看到如下的安装信息：

为了能够上传向量模型，我们必须订阅白金版或试用。

Python

我们需要安装相应的 Elasticsearch 包：

$ pwd
/Users/liuxg/python/elser
$ pip3 install elasticsearch -qU
$ pip3 list | grep elasticseach
elasticsearch             8.11.1
rag-elasticsearch         0.0.1        /Users/liuxg/python/rag-elasticsearch/my-app/packages/rag-elasticsearch

环境变量

在启动 Jupyter 之前，我们设置如下的环境变量：

export ES_USER="elastic"
export ES_PASSWORD="yarOjyX5CLqTsKVE3v*d"
export ES_ENDPOINT="localhost"

拷贝 Elasticsearch 证书

我们把 Elasticsearch 的证书拷贝到当前的目录下：

$ pwd
/Users/liuxg/python/elser
$ cp ~/elastic/elasticsearch-8.11.0/config/certs/http_ca.crt .
$ ls
 find_books_about_christmas_without_searching_for_christmas.ipynb
Chatbot with LangChain conversational chain and OpenAI.ipynb
ElasticKnnSearch.ipynb
ElasticVectorSearch.ipynb
ElasticsearchStore.ipynb
Mental Health FAQ.ipynb
Multilingual semantic search.ipynb
NLP text search using hugging face transformer model.ipynb
Question Answering with Langchain and OpenAI.ipynb
RAG-langchain-elasticsearch.ipynb
Semantic search - ELSER.ipynb
Semantic search quick start.ipynb
book_summaries_1000_chunked.json
books.json
data.json
http_ca.crt
lib
sample_data.json
upgrading-index-to-use-elser.ipynb
vector_search_implementation_guide_api.ipynb
workplace-docs.json

在上面，我们把 Elasticsearch 的证书 http_ca.crt 拷贝到当前的目录下。

运行应用

连接到 Elasticsearch

from elasticsearch import Elasticsearch
import os
 
elastic_user=os.getenv('ES_USER')
elastic_password=os.getenv('ES_PASSWORD')
elastic_endpoint=os.getenv("ES_ENDPOINT")
 
url = f"https://{elastic_user}:{elastic_password}@{elastic_endpoint}:9200"
es = Elasticsearch(url, ca_certs = "./http_ca.crt", verify_certs = True)
 
print(es.info())

从上面的输出中，我们可以看到，连接到 Elasticsearch 是成功的。

如果你对如何连接到 Elasticsearch 还不是很熟悉的话，那么请阅读文章 “Elasticsearch：关于在 Python 中使用 Elasticsearch 你需要知道的一切 - 8.x”。

下载及部署 ELSER 模型

下面，我们来尝试通过软件的方式来针对 ELSER 进行手动部署。在此示例中，我们将下载 ELSER 模型并将其部署到 ML 节点中。确保你有一个 ML 节点才能运行 ELSER 模型。如果你之前已经下载过，我们通过软件的方式来进行删除，并安装最新的模型：

# delete model if already downloaded and deployed
try:
  es.ml.delete_trained_model(model_id=".elser_model_2",force=True)
  print("Model deleted successfully, We will proceed with creating one")
except exceptions.NotFoundError:
  print("Model doesn't exist, but We will proceed with creating one")

# Creates the ELSER model configuration. Automatically downloads the model if it doesn't exist. 
es.ml.put_trained_model(
    model_id=".elser_model_2",
    input={
      "field_names": ["text_field"]
    }
  )

我们回到 Kibana 的界面中进行查看：

上面显示， .elser_model_2 正在被下载。我们需要等一段时间才能下载完毕。这个依赖于你自己的网路速度。使用以下命令检查模型下载的状态。

while True:
    status = es.ml.get_trained_models(
        model_id=".elser_model_2",
        include="definition_status"
    )
    
    if (status["trained_model_configs"][0]["fully_defined"]):
        print("ELSER Model is downloaded and ready to be deployed.")
        break
    else:
        print("ELSER Model is downloaded but not ready to be deployed.")
    time.sleep(5)

在 Kibana 中显示的状态为：

下载完模型后，我们可以将模型部署到 ML 节点中。使用以下命令部署模型。

import time

# Start trained model deployment if not already deployed
es.ml.start_trained_model_deployment(
  model_id=".elser_model_2",
  number_of_allocations=1,
  wait_for="starting"
)

如上所示，在 Kibana 的界面中，我们可以看到 .elser_model_2 已经被成功地部署了。我们可以使用如下的代码来查看状态：

while True:
  status = es.ml.get_trained_models_stats(
    model_id=".elser_model_2",
  )
  if (status["trained_model_stats"][0]["deployment_stats"]["state"] == "started"):
    print("ELSER Model has been successfully deployed.")
    break
  else:
    print("ELSER Model is currently being deployed.")
  time.sleep(5)

摄入一些文档到 Elasticsearch

为了在我们的 Elasticsearch 中使用 ELSER，我们需要创建一个包含运行 ELSER 模型的推理处理器的摄取管道。让我们使用 put_pipeline 方法添加该管道。

es.ingest.put_pipeline(
    id="elser-ingest-pipeline", 
    description="Ingest pipeline for ELSER",
    processors=[
    {
      "inference": {
        "model_id": ".elser_model_2",
        "input_output": [
            {
              "input_field": "plot",
              "output_field": "plot_embedding"
            }
          ]
      }
    }
  ]
)

让我们记下该 API 调用中的一些重要参数：

inference：使用机器学习模型执行推理的处理器。
model_id：指定要使用的机器学习模型的 ID。在此示例中，模型 ID 设置为 .elser_model_2。
input_output：指定输入和输出字段
input_field：创建稀疏向量表示的字段名称。
output_field：包含推理结果的字段名称。

创建索引

要在索引时使用 ELSER 模型，我们需要创建支持 text_expansion 查询的索引映射。该映射包括一个 sparse_vector 类型的字段，用于处理我们感兴趣的特征向量。该字段包含 ELSER 模型根据输入文本创建的 token-weight 对。

让我们使用我们需要的映射创建一个名为 elser-example-movies 的索引。

es.indices.delete(index="elser-example-movies", ignore_unavailable=True)
es.indices.create(
  index="elser-example-movies",
  settings={
      "index": {
          "default_pipeline": "elser-ingest-pipeline"
      }
  },
  mappings={
    "properties": {
      "plot": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "plot_embedding": { 
        "type": "sparse_vector" 
      }
    }
  }
)

摄入文档

让我们插入 12 部电影的示例数据集。

如果出现错误，请检查模型是否已部署并且在 ML 节点中可用。在较新版本的 Elastic Cloud 中，ML 节点是自动缩放的，并且 ML 节点可能尚未准备好。等待几分钟，然后重试。在进行下面的运行之前，我们先在项目的根目录下创建如下的一个 movies.json 文档：

movies.json

[
    {
    "title": "Pulp Fiction",
    "runtime": "154",
    "plot": "The lives of two mob hitmen, a boxer, a gangster and his wife, and a pair of diner bandits intertwine in four tales of violence and redemption.",
    "keyScene": "John Travolta is forced to inject adrenaline directly into Uma Thurman's heart after she overdoses on heroin.",
    "genre": "Crime, Drama",
    "released": "1994"
    },
    {
    "title": "The Dark Knight",
    "runtime": "152",
    "plot": "When the menace known as the Joker wreaks havoc and chaos on the people of Gotham, Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice.",
    "keyScene": "Batman angrily responds 'I’m Batman' when asked who he is by Falcone.",
    "genre": "Action, Crime, Drama, Thriller",
    "released": "2008"
    },
    {
    "title": "Fight Club",
    "runtime": "139",
    "plot": "An insomniac office worker and a devil-may-care soapmaker form an underground fight club that evolves into something much, much more.",
    "keyScene": "Brad Pitt explains the rules of Fight Club to Edward Norton. The first rule of Fight Club is: You do not talk about Fight Club. The second rule of Fight Club is: You do not talk about Fight Club.",
    "genre": "Drama",
    "released": "1999"
    },
    {
    "title": "Inception",
    "runtime": "148",
    "plot": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into thed of a C.E.O.",
    "keyScene": "Leonardo DiCaprio explains the concept of inception to Ellen Page by using a child's spinning top.",
    "genre": "Action, Adventure, Sci-Fi, Thriller",
    "released": "2010"
    },
    {
    "title": "The Matrix",
    "runtime": "136",
    "plot": "A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.",
    "keyScene": "Red pill or blue pill? Morpheus offers Neo a choice between the red pill, which will allow him to learn the truth about the Matrix, or the blue pill, which will return him to his former life.",
    "genre": "Action, Sci-Fi",
    "released": "1999"
    },
    {
    "title": "The Shawshank Redemption",
    "runtime": "142",
    "plot": "Two imprisoned men bond over a number of years, finding solace and eventual redemption through acts of common decency.",
    "keyScene": "Andy Dufresne escapes from Shawshank prison by crawling through a sewer pipe.",
    "genre": "Drama",
    "released": "1994"
    },
    {
    "title": "Goodfellas",
    "runtime": "146",
    "plot": "The story of Henry Hill and his life in the mob, covering his relationship with his wife Karen Hill and his mob partners Jimmy Conway and Tommy DeVito in the Italian-American crime syndicate.",
    "keyScene": "Joe Pesci's character Tommy DeVito shoots young Spider in the foot for not getting him a drink.",
    "genre": "Biography, Crime, Drama",
    "released": "1990"
    },
    {
    "title": "Se7en",
    "runtime": "127",
    "plot": "Two detectives, a rookie and a veteran, hunt a serial killer who uses the seven deadly sins as his motives.",
    "keyScene": "Brad Pitt's character David Mills shoots John Doe after he reveals that he murdered Mills' wife.",
    "genre": "Crime, Drama, Mystery, Thriller",
    "released": "1995"
    },
    {
    "title": "The Silence of the Lambs",
    "runtime": "118",
    "plot": "A young F.B.I. cadet must receive the help of an incarcerated and manipulative cannibal killer to help catch another serial killer, a madman who skins his victims.",
    "keyScene": "Hannibal Lecter explains to Clarice Starling that he ate a census taker's liver with some fava beans and a nice Chianti.",
    "genre": "Crime, Drama, Thriller",
    "released": "1991"
    },
    {
    "title": "The Godfather",
    "runtime": "175",
    "plot": "An organized crime dynasty's aging patriarch transfers control of his clandestine empire to his reluctant son.",
    "keyScene": "James Caan's character Sonny Corleone is shot to death at a toll booth by a number of machine gun toting enemies.",
    "genre": "Crime, Drama",
    "released": "1972"
    },
    {
    "title": "The Departed",
    "runtime": "151",
    "plot": "An undercover cop and a mole in the police attempt to identify each other while infiltrating an Irish gang in South Boston.",
    "keyScene": "Leonardo DiCaprio's character Billy Costigan is shot to death by Matt Damon's character Colin Sullivan.",
    "genre": "Crime, Drama, Thriller",
    "released": "2006"
    },
    {
    "title": "The Usual Suspects",
    "runtime": "106",
    "plot": "A sole survivor tells of the twisty events leading up to a horrific gun battle on a boat, which began when five criminals met at a seemingly random police lineup.",
    "keyScene": "Kevin Spacey's character Verbal Kint is revealed to be the mastermind behind the crime, when his limp disappears as he walks away from the police station.",
    "genre": "Crime, Mystery, Thriller",
    "released": "1995"
    }
]

$ pwd
/Users/liuxg/python/elser
$ ls Install\ ELSER.ipynb 
Install ELSER.ipynb
$ ls movies.json 
movies.json

import json
from elasticsearch import helpers
 
with open('movies.json') as f:
   data_json = json.load(f)
 
# Prepare the documents to be indexed
documents = []
for doc in data_json:
    documents.append({
        "_index": "elser-example-movies",
        "_source": doc,
    })
 
# Use helpers.bulk to index
helpers.bulk(es, documents)
 
print("Done indexing documents into `elser-example-movies` index!")
time.sleep(3)

检查新文档以确认它现在有一个 plot_embedding 字段，其中包含新的附加术语列表。这些术语是创建管道时在 input_field 中用于 ELSER 推理的目标字段的文本扩展。 ELSER 实质上创建了一个扩展术语树，以提高文档的语义可搜索性。我们将能够使用 text_expansion 查询来搜索这些文档。我们可以在 Kibana 中查看：

但首先让我们从简单的关键字搜索开始，看看 ELSER 如何提供开箱即用的语义相关结果。

搜索文档

response = es.search(
    index='elser-example-movies', 
    size=3,
    query={
        "text_expansion": {
            "plot_embedding": {
                "model_id":".elser_model_2",
                "model_text":"fighting movie"
            }
        }
    }
)

for hit in response['hits']['hits']:
    doc_id = hit['_id']
    score = hit['_score']
    title = hit['_source']['title']
    plot = hit['_source']['plot']
    print(f"Score: {score}\nTitle: {title}\nPlot: {plot}\n")