How to build an AI that can answer questions about your website 如何建立一个人工智能，回答有关您的网站的问题

This tutorial walks through a simple example of crawling a website (in this example, the OpenAI website), turning the crawled pages into embeddings using the Embeddings API, and then creating a basic search functionality that allows a user to ask questions about the embedded information. This is intended to be a starting point for more sophisticated applications that make use of custom knowledge bases.
本教程介绍了一个简单的示例，该示例包括爬取网站（在本例中为OpenAI网站），使用Embeddings API将爬取的页面转换为嵌入内容，然后创建一个基本的搜索功能，允许用户询问有关嵌入信息的问题。这旨在作为使用自定义知识库进行更复杂应用程序的起点。

Getting started 入门

Some basic knowledge of Python and GitHub is helpful for this tutorial. Before diving in, make sure to set up an OpenAI API key and walk through the quickstart tutorial. This will give a good intuition on how to use the API to its full potential.
Python和GitHub的一些基本知识对本教程很有帮助。在开始之前，请确保设置OpenAI API密钥并浏览快速入门教程。这将使您对如何充分利用API的潜力有一个很好的直觉。

Python is used as the main programming language along with the OpenAI, Pandas, transformers, NumPy, and other popular packages. If you run into any issues working through this tutorial, please ask a question on the OpenAI Community Forum.
Python与OpenAI、Pandas、transformers、NumPy和其他流行的功能包被用作主要编程语言。如果您在本教程中遇到任何问题，请在OpenAI社区论坛上提问。

To start with the code, clone the full code for this tutorial on GitHub. Alternatively, follow along and copy each section into a Jupyter notebook and run the code step by step, or just read along. A good way to avoid any issues is to set up a new virtual environment and install the required packages by running the following commands:
要从代码开始，请在GitHub上克隆本教程的完整代码。或者，按照下面的步骤操作，将每一部分复制到Jupyter笔记本中，然后一步一步地运行代码，或者只是阅读。避免任何问题的一个好方法是通过运行以下命令设置新的虚拟环境并安装所需的软件包：

python -m venv envsource env/bin/activatepip install -r requirements.txt

Setting up a web crawler 设置Web爬网程序

The primary focus of this tutorial is the OpenAI API so if you prefer, you can skip the context on how to create a web crawler and just download the source code. Otherwise, expand the section below to work through the scraping mechanism implementation.
本教程的重点是OpenAI API，因此如果您愿意，可以跳过有关如何创建网络爬虫的上下文，直接下载源代码。否则，请展开下面的部分以完成刮擦机制的实现。

Learn how to build a web crawler 了解如何构建网络爬虫

Acquiring data in text form is the first step to use embeddings. This tutorial creates a new set of data by crawling the OpenAI website, a technique that you can also use for your own company or personal website.
获取文本形式的数据是使用嵌入的第一步。本教程通过爬取OpenAI网站创建了一组新的数据，这种技术也可以用于您自己的公司或个人网站。

While this crawler is written from scratch, open source packages like Scrapy can also help with these operations.
虽然这个爬虫是从头开始编写的，但是像Scrapy这样的开源包也可以帮助完成这些操作。

This crawler will start from the root URL passed in at the bottom of the code below, visit each page, find additional links, and visit those pages as well (as long as they have the same root domain). To begin, import the required packages, set up the basic URL, and define a HTMLParser class.
这个爬虫将从下面代码底部传入的根URL开始，访问每个页面，查找其他链接，并访问这些页面（只要它们具有相同的根域）。开始，导入所需的包，设置基本URL，并定义一个HTMLParser类。

import requests
import re
import urllib.request
from bs4 import BeautifulSoup
from collections import deque
from html.parser import HTMLParser
from urllib.parse import urlparse
import os# Regex pattern to match a URL
HTTP_URL_PATTERN = r'^http[s]*://.+'domain = "openai.com" # <- put your domain to be crawled
full_url = "https://openai.com/" # <- put your domain to be crawled with https or http# Create a class to parse the HTML and get the hyperlinks
class HyperlinkParser(HTMLParser):def __init__(self):super().__init__()# Create a list to store the hyperlinksself.hyperlinks = []# Override the HTMLParser's handle_starttag method to get the hyperlinksdef handle_starttag(self, tag, attrs):attrs = dict(attrs)# If the tag is an anchor tag and it has an href attribute, add the href attribute to the list of hyperlinksif tag == "a" and "href" in attrs:self.hyperlinks.append(attrs["href"])

The next function takes a URL as an argument, opens the URL, and reads the HTML content. Then, it returns all the hyperlinks found on that page.
下一个函数将URL作为参数，打开URL并读取HTML内容。然后，它返回在该页上找到的所有超链接。

# Function to get the hyperlinks from a URL
def get_hyperlinks(url):# Try to open the URL and read the HTMLtry:# Open the URL and read the HTMLwith urllib.request.urlopen(url) as response:# If the response is not HTML, return an empty listif not response.info().get('Content-Type').startswith("text/html"):return []# Decode the HTMLhtml = response.read().decode('utf-8')except Exception as e:print(e)return []# Create the HTML Parser and then Parse the HTML to get hyperlinksparser = HyperlinkParser()parser.feed(html)return parser.hyperlinks

The goal is to crawl through and index only the content that lives under the OpenAI domain. For this purpose, a function that calls the get_hyperlinks function but filters out any URLs that are not part of the specified domain is needed.
目标是只对OpenAI域下的内容进行爬行和索引。为此，需要一个调用 get_hyperlinks 函数但过滤掉不属于指定域的任何URL的函数。

# Function to get the hyperlinks from a URL that are within the same domain
def get_domain_hyperlinks(local_domain, url):clean_links = []for link in set(get_hyperlinks(url)):clean_link = None# If the link is a URL, check if it is within the same domainif re.search(HTTP_URL_PATTERN, link):# Parse the URL and check if the domain is the sameurl_obj = urlparse(link)if url_obj.netloc == local_domain:clean_link = link# If the link is not a URL, check if it is a relative linkelse:if link.startswith("/"):link = link[1:]elif link.startswith("#") or link.startswith("mailto:"):continueclean_link = "https://" + local_domain + "/" + linkif clean_link is not None:if clean_link.endswith("/"):clean_link = clean_link[:-1]clean_links.append(clean_link)# Return the list of hyperlinks that are within the same domainreturn list(set(clean_links))

The crawl function is the final step in the web scraping task setup. It keeps track of the visited URLs to avoid repeating the same page, which might be linked across multiple pages on a site. It also extracts the raw text from a page without the HTML tags, and writes the text content into a local .txt file specific to the page.
crawl 函数是网页抓取任务设置的最后一步。它跟踪访问过的URL，以避免重复同一个页面，这可能是跨网站上的多个页面链接。它还从没有HTML标记的页面中提取原始文本，并将文本内容写入特定于该页面的本地.txt文件。

def crawl(url):# Parse the URL and get the domainlocal_domain = urlparse(url).netloc# Create a queue to store the URLs to crawlqueue = deque([url])# Create a set to store the URLs that have already been seen (no duplicates)seen = set([url])# Create a directory to store the text filesif not os.path.exists("text/"):os.mkdir("text/")if not os.path.exists("text/"+local_domain+"/"):os.mkdir("text/" + local_domain + "/")# Create a directory to store the csv filesif not os.path.exists("processed"):os.mkdir("processed")# While the queue is not empty, continue crawlingwhile queue:# Get the next URL from the queueurl = queue.pop()print(url) # for debugging and to see the progress# Save text from the url to a <url>.txt filewith open('text/'+local_domain+'/'+url[8:].replace("/", "_") + ".txt", "w", encoding="UTF-8") as f:# Get the text from the URL using BeautifulSoupsoup = BeautifulSoup(requests.get(url).text, "html.parser")# Get the text but remove the tagstext = soup.get_text()# If the crawler gets to a page that requires JavaScript, it will stop the crawlif ("You need to enable JavaScript to run this app." in text):print("Unable to parse page " + url + " due to JavaScript being required")# Otherwise, write the text to the file in the text directoryf.write(text)# Get the hyperlinks from the URL and add them to the queuefor link in get_domain_hyperlinks(local_domain, url):if link not in seen:queue.append(link)seen.add(link)crawl(full_url)

The last line of the above example runs the crawler which goes through all the accessible links and turns those pages into text files. This will take a few minutes to run depending on the size and complexity of your site.
上例的最后一行运行了爬行器，它遍历所有可访问的链接，并将这些页面转换为文本文件。根据站点的大小和复杂性，这将需要几分钟的时间来运行。

Building an embeddings index 构建嵌入索引

def remove_newlines(serie):serie = serie.str.replace('\n', ' ')serie = serie.str.replace('\\n', ' ')serie = serie.str.replace('  ', ' ')serie = serie.str.replace('  ', ' ')return serie

import pandas as pd# Create a list to store the text files
texts=[]# Get all the text files in the text directory
for file in os.listdir("text/" + domain + "/"):# Open the file and read the textwith open("text/" + domain + "/" + file, "r", encoding="UTF-8") as f:text = f.read()# Omit the first 11 lines and the last 4 lines, then replace -, _, and #update with spaces.texts.append((file[11:-4].replace('-',' ').replace('_', ' ').replace('#update',''), text))# Create a dataframe from the list of texts
df = pd.DataFrame(texts, columns = ['fname', 'text'])# Set the text column to be the raw text with the newlines removed
df['text'] = df.fname + ". " + remove_newlines(df.text)
df.to_csv('processed/scraped.csv')
df.head()

import tiktoken# Load the cl100k_base tokenizer which is designed to work with the ada-002 model
tokenizer = tiktoken.get_encoding("cl100k_base")df = pd.read_csv('processed/scraped.csv', index_col=0)
df.columns = ['title', 'text']# Tokenize the text and save the number of tokens to a new column
df['n_tokens'] = df.text.apply(lambda x: len(tokenizer.encode(x)))# Visualize the distribution of the number of tokens per row using a histogram
df.n_tokens.hist()

max_tokens = 500# Function to split the text into chunks of a maximum number of tokens
def split_into_many(text, max_tokens = max_tokens):# Split the text into sentencessentences = text.split('. ')# Get the number of tokens for each sentencen_tokens = [len(tokenizer.encode(" " + sentence)) for sentence in sentences]chunks = []tokens_so_far = 0chunk = []# Loop through the sentences and tokens joined together in a tuplefor sentence, token in zip(sentences, n_tokens):# If the number of tokens so far plus the number of tokens in the current sentence is greater # than the max number of tokens, then add the chunk to the list of chunks and reset# the chunk and tokens so farif tokens_so_far + token > max_tokens:chunks.append(". ".join(chunk) + ".")chunk = []tokens_so_far = 0# If the number of tokens in the current sentence is greater than the max number of # tokens, go to the next sentenceif token > max_tokens:continue# Otherwise, add the sentence to the chunk and add the number of tokens to the totalchunk.append(sentence)tokens_so_far += token + 1return chunksshortened = []# Loop through the dataframe
for row in df.iterrows():# If the text is None, go to the next rowif row[1]['text'] is None:continue# If the number of tokens is greater than the max number of tokens, split the text into chunksif row[1]['n_tokens'] > max_tokens:shortened += split_into_many(row[1]['text'])# Otherwise, add the text to the list of shortened textselse:shortened.append( row[1]['text'] )

df = pd.DataFrame(shortened, columns = ['text'])
df['n_tokens'] = df.text.apply(lambda x: len(tokenizer.encode(x)))
df.n_tokens.hist()

import openaidf['embeddings'] = df.text.apply(lambda x: openai.Embedding.create(input=x, engine='text-embedding-ada-002')['data'][0]['embedding'])df.to_csv('processed/embeddings.csv')
df.head()

Building a question answer system with your embeddings 使用嵌入式构建问答系统

The embeddings are ready and the final step of this process is to create a simple question and answer system. This will take a user’s question, create an embedding of it, and compare it with the existing embeddings to retrieve the most relevant text from the scraped website. The text-davinci-003 model will then generate a natural sounding answer based on the retrieved text.
嵌入已经准备好了，这个过程的最后一步是创建一个简单的问答系统。这将接受用户的问题，创建一个嵌入，并将其与现有的嵌入进行比较，以检索从抓取的网站最相关的文本。然后，text-davinci-003模型将基于检索到的文本生成听起来自然的回答。

Turning the embeddings into a NumPy array is the first step, which will provide more flexibility in how to use it given the many functions available that operate on NumPy arrays. It will also flatten the dimension to 1-D, which is the required format for many subsequent operations.
将嵌入转换为NumPy数组是第一步，这将为如何使用NumPy数组提供更大的灵活性，因为NumPy数组上有许多可用的函数。它还会将尺寸展平为1-D，这是许多后续操作所需的格式。

import numpy as np
from openai.embeddings_utils import distances_from_embeddingsdf=pd.read_csv('processed/embeddings.csv', index_col=0)
df['embeddings'] = df['embeddings'].apply(eval).apply(np.array)df.head()

The question needs to be converted to an embedding with a simple function, now that the data is ready. This is important because the search with embeddings compares the vector of numbers (which was the conversion of the raw text) using cosine distance. The vectors are likely related and might be the answer to the question if they are close in cosine distance. The OpenAI python package has a built in distances_from_embeddings function which is useful here.
既然数据已经准备好了，这个问题需要转换成一个简单函数的嵌入。这一点很重要，因为嵌入搜索使用余弦距离比较数字向量（原始文本的转换）。这两个向量很可能是相关的，如果它们的余弦距离很近，就可能是这个问题的答案。OpenAI python包有一个内置的 distances_from_embeddings 函数，在这里很有用。

def create_context(question, df, max_len=1800, size="ada"
):"""Create a context for a question by finding the most similar context from the dataframe"""# Get the embeddings for the questionq_embeddings = openai.Embedding.create(input=question, engine='text-embedding-ada-002')['data'][0]['embedding']# Get the distances from the embeddingsdf['distances'] = distances_from_embeddings(q_embeddings, df['embeddings'].values, distance_metric='cosine')returns = []cur_len = 0# Sort by distance and add the text to the context until the context is too longfor i, row in df.sort_values('distances', ascending=True).iterrows():# Add the length of the text to the current lengthcur_len += row['n_tokens'] + 4# If the context is too long, breakif cur_len > max_len:break# Else add it to the text that is being returnedreturns.append(row["text"])# Return the contextreturn "\n\n###\n\n".join(returns)

The text was broken up into smaller sets of tokens, so looping through in ascending order and continuing to add the text is a critical step to ensure a full answer. The max_len can also be modified to something smaller, if more content than desired is returned.
文本被分解为更小的标记集，因此按升序循环并继续添加文本是确保完整答案的关键步骤。如果返回的内容多于所需的内容，max_len也可以修改为更小的值。

The previous step only retrieved chunks of texts that are semantically related to the question, so they might contain the answer, but there’s no guarantee of it. The chance of finding an answer can be further increased by returning the top 5 most likely results.
上一步只检索了与问题语义相关的文本块，因此它们可能包含答案，但不能保证一定包含答案。通过返回前5个最可能的结果，可以进一步增加找到答案的机会。

The answering prompt will then try to extract the relevant facts from the retrieved contexts, in order to formulate a coherent answer. If there is no relevant answer, the prompt will return “I don’t know”.
然后，回答提示将尝试从检索到的上下文中提取相关事实，以便形成连贯的回答。如果没有相关答案，提示会返回“我不知道”。

A realistic sounding answer to the question can be created with the completion endpoint using text-davinci-003.
可以使用 text-davinci-003 创建具有完成终点的问题的实际答案。

def answer_question(df,model="text-davinci-003",question="Am I allowed to publish model outputs to Twitter, without a human review?",max_len=1800,size="ada",debug=False,max_tokens=150,stop_sequence=None
):"""Answer a question based on the most similar context from the dataframe texts"""context = create_context(question,df,max_len=max_len,size=size,)# If debug, print the raw model responseif debug:print("Context:\n" + context)print("\n\n")try:# Create a completions using the question and contextresponse = openai.Completion.create(prompt=f"Answer the question based on the context below, and if the question can't be answered based on the context, say \"I don't know\"\n\nContext: {context}\n\n---\n\nQuestion: {question}\nAnswer:",temperature=0,max_tokens=max_tokens,top_p=1,frequency_penalty=0,presence_penalty=0,stop=stop_sequence,model=model,)return response["choices"][0]["text"].strip()except Exception as e:print(e)return ""

It is done! A working Q/A system that has the knowledge embedded from the OpenAI website is now ready. A few quick tests can be done to see the quality of the output:
完成了！现在已经准备好了一个工作中的Q/A系统，其中包含了来自OpenAI网站的知识。可以进行一些快速测试来查看输出的质量：

answer_question(df, question="What day is it?", debug=False)answer_question(df, question="What is our newest embeddings model?")answer_question(df, question="What is ChatGPT?")

The responses will look something like the following:
响应将如下所示：

"I don't know."'The newest embeddings model is text-embedding-ada-002.''ChatGPT is a model trained to interact in a conversational way. It is able to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.'

If the system is not able to answer a question that is expected, it is worth searching through the raw text files to see if the information that is expected to be known actually ended up being embedded or not. The crawling process that was done initially was setup to skip sites outside the original domain that was provided, so it might not have that knowledge if there was a subdomain setup.
如果系统不能回答预期的问题，则需要搜索原始文本文件，以查看预期已知的信息是否最终被嵌入。最初完成的爬网过程被设置为跳过所提供的原始域之外的站点，因此如果设置了子域，它可能不知道这些站点。

Currently, the dataframe is being passed in each time to answer a question. For more production workflows, a vector database solution should be used instead of storing the embeddings in a CSV file, but the current approach is a great option for prototyping.
目前，每次都要传入数据帧来回答一个问题。对于更多的生产工作流，应该使用矢量数据库解决方案，而不是将嵌入存储在CSV文件中，但当前的方法是原型化的一个很好的选择。

其它资料下载

如果大家想继续了解人工智能相关学习路线和知识体系，欢迎大家翻阅我的另外一篇博客《重磅 | 完备的人工智能AI 学习——基础知识学习路线，所有资料免关注免套路直接网盘下载》
这篇博客参考了Github知名开源平台，AI技术平台以及相关领域专家：Datawhale，ApacheCN，AI有道和黄海广博士等约有近100G相关资料，希望能帮助到所有小伙伴们。