部署参考视频

通义千问-7B-预训练-模型库

一、部署1

1.1 打开通义千问-7B-预训练-模型库-选择资源

1.2 使用Netbook

弹出新页面

2.1 运行

2.2 复制脚本

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig
import datetime
print("启动时间：" + str(datetime.datetime.now()))
tokenizer = AutoTokenizer.from_pretrained("qwen/Qwen-7B-Chat", revision = 'v1.0.5',trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("qwen/Qwen-7B-Chat", revision = 'v1.0.5',device_map="auto",offload_folder="offload_folder", trust_remote_code=True,fp16 = True).eval()
model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B-Chat",revision = 'v1.0.5', trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参
model.float()print("开始执行：" + str(datetime.datetime.now()))
response, history = model.chat(tokenizer, "你好", history=None)
print(response)
print("第一个问题处理完毕：" + str(datetime.datetime.now()))
response, history = model.chat(tokenizer, "浙江的省会在哪里？", history=history) 
print(response)
print("第二个问题处理完毕：" + str(datetime.datetime.now()))
response, history = model.chat(tokenizer, "它有什么好玩的景点", history=history)
print(response)
print("第三个问题处理完毕：" + str(datetime.datetime.now()))

2.2.1 问题1 ：ImportError: This modeling file requires the following packages that were not found in your environment: transformers_stream_generator. Run pip install transformers_stream_generator

解决方法
pip install transformers_stream_generator




这就好了，重新运行下

ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

参照这哥们的
https://zhuanlan.zhihu.com/p/649272911

2.3 查看结果

3.其他作者的demo

3.1 克隆代码

git clone https://gitee.com/JokerBao/Qwen-7B-FastWeb.git

3.2 加载依赖

pip install -r requirements.txt

3.3 运行代码

python Qwen_demo.py

二、部署2

1.1 启动服务器

1.2 使用官方提供的demo

拉代码

git clone https://github.com/QwenLM/Qwen-7B.git

1.2 为了拉去大模型，更新LFS

apt-get update

apt-get install git-lfs

2.1 拉千问模型文件

git clone https://www.modelscope.cn/qwen/Qwen-7B-Chat.git

初始化一下

2.1.1模型下载成功

2.1.2 安装依赖

为了方便我把模型移动到一开始的文件夹里面

pip install -r requirements.txt

也可以使用web依赖

pip install -r requirements_web_demo.txt

3.1 修改web.demo.py地址

然后ctrl+s 保存

4 运行

 python web_demo.py 

5 成功

5开启量化

pip install bitsandbytes

添加依赖

from transformers import BitsAndBytesConfig
import torch

添加

quantization_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type='nf4',bnb_4bit_compute_dtype=torch.bfloat16)