Python爬虫进阶：提升爬虫效率

文章目录

一、单线程+多任务异步协程
二、线程池+requests模块
三、两个方法提升爬虫效率
总结
- - 关于Python技术储备
  - - 一、Python所有方向的学习路线
    - 二、Python基础学习视频
    - 三、精品Python学习书籍
    - 四、Python工具包+项目源码合集
    - ①Python工具包
    - ②Python实战案例
    - ③Python小游戏源码
    - 五、面试资料
    - 六、Python兼职渠道

在这里插入图片描述

一、单线程+多任务异步协程

协程

在函数(特殊函数)定义的时候,使用async修饰,函数调用后,内部语句不会立即执行,而是会返回一个协程对象

任务对象

任务对象=高级的协程对象(进一步封装)=特殊的函数
任务对象必须要注册到时间循环对象中
给任务对象绑定回调:爬虫的数据解析中

事件循环

当做是一个装载任务对象的容器
当启动事件循环对象的时候,存储在内的任务对象会异步执行

特殊函数内部不能写不支持异步请求的模块,如time,requests…否则虽然不报错但实现不了异步

time.sleep – asyncio.sleep
requests – aiohttp

import asyncio
import timestart_time = time.time()
async def get_request(url):await asyncio.sleep(2)print(url,'下载完成!')urls = ['www.1.com','www.2.com',
]task_lst = [] # 任务对象列表
for url in urls:c = get_request(url) # 协程对象task = asyncio.ensure_future(c) # 任务对象# task.add_done_callback(...)  # 绑定回调task_lst.append(task)loop = asyncio.get_event_loop() # 事件循环对象
loop.run_until_complete(asyncio.wait(task_lst)) # 注册,手动挂起

二、线程池+requests模块

# 线程池
import time
from multiprocessing.dummy import Poolstart_time = time.time()
url_list = ['www.1.com','www.2.com','www.3.com',
]
def get_request(url):print('正在下载...',url)time.sleep(2)print('下载完成!',url)pool = Pool(3)
pool.map(get_request,url_list)
print('总耗时:',time.time()-start_time)

三、两个方法提升爬虫效率

起一个flask服务端

from flask import Flask
import timeapp = Flask(__name__)@app.route('/bobo')
def index_bobo():time.sleep(2)return 'hello bobo!'@app.route('/jay')
def index_jay():time.sleep(2)return 'hello jay!'@app.route('/tom')
def index_tom():time.sleep(2)return 'hello tom!'if __name__ == '__main__':app.run(threaded=True)

aiohttp模块+单线程多任务异步协程

import asyncio
import aiohttp
import requests
import timestart = time.time()
async def get_page(url):# page_text = requests.get(url=url).text# print(page_text)# return page_textasync with aiohttp.ClientSession() as s: #生成一个session对象async with await s.get(url=url) as response:page_text = await response.text()print(page_text)return page_texturls = ['http://127.0.0.1:5000/bobo','http://127.0.0.1:5000/jay','http://127.0.0.1:5000/tom',
]
tasks = []
for url in urls:c = get_page(url)task = asyncio.ensure_future(c)tasks.append(task)
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))end = time.time()
print(end-start)# 异步执行!
# hello tom!
# hello bobo!
# hello jay!
# 2.0311079025268555

'''
aiohttp模块实现单线程+多任务异步协程
并用xpath解析数据
'''
import aiohttp
import asyncio
from lxml import etree
import timestart = time.time()
# 特殊函数:请求的发送和数据的捕获
# 注意async with await关键字
async def get_request(url):async with aiohttp.ClientSession() as s:async with await s.get(url=url) as response:page_text = await response.text()return page_text    # 返回页面源码# 回调函数,解析数据
def parse(task):page_text = task.result()tree = etree.HTML(page_text)msg = tree.xpath('/html/body/ul//text()')print(msg)urls = ['http://127.0.0.1:5000/bobo','http://127.0.0.1:5000/jay','http://127.0.0.1:5000/tom',
]
tasks = []
for url in urls:c = get_request(url)task = asyncio.ensure_future(c)task.add_done_callback(parse) #绑定回调函数!tasks.append(task)
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))end = time.time()
print(end-start)

requests模块+线程池

import time
import requests
from multiprocessing.dummy import Poolstart = time.time()
urls = ['http://127.0.0.1:5000/bobo','http://127.0.0.1:5000/jay','http://127.0.0.1:5000/tom',
]
def get_request(url):page_text = requests.get(url=url).textprint(page_text)return page_textpool = Pool(3)
pool.map(get_request, urls)
end = time.time()
print('总耗时:', end-start)# 实现异步请求
# hello jay!
# hello bobo!
# hello tom!
# 总耗时: 2.0467123985290527