Python爬取糗事百科段子

---

糗事百科是一个原创的糗事笑话分享社区,糗百网友分享的搞笑段子，使用Python爬取段子

翻页操作

http://www.qiushibaike.com/text/
http://www.qiushibaike.com/text/page/2/?s=4964629
http://www.qiushibaike.com/text/page/3/?s=4964629
http://www.qiushibaike.com/text/page/4/?s=4964629

观察可知，我们只需要修改page/{i}参数即可

这里使用正则表达式来提取数据

import requests
import re
from fake_useragent import UserAgent
headers= {'User-Agent':str(UserAgent().random)}def get_info(url):res = requests.get(url,headers=headers)ids = re.findall('<h2>(.*?)</h2>',res.text,re.S)levels = re.findall('<div class="articleGender \D+Icon">(.*?)</div>',res.text,re.S)contents = re.findall('<div class="content">.*?<span>(.*?)</span>',res.text,re.S)laughs = re.findall('<span class="stats-vote"><i class="number">(\d+)</i>',res.text,re.S)comments = re.findall('<i class="number">(\d+)</i> 评论',res.text,re.S)for id,level,sex,content,laugh,comment in zip(ids,levels,sexs,contents,laughs,comments):info = {'id':id,'level':level,'content':content,'laugh':laugh,'comment':comment}print(info)if __name__ == '__main__':urls = ['http://www.qiushibaike.com/text/page/{}/'.format(str(i)) for i in range(1,15)]for url in urls:get_info(url)