爬虫
爬虫是什么
爬虫简单的来说就是用程序获取网络上数据这个过程的一种名称。
爬虫的原理
如果要获取网络上数据,我们要给爬虫一个网址(程序中通常叫URL),爬虫发送一个HTTP请求给目标网页的服务器,服务器返回数据给客户端(也就是我们的爬虫),爬虫再进行数据解析、保存等一系列操作。
流程
爬虫可以节省我们的时间,比如我要获取豆瓣电影 Top250 榜单,如果不用爬虫,我们要先在浏览器上输入豆瓣电影的 URL ,客户端(浏览器)通过解析查到豆瓣电影网页的服务器的 IP 地址,然后与它建立连接,浏览器再创造一个 HTTP 请求发送给豆瓣电影的服务器,服务器收到请求之后,把 Top250 榜单从数据库中提出,封装成一个 HTTP 响应,然后将响应结果返回给浏览器,浏览器显示响应内容,我们看到数据。我们的爬虫也是根据这个流程,只不过改成了代码形式。
HTTP请求
HTTP 请求由请求行、请求头、空行、请求体组成。
请求行由三部分组成:
1.请求方法,常见的请求方法有 GET、POST、PUT、DELETE、HEAD 2.客户端要获取的资源路径 3.是客户端使用的 HTTP 协议版本号
请求头是客户端向服务器发送请求的补充说明,比如说明访问者身份,这个下面会讲到。
请求体是客户端向服务器提交的数据,比如用户登录时需要提高的账号密码信息。请求头与请求体之间用空行隔开。请求体并不是所有的请求都有的,比如一般的GET都不会带有请求体。
上图就是浏览器登录豆瓣时向服务器发送的HTTP POST 请求,请求体中指定了用户名和密码。
HTTP 响应
HTTP 响应格式与请求的格式很相似,也是由响应行、响应头、空行、响应体组成。
响应行也包含三部分,分别是服务端的 HTTP 版本号、响应状态码和状态说明。
这里状态码有一张表,对应了各个状态码的意思
第二部分就是响应头,响应头与请求头对应,是服务器对该响应的一些附加说明,比如响应内容的格式是什么,响应内容的长度有多少、什么时间返回给客户端的、甚至还有一些 Cookie 信息也会放在响应头里面。
第三部分是响应体,它才是真正的响应数据,这些数据其实就是网页的 HTML 源代码。
爬虫代码怎么写
爬虫可以用很多语言比如 Python、C++等等,但是我觉得Python是最简单的,
因为Python有现成可用的库,已经封装到几乎完美,
C++虽然也有现成的库,但是它的爬虫还是比较小众,仅有的库也不足以算上简单,而且代码在各个编译器上,甚至同一个编译器上不同版本的兼容性不强,所以不是特别好用。所以今天主要介绍python爬虫。
安装requests库
cmd运行:pip install requests ,安装 requests。
然后在 IDLE 或者编译器(个人推荐 VS Code 或者 Pycharm )上输入
import requests 运行,如果没有报错,证明安装成功。
安装大部分库的方法都是:pip install xxx(库的名字)
requests的方法
requests.request() | 构造一个请求,支撑一下各方法的基本方法 |
requests.get() | 获取HTML网页的主要方法,对应于HTTP的GET |
requests.head() | 获取HTML网页头信息的方法,对应于HTTP的HEAD |
requests.post() | 向HTML网页提交POST请求的方法,对应于HTTP的POST |
requests.put() | 向HTML网页提交PUT请求的方法,对应于HTTP的PUT |
requests.patch( ) | 向HTML网页提交局部修改请求,对应于HTTP的PATCT |
requests.delete() | 向HTML网页提交删除请求,对应于HTTP的DELETE |
最常用的get方法
r = requests.get(url)
包括两个重要的对象:
构造一个向服务器请求资源的Request对象;返回一个包含服务器资源的Response对象
r.status_code | HTTP请求的返回状态,200表示连接成功,404表示失败 |
r.text | HTTP响应内容的字符串形式,即,url对应的页面内容 |
r.encoding | 从HTTP header中猜测的响应内容编码方式(如果header中不存在charset,则认为编码为ISO-8859-1) |
r.apparent_encoding | 从内容中分析的响应内容编码方式(备选编码方式) |
r.content | HTTP响应内容的二进制形式 |
requests.ConnectionError | 网络连接错误异常,如DNS查询失败、拒绝连接等 |
requests.HTTPError | HTTP错误异常 |
requests.URLRequired | URL缺失异常 |
requests.TooManyRedirects | 超过最大重定向次数,产生重定向异常 |
requests.ConnectTimeout | 连接远程服务器超时异常 |
requests.Timeout | 请求URL超时,产生超时异常 |
爬虫小demo
requests是最基础的爬虫库,但是我们可以做一个简单的翻译
我先把我做的一个爬虫的小项目的项目结构放上,完整源码可以私聊我下载。
下面是翻译部分的源码
import requests
def English_Chinese():url = "https://fanyi.baidu.com/sug"s = input("请输入要翻译的词(中/英):")dat = {"kw":s}resp = requests.post(url,data = dat)# 发送post请求ch = resp.json() # 将服务器返回的内容直接处理成json => dictresp.close()dic_lenth = len(ch['data'])for i in range(dic_lenth):print("词:"+ch['data'][i]['k']+" "+"单词意思:"+ch['data'][i]['v'])
代码详解:
导入requests模块,设置 url为百度翻译网页的网址。
然后通过 post 方法发送请求,再把返回的结果打成一个 dic (字典),但是这个时候我们打印出来结果发现是这样的。
这是一个字典里套列表套字典的样子,大概就是这样的
{ xx:xx , xx:[ {xx:xx} , {xx:xx} , {xx:xx} , {xx:xx} ] }
我标红的地方是我们需要的信息。
假如说我标蓝色的列表里面有 n 个字典,我们可以通过 len() 函数获取 n 的数值,
并使用 for 循环遍历,得到结果。
dic_lenth = len(ch['data']
for i in range(dic_lenth):print("词:"+ch['data'][i]['k']+" "+"单词意思:"+ch['data'][i]['v'])
最后
好了,今天的分享就到这里了,拜拜~
哎?忘了一件事,再给你们一个爬取天气的代码!
# -*- coding:utf-8 -*-
import requests
import bs4def get_web(url):header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"}res = requests.get(url, headers=header, timeout=5)# print(res.encoding)content = res.text.encode('ISO-8859-1')return contentdef parse_content(content):soup = bs4.BeautifulSoup(content, 'lxml')'''存放天气情况'''list_weather = []weather_list = soup.find_all('p', class_='wea')for i in weather_list:list_weather.append(i.text)'''存放日期'''list_day = []i = 0day_list = soup.find_all('h1')for each in day_list:if i <= 6:list_day.append(each.text.strip())i += 1# print(list_day)'''存放温度:最高温度和最低温度'''tem_list = soup.find_all('p', class_='tem')i = 0list_tem = []for each in tem_list:if i == 0:list_tem.append(each.i.text)i += 1elif i > 0:list_tem.append([each.span.text, each.i.text])i += 1# print(list_tem)'''存放风力'''list_wind = []wind_list = soup.find_all('p', class_='win')for each in wind_list:list_wind.append(each.i.text.strip())# print(list_wind)return list_day, list_weather, list_tem, list_winddef get_content(url):content = get_web(url)day, weather, tem, wind = parse_content(content)item = 0for i in range(0, 7):if item == 0:print(day[i]+':\t')print(weather[i]+'\t')print("今日气温:"+tem[i]+'\t')print("风力:"+wind[i]+'\t')print('\n')item += 1elif item > 0:print(day[i]+':\t')print(weather[i] + '\t')print("最高气温:"+tem[i][0]+'\t')print("最低气温:"+tem[i][1] + '\t')print("风力:"+wind[i]+'\t')print('\n')> 如果是初学python爬虫,觉得自学比较难的,那么我接下来分享的这全套 Python 学习资料一定不要错过,希望能给那些想学习 Python 的小伙伴们带来帮助!#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)python学习路线![](https://img-blog.csdnimg.cn/img_convert/20007356b0039036a10dfd04dac0f8cd.png)#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)环境搭建使用 Python 首先需要搭建 Python 环境,我们直接到 [Python 官网](https://link.zhihu.com/?target=https%3A//www.python.org/downloads) 下载自己对应平台和版本的安装包进行安装即可#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)python开发工具俗话说:工欲善其事,必先利其器,Python 的学习也是一样,新手建议首选 PyCharm,可以快速上手,减少配置时间。![](https://img-blog.csdnimg.cn/img_convert/f9eb58520b98389d7cce601939a25def.png)#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)[](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)学习软件
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------工欲善其事必先利其器。学习Python常用的开发软件都在这里了,给大家节省了很多时间。![](https://img-blog.csdnimg.cn/img_convert/f874af3c700721cfa7b482156e7da47c.png)### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)全套PDF电子书书籍的好处就在于权威和体系健全,刚开始学习的时候你可以只看视频或者听某个人讲课,但等你学完之后,你觉得你掌握了,这时候建议还是得去看一下书籍,看权威技术书籍也是每个程序员必经之路。![在这里插入图片描述](https://img-blog.csdnimg.cn/7844e927f29841ed8992e71b5f3a9919.jpeg#pic_center)### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)入门学习视频我们在看视频学习的时候,不能光动眼动脑不动手,比较科学的学习方法是在理解之后运用它们,这时候练手项目就很适合了。![](https://img-blog.csdnimg.cn/img_convert/06e6d22e6bd1624b0689923a2e55ff81.png)### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)实战案例光学理论是没用的,要学会跟着一起敲,要动手实操,才能将自己的所学运用到实际当中去,这时候可以搞点实战案例来学习。![](https://img-blog.csdnimg.cn/img_convert/982df5e1a2e44f99cb4d19f3f7aad16c.png)[](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)五、清华编程大佬出品《漫画看学Python》
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------用通俗易懂的漫画,来教你学习Python,让你更容易记住,并且不会枯燥乏味。![在这里插入图片描述](https://img-blog.csdnimg.cn/097b0de3966849f6a546a68612cc6fd1.png#pic_center)
**配套600集视频:**![在这里插入图片描述](https://img-blog.csdnimg.cn/76e119b440784cac8f92c00b12d169e1.png#pic_center)### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)六、面试资料我们学习Python必然是为了找到高薪的工作,下面这些面试题是来自阿里、腾讯、字节等一线互联网大厂最新的面试资料,并且有阿里大佬给出了权威的解答,刷完这一套面试资料相信大家都能找到满意的工作。![](https://img-blog.csdnimg.cn/img_convert/017aa5fae0c560e129e56957eaa86eb4.png)
![](https://img-blog.csdnimg.cn/img_convert/ac350b25859669a72511ef8d063425eb.png)
**上面这份完整版的Python全套学习资料已经上传至CSDN官方,朋友如果需要可以直接点击[CSDN官方认证二维码免费领取【保证100%免费】。](https://mp.weixin.qq.com/s?__biz=MzI3OTQ1NjMyMg==&mid=2247485003&idx=1&sn=0b3e59328d8cedb37fc756fde6b9f422&chksm=eb463966dc31b070fa76b5a3644d615da265619109ec42ff48976e5a65399f9cb0fc6cc3c17b#rd)**