Python爬虫实现HTTP网络请求多种实现方式

(编辑：jimmy 日期: 2024/11/18 浏览：3 次 )

1、通过urllib.requests模块实现发送请求并读取网页内容的简单示例如下：

#导入模块
import urllib.request
#打开需要爬取的网页
response = urllib.request.urlopen('http://www.baidu.com')
#读取网页代码
html = response.read()
#打印读取的内容
print(html)

结果：

b'<!DOCTYPE html><!--STATUS OK-->\n\n\n \n \n       <html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#2932e1"><meta name="description" content="\xe5\x85\xa8\xe7\x90\x83\xe6\x9c\x80\xe5\xa4\xa7\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe6\x90\x9c\xe7\xb4\xa2\xe5\xbc\x95\xe6\x93\x8e\xe3\x80\x81\xe8\x87\xb4\xe5\x8a\x9b\xe4\xba\x8e\xe8\xae\xa9\xe7\xbd\x91\xe6\xb0\x91\xe6\x9b\xb4\xe4\xbe\xbf\xe6\x8d\xb7\xe5\x9c\xb0\xe8\x8e\xb7\xe5\x8f\x96\xe4\xbf\xa1\xe6\x81\xaf\xef\xbc\x8c\xe6\x89\xbe\xe5\x88\xb0\xe6\x89\x80\xe6\xb1\x82\xe3\x80\x82\xe7\x99\xbe\xe5\xba\xa6\xe8\xb6\x85\xe8\xbf\x87\xe5\x8d\x83\xe4\xba\xbf\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe7\xbd\x91\xe9\xa1\xb5\xe6\x95\xb0\xe6\x8d\xae\xe5\xba\x93\xef\xbc\x8c\xe5\x8f\xaf\xe4\xbb\xa5\xe7\x9e\xac\xe9\x97\xb4\xe6\x89\xbe\xe5\x88\xb0\xe7\x9b\xb8\xe5\x85\xb3\xe7\x9a\x84\xe6\x90\x9c\xe7\xb4\xa2\xe7\xbb\x93\xe6\x9e\x9c\xe3\x80\x82"><link rel="shortcut icon" href="/favicon.ico" rel="external nofollow" type="image/x-icon" /><link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" rel="external nofollow" title="\xe7\x99\xbe\xe5\xba\xa6\xe6\x90\x9c\xe7\xb4\xa2" /><link rel="icon" sizes="any" mask href="//www.baidu.com/img/baidu_85beaf5496f291521eb75ba38eacbd87.svg" rel="external nofollow" ><link rel="dns-prefetch" href="//dss0.bdstatic.com" rel="external nofollow" /><link rel="dns-prefetch" href="//dss1.bdstatic.com" rel="external nofollow" /><link rel="dns-prefetch" href="//ss1.bdstatic.com" rel="external nofollow" /><link rel="dns-prefetch" href="//sp0.baidu.com" rel="external nofollow" /><link rel="dns-prefetch" href="//sp1.baidu.com" rel="external nofollow" /><link rel="dns-prefetch" href="//sp2.baidu.com" rel="external nofollow" /><title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9\x81\x93</title><style index="newi" type="text/css">#form .bdsug{top:39px}.bdsug{display:none;position:absolute;width:535px;background:#fff;border:1px solid 
………………（太多省略）

以上示例中是通过get请求方式获取百度的网页内容。

下面是通过urllib.request模块的post请求实现获取网页信息的内容：

#导入模块
import urllib.parse
import urllib.request
#将数据使用urlencode编码处理后，再使用encoding设置为utf-8编码
data = bytes(urllib.parse.urlencode({'word':'hello'}),encoding='utf-8')
#打开指定需要爬取的网页
response = urllib.request.urlopen('http://httpbin.org/post',data=data)
html = response.read()
#打印读取的内容
print(html)

结果：

b'{\n "args": {}, \n "data": "", \n "files": {}, \n "form": {\n "word": "hello"\n }, \n "headers": {\n "Accept-Encoding": "identity", \n "Content-Length": "10", \n "Content-Type": "application/x-www-form-urlencoded", \n "Host": "httpbin.org", \n "User-Agent": "Python-urllib/3.7", \n "X-Amzn-Trace-Id": "Root=1-5ec3f607-00f717e823a5c268fe0e0be8"\n }, \n "json": null, \n "origin": "123.139.39.71", \n "url": "http://httpbin.org/post"\n}\n'

2、urllib3模块

通过urllib3模块实现发送网络请求的示例代码：

#导入模块
import urllib3
#创建PoolManager对象，用于处理与线程池的连接以及线程安全的所有细节
http = urllib3.PoolManager()
#对需要爬取的网页发送请求
response = http.request('GET','https://www.baidu.com/')
#打印读取的内容
print(response.data)

结果：

b'<!DOCTYPE html><!--STATUS OK-->\r\n<html>\r\n<head>\r\n\t<meta http-equiv="content-type" content="text/html;charset=utf-8">\r\n\t<meta http-equiv="X-UA-Compatible" content="IE=Edge">\r\n\t<link rel="dns-prefetch" href="//s1.bdstatic.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t1.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t2.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t3.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t10.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t11.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//t12.baidu.com" rel="external nofollow" />\r\n\t<link rel="dns-prefetch" href="//b1.bdstatic.com" rel="external nofollow" />\r\n\t<title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9\x81\x93</title>\r\n\t<link href="https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/home/css/index.css" rel="external nofollow" rel="stylesheet" type="text/css" />\r\n\t<!--[if lte IE 8]><style index="index" >#content{height:480px\\9}#m{top:260px\\9}</style><![endif]-->\r\n\t<!--[if IE 8]><style index="index" >#u1 a.mnav,#u1 a.mnav:visited{font-family:simsun}</style><![endif]-->\r\n\t<script>var hashMatch = document.location.href.match(/#+(.*wd=[^&].+)/);if (hashMatch && hashMatch[0] && hashMatch[1]) {document.location.replace("http://"+location.host+"/s"+hashMatch[1]);}
…………………………（太多省略）

post请求实现获取网页信息的内容：

#导入模块
import urllib3
#创建PoolManager对象，用于处理与线程池的连接以及线程安全的所有细节
http = urllib3.PoolManager()
#对需要爬取的网页发送请求
response = http.request('POST','http://httpbin.org/post',fields={'word':'hello'})
#打印读取的内容
print(response.data)

结果：

b'{\n "args": {}, \n "data": "", \n "files": {}, \n "form": {\n "word": "hello"\n }, \n "headers": {\n "Accept-Encoding": "identity", \n "Content-Length": "128", \n "Content-Type": "multipart/form-data; boundary=06ff68d7a4a22f600244a70bf9382ab2", \n "Host": "httpbin.org", \n "X-Amzn-Trace-Id": "Root=1-5ec3f8c3-9f33c46c1c1b37f6774b84f2"\n }, \n "json": null, \n "origin": "123.139.39.71", \n "url": "http://httpbin.org/post"\n}\n'

3、requests模块

以GET请求方式为例，打印多种请求信息的代码:

#导入模块
import requests
#对需要爬取的网页发送请求
response = requests.get('http://www.baidu.com')
#打印状态码
print('状态码:',response.status_code)
#打印请求url
print('url:',response.url)
#打印头部信息
print('header:',response.headers)
#打印cookie信息
print('cookie:',response.cookies)
#以文本形式打印网页源码
print('text:',response.text)
#以字节流形式打印网页源码
print('content:',response.content)

结果：

状态码: 200
url: http://www.baidu.com/
header: {'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Tue, 19 May 2020 15:28:30 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:27:32 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}
cookie: <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
text: <!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8>
………………（此处省略）
content: b'<!DOCTYPE html>\r\n<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8>
………………（此处省略）

以POST请求方式，发送HTTP网页请求的示例：

#导入模块
import requests
#表单参数
data = {'word':'hello'}
#对需要爬取的网页发送请求
response = requests.post('http://httpbin.org/post',data=data)
#以字节流形式打印网页源码
print(response.content)

结果：

b'{\n "args": {}, \n "data": "", \n "files": {}, \n "form": {\n "word": "hello"\n }, \n "headers": {\n "Accept": "*/*", \n "Accept-Encoding": "gzip, deflate", \n "Content-Length": "10", \n "Content-Type": "application/x-www-form-urlencoded", \n "Host": "httpbin.org", \n "User-Agent": "python-requests/2.23.0", \n "X-Amzn-Trace-Id": "Root=1-5ec3fc97-965139d919e5a08e8135e731"\n }, \n "json": null, \n "origin": "123.139.39.71", \n "url": "http://httpbin.org/post"\n}\n'

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持。

上一篇：Keras设置以及获取权重的实现
下一篇：Python包和模块的分发详细介绍