投稿    登录
  《Python3网络爬虫开发实战》赠书活动正在进行中!详情请戳赠书活动!欢迎参与!非常感谢!

Python爬虫入门三之Urllib库的基本使用

Python 崔庆才 358181浏览 3037评论

那么接下来,小伙伴们就一起和我真正迈向我们的爬虫之路吧。

1.分分钟扒一个网页下来

怎样扒网页呢?其实就是根据URL来获取它的网页信息,虽然我们在浏览器中看到的是一幅幅优美的画面,但是其实是由浏览器解释才呈现出来的,实质它是一段HTML代码,加 JS、CSS,如果把网页比作一个人,那么HTML便是他的骨架,JS便是他的肌肉,CSS便是它的衣服。所以最重要的部分是存在于HTML中的,下面我们就写个例子来扒一个网页下来。

是的你没看错,真正的程序就两行,把它保存成 demo.py,进入该文件的目录,执行如下命令查看运行结果,感受一下。

2015-02-13 00:09:09 的屏幕截图

看,这个网页的源码已经被我们扒下来了,是不是很酸爽?

2.分析扒网页的方法

那么我们来分析这两行代码,第一行

首先我们调用的是urllib2库里面的urlopen方法,传入一个URL,这个网址是百度首页,协议是HTTP协议,当然你也可以把HTTP换做FTP,FILE,HTTPS 等等,只是代表了一种访问控制协议,urlopen一般接受三个参数,它的参数如下:

第一个参数url即为URL,第二个参数data是访问URL时要传送的数据,第三个timeout是设置超时时间。

第二三个参数是可以不传送的,data默认为空None,timeout默认为 socket._GLOBAL_DEFAULT_TIMEOUT

第一个参数URL是必须要传送的,在这个例子里面我们传送了百度的URL,执行urlopen方法之后,返回一个response对象,返回信息便保存在这里面。

response对象有一个read方法,可以返回获取到的网页内容。

如果不加read直接打印会是什么?答案如下:

直接打印出了该对象的描述,所以记得一定要加read方法,否则它不出来内容可就不怪我咯!

3.构造Request

其实上面的urlopen参数可以传入一个request请求,它其实就是一个Request类的实例,构造时需要传入Url,Data等等的内容。比如上面的两行代码,我们可以这么改写

运行结果是完全一样的,只不过中间多了一个request对象,推荐大家这么写,因为在构建请求时还需要加入好多内容,通过构建一个request,服务器响应请求得到应答,这样显得逻辑上清晰明确。

4.POST和GET数据传送

上面的程序演示了最基本的网页抓取,不过,现在大多数网站都是动态网页,需要你动态地传递参数给它,它做出对应的响应。所以,在访问时,我们需要传递数据给它。最常见的情况是什么?对了,就是登录注册的时候呀。

把数据用户名和密码传送到一个URL,然后你得到服务器处理之后的响应,这个该怎么办?下面让我来为小伙伴们揭晓吧!

数据传送分为POST和GET两种方式,两种方式有什么区别呢?

最重要的区别是GET方式是直接以链接形式访问,链接中包含了所有的参数,当然如果包含了密码的话是一种不安全的选择,不过你可以直观地看到自己提交了什么内容。POST则不会在网址上显示所有的参数,不过如果你想直接查看提交了什么就不太方便了,大家可以酌情选择。

POST方式:

上面我们说了data参数是干嘛的?对了,它就是用在这里的,我们传送的数据就是这个参数data,下面演示一下POST方式。

我们引入了urllib库,现在我们模拟登陆CSDN,当然上述代码可能登陆不进去,因为CSDN还有个流水号的字段,没有设置全,比较复杂在这里就不写上去了,在此只是说明登录的原理。一般的登录网站一般是这种写法。

我们需要定义一个字典,名字为values,参数我设置了username和password,下面利用urllib的urlencode方法将字典编码,命名为data,构建request时传入两个参数,url和data,运行程序,返回的便是POST后呈现的页面内容。

注意上面字典的定义方式还有一种,下面的写法是等价的

以上方法便实现了POST方式的传送

GET方式:

至于GET方式我们可以直接把参数写到网址上面,直接构建一个带参数的URL出来即可。

你可以print geturl,打印输出一下url,发现其实就是原来的url加?然后加编码后的参数

和我们平常GET访问方式一模一样,这样就实现了数据的GET方式传送。

本节讲解了一些基本使用,可以抓取到一些基本的网页信息,小伙伴们加油!

转载请注明:静觅 » Python爬虫入门三之Urllib库的基本使用

喜欢 (2172)or分享 (0)

想学更多爬虫知识?《Python3网络爬虫开发实战》这本书也许更适合你~

了解详情or立即购买

我的个人微信公众号

扫码或搜索:进击的Coder

进击的Coder

微信公众号 扫一扫关注

想结交更多的朋友吗?

来进击的Coder瞧瞧吧

进击的Coder

QQ群号 99350970 立即加入

进击的Coder灌水太多?

这里是纯粹的技术领地

激进的Coder

QQ群号 627725766 立即加入

您的支持是博主写作最大的动力,如果您喜欢我的文章,感觉我的文章对您有帮助,请狠狠点击下面的

发表我的评论
取消评论
表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
(3037)个小伙伴在吐槽
  1. Although sites we backlink to below are considerably not related to ours, we feel they're in fact worth a go by, so have a look.
    Google2018-09-23 07:50 (1天前)回复
  2. I'm just commenting to let you be aware of of the beneficial discovery my cousin's princess gained going through your webblog. She even learned too many details, most notably how it is like to have an ideal teaching spirit to make men and women with ease learn chosen complicated topics. You actually surpassed readers' desires. Many thanks for providing such powerful, dependable, educational and also easy tips on this topic to Julie.
    nike air max2018-09-21 09:24 (3天前)回复
  3. I and also my buddies were found to be studying the excellent tips and tricks on the website and so then I got an awful suspicion I had not expressed respect to the blog owner for those strategies. The women happened to be so excited to learn all of them and have in effect sincerely been making the most of those things. Thanks for actually being so considerate and for choosing variety of good subject matter millions of individuals are really needing to discover. Our sincere apologies for not saying thanks to earlier.
    nike foamposite2018-09-20 19:40 (4天前)回复
  4. I want to express my thanks to you just for bailing me out of such a crisis. After exploring through the the web and finding suggestions which were not powerful, I thought my life was well over. Living without the answers to the issues you have fixed by way of your short article is a serious case, as well as ones that would have in a wrong way affected my entire career if I had not discovered your web blog. Your own mastery and kindness in touching every part was very useful. I am not sure what I would've done if I hadn't come upon such a point like this. I'm able to at this moment look forward to my future. Thanks a lot very much for this skilled and effective help. I won't hesitate to suggest your blog post to any person who ought to have guidance about this matter.
    air max 902018-09-20 05:18 (4天前)回复
  5. I would like to express my thanks to you for rescuing me from this type of setting. Because of surfing around through the internet and coming across recommendations that were not beneficial, I assumed my entire life was well over. Existing minus the answers to the problems you've sorted out all through your main post is a critical case, and those which may have in a wrong way damaged my entire career if I hadn't encountered your blog post. Your own personal understanding and kindness in maneuvering all areas was valuable. I'm not sure what I would have done if I had not discovered such a solution like this. I can also at this moment relish my future. Thank you very much for this skilled and sensible help. I will not think twice to endorse your blog post to any person who needs and wants assistance on this matter.
    nike foamposite2018-09-19 14:45 (5天前)回复
  6. My wife and i have been so ecstatic Edward could complete his homework while using the precious recommendations he discovered while using the web site. It is now and again perplexing to simply be giving freely methods that many many others may have been trying to sell. We grasp we've got the blog owner to be grateful to for this. These illustrations you've made, the straightforward blog navigation, the friendships your site give support to create - it's got most terrific, and it's letting our son in addition to our family recognize that this matter is satisfying, which is certainly wonderfully essential. Thank you for the whole thing!
    jordan 122018-09-19 02:42 (5天前)回复
  7. pay2daymyloans.com
    Boignee2018-09-18 09:56 (6天前)回复
  8. pay2daymyloans.com
    Arract2018-09-18 07:57 (6天前)回复
  9. pay2daymyloans.com
    Flefenum2018-09-18 05:31 (6天前)回复
1 199 200 201