投稿    登录
  《Python3网络爬虫开发实战》赠书活动正在进行中!详情请戳赠书活动!欢迎参与!非常感谢!

Python爬虫实战二之爬取百度贴吧帖子

Python 崔庆才 241570浏览 114评论

大家好,上次我们实验了爬取了糗事百科的段子,那么这次我们来尝试一下爬取百度贴吧的帖子。与上一篇不同的是,这次我们需要用到文件的相关操作。

前言

亲爱的们,教程比较旧了,百度贴吧页面可能改版,可能代码不好使,八成是正则表达式那儿匹配不到了,请更改一下正则,当然最主要的还是帮助大家理解思路。

2016/12/2

本篇目标

1.对百度贴吧的任意帖子进行抓取

2.指定是否只抓取楼主发帖内容

3.将抓取到的内容分析并保存到文件

1.URL格式的确定

首先,我们先观察一下百度贴吧的任意一个帖子。

比如:http://tieba.baidu.com/p/3138733512?see_lz=1&pn=1,这是一个关于NBA50大的盘点,分析一下这个地址。

所以我们可以把URL分为两部分,一部分为基础部分,一部分为参数部分。

例如,上面的URL我们划分基础部分是 http://tieba.baidu.com/p/3138733512,参数部分是 ?see_lz=1&pn=1

2.页面的抓取

熟悉了URL的格式,那就让我们用urllib2库来试着抓取页面内容吧。上一篇糗事百科我们最后改成了面向对象的编码方式,这次我们直接尝试一下,定义一个类名叫BDTB(百度贴吧),一个初始化方法,一个获取页面的方法。

其中,有些帖子我们想指定给程序是否要只看楼主,所以我们把只看楼主的参数初始化放在类的初始化上,即init方法。另外,获取页面的方法我们需要知道一个参数就是帖子页码,所以这个参数的指定我们放在该方法中。

综上,我们初步构建出基础代码如下:

运行代码,我们可以看到屏幕上打印出了这个帖子第一页楼主发言的所有内容,形式为HTML代码。

20150219162232

3.提取相关信息

1)提取帖子标题

首先,让我们提取帖子的标题。

在浏览器中审查元素,或者按F12,查看页面源代码,我们找到标题所在的代码段,可以发现这个标题的HTML代码是

所以我们想提取<h1>标签中的内容,同时还要指定这个class确定唯一,因为h1标签实在太多啦。

正则表达式如下

所以,我们增加一个获取页面标题的方法

2)提取帖子页数

同样地,帖子总页数我们也可以通过分析页面中的共?页来获取。所以我们的获取总页数的方法如下

3)提取正文内容

审查元素,我们可以看到百度贴吧每一层楼的主要内容都在<div id=”post_content_xxxx”></div>标签里面,所以我们可以写如下的正则表达式

相应地,获取页面所有楼层数据的方法可以写成如下方法

好,我们运行一下结果看一下

20150219235120

真是醉了,还有一大片换行符和图片符,好口怕!既然这样,我们就要对这些文本进行处理,把各种各样复杂的标签给它剔除掉,还原精华内容,把文本处理写成一个方法也可以,不过为了实现更好的代码架构和代码重用,我们可以考虑把标签等的处理写作一个类。

那我们就叫它Tool(工具类吧),里面定义了一个方法,叫replace,是替换各种标签的。在类中定义了几个正则表达式,主要利用了re.sub方法对文本进行匹配后然后替换。具体的思路已经写到注释中,大家可以看一下这个类

在使用时,我们只需要初始化一下这个类,然后调用replace方法即可。

现在整体代码是如下这样子的,现在我的代码是写到这样子的

我们尝试一下,重新再看一下效果,这下经过处理之后应该就没问题了,是不是感觉好酸爽!

20150220000103

4)替换楼层

至于这个问题,我感觉直接提取楼层没什么必要呀,因为只看楼主的话,有些楼层的编号是间隔的,所以我们得到的楼层序号是不连续的,这样我们保存下来也没什么用。

所以可以尝试下面的方法:

1.每打印输出一段楼层,写入一行横线来间隔,或者换行符也好。

2.试着重新编一个楼层,按照顺序,设置一个变量,每打印出一个结果变量加一,打印出这个变量当做楼层。

这里我们尝试一下吧,看看效果怎样

把getContent方法修改如下

运行一下看看效果

20150220000947

嘿嘿,效果还不错吧,感觉真酸爽!接下来我们完善一下,然后写入文件

4.写入文件

最后便是写入文件的过程,过程很简单,就几句话的代码而已,主要是利用了以下两句

file = open(“tb.txt”,”w”)

file.writelines(obj)

这里不再赘述,稍后直接贴上完善之后的代码。

5.完善代码

现在我们对代码进行优化,重构,在一些地方添加必要的打印信息,整理如下

现在程序演示如下

20150220012351

 

完成之后,可以查看一下当前目录下多了一个以该帖子命名的txt文件,内容便是帖子的所有数据。

抓贴吧,就是这么简单和任性!

转载请注明:静觅 » Python爬虫实战二之爬取百度贴吧帖子

喜欢 (522)or分享 (0)

我的个人微信公众号,联系我请直接在公众号留言即可~

扫码或搜索:进击的Coder

进击的Coder

微信公众号 扫一扫关注

想结交更多的朋友吗?

来进击的Coder瞧瞧吧

进击的Coder

QQ群号 99350970 立即加入

进击的Coder灌水太多?

这里是纯粹的技术领地

激进的Coder

QQ群号 627725766 立即加入

您的支持是博主写作最大的动力,如果您喜欢我的文章,感觉我的文章对您有帮助,请狠狠点击下面的

发表我的评论
取消评论
表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
(114)个小伙伴在吐槽
  1. スーパーコピーブランド時計激安通販専門店 2018年人気最高品質時計コピー、国際ブランド腕時計コピー最新入荷! 商品の数量は多い、品質はよい。 海外直営店直接買い付け! 製品はされています高品質と低価格で提供!歓迎新旧顧客お願いします! ★ 2018年注文割引開催中,全部の商品割引10% ★ 在庫情報随時更新! ★ 実物写真、付属品を完備する。 ★ 100%を厳守する。 ★ 送料は無料です(日本全国)! ★ お客さんたちも大好評です ★ 経営方針: 品質を重視、納期も厳守、信用第一! 税関の没収する商品は再度無料にして発送します
  2. Adult Webcam Performer Recommendations - Make Your Very first $5000.The adult Internet cam marketplace is The most profitable chances you will find anyplace on the internet right now. As an web cam performer, you may make an hourly pay out similar to that of organization executive; not simply will it pay out extremely perfectly, however , you'll also gain plenty of time by not having to waste time commuting to work every day.Male or feminine, you can begin building a substantial amount of money working from your ease and comfort of your personal household, with as small as two hrs per day.The amount you is likely to make is dependent upon a number of components which include your Bodily attractiveness, your individuality, as well as your willingness to complete from the non-public place. New Grownup webcam performers can expect to create $2000 to $5000 in their extremely very first month Operating as minimal as two hours working day, although leading carrying out models could make more than $twenty,000 every month.You might think that Operating being an adult webcam performer is not hard operate, but do you truly have what it takes to generally be a online product?Below would be the tips on the trade for building your initially $5000.Webcam Model Tip #one - Have some fun.Never imagine it as operate. Visualize it as flirting. Get on camera to rejoice! Preserve this your Major target and you'll have no issue building loads of cash.Webcam Design Idea #2 - Be Prepared.In no way make your visitors hold out. To stay on camera, keep your props, toys, or any variations for your outfit close by so that you is not going to really need to go "off cam" to go get a thing.Webcam Product Idea #3 - Get Repeat Website visitors.Permitting your customers know when you'll be again on the web (no matter if It is several hours from now or another day) is a fairly easy method of getting having to pay consumers to come back for a lot more! And once you do established a time and date, make sure you be on time. This can help Create belief using your customers, which means extra money in your case.Webcam Model Tip #4 - Use Visitors Identify.In totally free chat, be sure to use your buyers' nickname. Website visitors need to have to learn that you are chatting to them personally. Also, master their serious names. Shelling out shoppers wish to experience an psychological link to you. Mastering and memorizing names can get you a lot more repeat spending shoppers.Webcam Model Suggestion #5 - Look Sexy.Use make-up and have your hair finished. Also be sure you have a variety of sexy lingerie, outfits, see by way of clothes, and many others. and alter outfits from day after day to give shoppers assortment. Dazzling hues for example crimson, orange, blue, environmentally friendly and pink bring in a lot more clients, so you should also be matching the colours of your respective sheets/curtains along with your garments.Dont be shy, have a great time and generate some funds $$$ : More Details if you click HERE !!!
    Maurine Nimmons2019-01-14 13:37 (2天前)回复
  3. Chat Rooms to satisfy Girls – A simple Way to talk with Local Women!There are terrific and undesirable boards to satisfy Women of all ages. The top rooms are rather few. This text is below to information you throughout the suitable path. What You need to do is acquire a membership at the massive-title dating Local community having a population inside of thousands and thousands. You could generally sign up for these products and services gratis and use almost all of their attributes (something little men and women know). Another thing Many of us will not be aware of is internet Internet sites have the most significant chatting rooms to meet Women of all ages on the planet.These internet sites have various customers. At just one periods they have a tendency to possess 10 or tens of A large number of men and girls Internet with their chatting rooms. Plus the rooms are assorted naturally. You have got geographic rooms; kinds depending on fetishes; Other folks which might be for people of your respective specified sexual persuasion; and so forth.What separates these chatting environments from Other people you might have found are with the additional options. You are able to click another person’s title and look at their profile, see their Photograph albums, observe their videos, and likewise deliver them private messages, email messages, and check with them on webcams.The rooms built available from these well-known relationship products and services definitely are The easiest way to satisfy Females in a very chatting surroundings. But you are able to do noticeably a lot more than merely meet Girls. You are able to satisfy Females who really live in your area or share your interests or sexual proclivities. So many people ignore these spots to have a chat as You should produce a profile to use them and sometimes pay a membership fee.Offered these rooms offer, though, it's possible you'll invariably find out essentially the absolute ideal put on the internet for chatting to ladies. Chat Rooms to Meet Women of all ages – A straightforward Way to talk with Regional Women!
    Tasia Seaburg2019-01-13 19:49 (3天前)回复
  4. tadalafil green pack cialis tadalafil comprar sin receta
    cialisps.com2019-01-13 02:18 (3天前)回复
  5. MayaMuhammad buy generic cialis cheap cialis online can you take lipitor and cialis [url=http://buyscialisrx.com/]cialis without prescription[/url]
    cialis prescription2019-01-13 01:41 (3天前)回复
  6. [url=http://cialisle.com]http://cialisle.com[/url]
    generic cialis2019-01-12 10:32 (4天前)回复
  7. precio sildenafil españa http://viagrarow.com sildenafil blueberry 100 [url=http://viagrarow.com]sildenafil generic[/url]
    sildenafil generic2019-01-11 12:11 (5天前)回复
  8. Joe Bonamassa is a famous country singer, so don't miss the possibility to visit Joe Bonamassa tickets
    CliffKal2019-01-07 20:25 回复
  9. effects of mixing alcohol and viagra generic cialis overnight risque achat cialis sur internet
    ZentMoorospoirl2019-01-07 18:50 回复
1 6 7 8