投稿    登录
欢迎来访~

小白进阶之Scrapy第三篇(基于Scrapy-Redis的分布式以及cookies池)

Python 哎哟卧槽 168854浏览 5评论

扫码或搜索:进击的Coder

发送

即可立即永久解锁本站全部文章

现在开始大业了!重写cookie中间件;估摸着吧!聪明的小伙儿看了上面重写User-Agent的方法,十之八九也知道怎么重写Cookie中间件了。

好啦,现在继续写middlewares.py啦!

第一行:不说

第二行第三行得说一下 这玩意儿叫重载(我想了大半天都没想起来叫啥,还是问了大才。尴尬)有啥用呢:

也不扯啥子高深问题了,小伙伴儿可能发现,当你继承父类之后;子类是不能用 def init()方法的,不过重载父类之后就能用啦!

第四行:settings[‘REDIS_URL’]是个什么鬼?这是访问scrapy的settings。怎么访问的?下面说

第五行:往redis中添加cookie。第二个参数就是spidername的获取方法(其实就是字典啦!)

这个貌似不好理解,作用看下面:

D9DF3655-F28A-482C-8B02-C53B152958A0

这样是不是一下就知道了??

至于访问settings的方法官方文档给出了详细的方法:

http://scrapy-chs.readthedocs.io/zh_CN/latest/topics/settings.html#how-to-access-settings

QQ20170207-233701@2x

下面就是完整的middlewares.py文件:

存储我也不写啦!就是这么简单一个分布式的scrapy就这么完成啦!!!

我试了下 三台机器 两个小时 就把整个站点全部爬完了。

弄好你的存储 放在不同的机器上就可以跑啦!

完整的代码在GitHub上:

GitHub:https://github.com/thsheep/haoduofuli

Y(^o^)Y完工

下篇博文来对付爬虫的大敌:Ajax

以后的教程用微博做靶子,那些数据比较有用,可以玩玩分析什么的。

 

 

 

 

转载请注明:静觅 » 小白进阶之Scrapy第三篇(基于Scrapy-Redis的分布式以及cookies池)

更多文章、联系博主、技术交流、商务合作

扫码或搜索:进击的Coder

进击的Coder

微信公众号 扫一扫关注

喜欢 (75)or分享 (0)

您的支持是博主写作最大的动力,如果您喜欢我的文章,感觉我的文章对您有帮助,请狠狠点击下面的

发表我的评论
取消评论

表情

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
(5)个小伙伴在吐槽
  1. Thaks for this useful information. Your blog become so helpful to me. You can also Check here: http://anxietymedz.com/buy-xanax-online-overnight-for-treating-anxiety-disorders/
  2. Irrespective of the reason for which you have been prescribed this medication, you should take it only when some symptoms of anxiety or panic attacks surface. Then, this drug reaches your brain and produces effects very fast. Order xanax bars online for fast relief from panic disorder. Order Now: http://anxietymedz.com/how-can-i-order-xanax-online/
  3. In particular, this drug produces relaxation and calmness in order to relieve the symptoms of panic disorder, generalized anxiety disorder and also agoraphobia. It is a fast-acting medication frequently producing the desired effects in a short span of time. Thereafter, it starts to relieve the tangible signs of panic disorder (with or without agoraphobia) within days or hours of taking the first dose. Buy xanax pills online to save you the trouble of going outdoors.
  4. Xanax is a trade name of a medication called alprazolam that is prescribed in the treatment of a range of anxiety and panic disorders. This medication works as a chemical enhancer in the brain to decrease feelings of panic and stress and also reduce excessive brain activity. This drug is very effective but can also have fatal consequences if taken in higher dosage than prescribed. In order to enjoy the therapeutic benefits without going outside, buy xanax online.
  5. http://cheapgogenvia.com/ buy cialis without prescription
    jjesouro2019-03-23 23:36 回复