scrapy爬虫实例之爬取师资信息

scrapy是什么

Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架，我们只需要实现少量的代码，就能够快速的抓取。使用了Twisted异步网络框架，可以加快我们的下载速度。

scrapy engine充当交通枢纽的核心作用，调度各个模块之间的联结，最终是达到分布式爬虫的目的。
需要我们更改的代码，就是爬虫模块和pipeline模块。爬虫模块负责爬取并提取数据，pipeline模块负责保存数据结果。在使用pipeline模块之前，需要在settings模块开启pipeline功能。
pipeline翻译成中文是「管线」的意思。起到数据传输的作用。

scrapy基本用法

scrapy是个爬虫框架，框架的意思就是模板，它会预定义一些设置，我们只需要加上自定义的内容就可以实现个性化的爬虫需求。
scrapy爬虫的步骤：

1. 创建爬虫项目
   • scrapy startproject 项目名
2. 创建爬虫
   • scrapy genspider 爬虫名 allowdomain
   • allowdomain设置的是允许爬取的域名范围
3. 写爬虫
   • 修改爬虫文件，爬取并提取数据
   • 修改settings文件，开启pipeline功能
   • 修改pipeline文件，保存数据
4. 运行爬虫
   • scrapy crawl 爬虫名

实例

抓取itcast.cn网站的师资信息。
itcast.py:

# -*- coding: utf-8 -*-
import scrapy


class ItcastSpider(scrapy.Spider):
    name = 'itcast'
    allowed_domains = ['itcast.cn']
    start_urls = ['http://www.itcast.cn/channel/teacher.shtml']

    def parse(self, response):
        #使用xpath处理start_url对应的响应
        # result = response.xpath("//div[@class='tea_con']//h3/text()").extract()
        # print(result)

        #分组
        li_list = response.xpath("//div[@class='tea_con']//li")
        for li in li_list:
            item = {}
            # item["name"] = li.xpath(".//div[@class='li_txt']/h3/text()")
            # item["name"] = li.xpath(".//div[@class='li_txt']/h3/text()").extract()[0]
            item["name"] = li.xpath(".//div[@class='li_txt']/h3/text()").extract_first()
            item["title"] = li.xpath(".//h4/text()").extract_first()
            item["introduction"] = li.xpath(".//div[@class='li_txt']/p/text()").extract_first()
            # print(item)
            yield item

pipelines.py:

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


class MyspiderPipeline(object):
    def process_item(self, item, spider):
        item["hello"] = "world"
        return item


class MyspiderPipeline2(object):
    def process_item(self, item, spider):
        print(item)
        return item

settings.py:

# -*- coding: utf-8 -*-

# Scrapy settings for myspider project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'myspider'

SPIDER_MODULES = ['myspider.spiders']
NEWSPIDER_MODULE = 'myspider.spiders'

LOG_LEVEL = 'WARNING'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'myspider (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'myspider.middlewares.MyspiderSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'myspider.middlewares.MyspiderDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'myspider.pipelines.MyspiderPipeline': 300,
    'myspider.pipelines.MyspiderPipeline2': 301
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

总结

xpath用法：item["name"] = li.xpath(".//div[@class='li_txt']/h3/text()").extract_first()
这里的extract_first()是用来提取我们所需要的数据。等价于item["name"] = li.xpath(".//div[@class='li_txt']/h3/text()").extract()[0]
item["name"] = li.xpath(".//div[@class='li_txt']/h3/text()")这样的用法，返回的结果是selector对象。

.extract()[0]和.extract_first()的区别：
.extract()[0]：如果没有结果，则程序报错。

.extract_first()：如果没有结果，则返回None。

pipeline设置的权重值，指的是距离中心引擎的管道长度，距离引擎越近的pipeline，越先执行。