test_queue.py 文件源码

python
阅读 25 收藏 0 点赞 0 评论 0

项目:domain-discovery-crawler 作者: TeamHG-Memex 项目源码 文件源码
def make_queue(redis_server, cls: type, slots=None, skip_cache=True, settings=None,
               hints=None) -> BaseRequestQueue:
    global logging_configured
    if not logging_configured:
        configure_logging(settings=settings)
        logging_configured = True
    crawler = Crawler(Spider, settings=settings)
    if slots is None:
        slots = {}
    spider = Spider.from_crawler(crawler, 'test_dd_spider')
    if hints:
        spider.hint_urls = hints
    return cls(server=redis_server, spider=spider, key=SCHEDULER_QUEUE_KEY,
               slots_mock=slots, skip_cache=skip_cache)
评论列表
文章目录


问题


面经


文章

微信
公众号

扫码关注公众号