manager.py 文件源码

python
阅读 29 收藏 0 点赞 0 评论 0

项目:dcard-spider 作者: leVirve 项目源码 文件源码
def download(self):
        logger.info('[Downloader] takes hand')

        self.mkdir(self.resources_folder)

        tasks = [
            (self.get_filepath(meta, url), url)
            for meta, urls in self.resource_bundles
            for url in urls
        ]

        with contextlib.closing(Pool(8)) as pool:
            results = pool.map(self.downloading, tasks)

        status = [ok for ok, _ in results]
        fails = [src for ok, src in results if not ok]

        logger.info('[Downloader] download %d items (Total: %d)!',
                    sum(status), len(status))

        return sum(status), fails
评论列表
文章目录


问题


面经


文章

微信
公众号

扫码关注公众号