CEF内核和高级爬虫知识

(转)关于MFC中如何使用CEF内核（CEF初解析）
Python GUI: cefpython3的简单分析和应用
cefpython3：一款强大的Python库

在这里插入图片描述

开始大多数抓取尝试可以从几乎一行代码开始：

fun main() = PulsarContexts.createSession().scrapeOutPages("https://www.amazon.com/", "-outLink a[href~=/dp/]", listOf("#title", "#acrCustomerReviewText"))

上面的代码从一组产品页面中抓取由 css 选择器 #title 和 #acrCustomerReviewText 指定的字段。示例代码可以在这里找到：kotlin，java，国内镜像：kotlin，java。大多数生产环境数据采集项目可以从以下代码片段开始：

fun main() {val context = PulsarContexts.create()val parseHandler = { _: WebPage, document: Document ->// use the document// ...// and then extract further hyperlinkscontext.submitAll(document.selectHyperlinks("a[href~=/dp/]"))}val urls = LinkExtractors.fromResource("seeds10.txt").map { ParsableHyperlink("$it -refresh", parseHandler) }context.submitAll(urls).await()
}

最复杂的数据采集项目可以使用 RPA 模式：

最复杂的数据采集项目往往需要和网页进行复杂交互，为此我们提供了简洁强大的 API。以下是一个典型的 RPA 代码片段，它是从顶级电子商务网站收集数据所必需的：

val options = session.options(args)
val event = options.event.browseEvent
event.onBrowserLaunched.addLast { page, driver ->// warp up the browser to avoid being blocked by the website,// or choose the global settings, such as your location.warnUpBrowser(page, driver)
}
event.onWillFetch.addLast { page, driver ->// have to visit a referrer page before we can visit the desired pagewaitForReferrer(page, driver)// websites may prevent us from opening too many pages at a time, so we should open links one by one.waitForPreviousPage(page, driver)
}
event.onWillCheckDocumentState.addLast { page, driver ->// wait for a special fields to appear on the pagedriver.waitForSelector("body h1[itemprop=name]")// close the mask layer, it might be promotions, ads, or something else.driver.click(".mask-layer-close-button")
}
// visit the URL and trigger events
session.load(url, options)