A compliance technology partner came to us recently because their internal adverse media screening<\/a> tool had stopped working. This situation highlighted the challenges of adverse media screening in the compliance industry. The engineering was fine. However, the data supply chain wasn’t. They had been pulling from a third-party search scraping service. When Google launched SearchGuard in early 2025 and filed suit against SerpApi under the DMCA in December, their pipeline went with it.<\/p>\n That conversation triggered an internal evaluation that turned out to be more useful than we expected. There were things about how the broader market sources adverse media data that we hadn’t fully articulated, even to ourselves. This post is the outcome of that work.<\/p>\n There are three meaningfully different architectures in use across adverse media screening platforms today. Most buyers never ask which one their vendor uses. Instead, they evaluate the demo, the match rates, the alert workflow, and miss the question that actually determines whether the product holds up at scale and under regulatory scrutiny.<\/p>\n Some platforms fire a live search query when a name comes in for screening. A commercial search API returns results, those results get presented to the analyst, and that is the product. There is no pre-built database. Instead, the search engine index is the database. It is cheap to build and fast to launch, which is why it exists.<\/p>\n At low volumes it is easy to miss how badly this scales. A mid-size payment processor screening 100,000 names a day is making 100,000 live API calls a day. At commercial search API rates that cost is significant. Unlike almost every other component of a compliance stack it grows in direct proportion to your business. Beyond cost, search engines rank results for relevance, not compliance utility. A name search returns everyone who has appeared near a risk keyword in any article indexed on the web. No entity disambiguation is applied before the result reaches an analyst. Industry research puts false positive rates for keyword-based search at 70 to 90 percent.<\/a> That burden lands on your compliance team.<\/p>\n The legal situation has shifted too. Google built SearchGuard specifically to block automated commercial querying at scale. The December 2025 DMCA suit against SerpApi<\/a> was enforcement, not a warning. Any pipeline that depends on circumventing bot detection, directly or through a vendor doing it on your behalf, is carrying exposure that can surface without notice.<\/p>\n Several of the larger platforms in this space crawl the open web continuously, ingest articles, apply NLP and machine learning to classify and enrich content, and build a proprietary database. Screening queries then hit that database rather than a live search. This solves the query-time performance problem and improves false positive rates. This is because enrichment happens at ingestion rather than when the result lands on the analyst’s screen.<\/p>\n The copyright question is the one this architecture hasn’t fully answered. Scraping publisher content and storing article text commercially, then serving it to paying clients as part of a licensed product, means operating a content database built on material you don’t have rights to. Publishers have become considerably more aggressive about enforcement. The legal environment that produced the SerpApi case has also produced suits against AI training datasets and news aggregators. If your vendor stores full article body text in their database, that is a question worth asking directly. This is because it is a liability that could affect service continuity.<\/p>\n The other issue is provenance. Open web scraping ingests at breadth and the quality of the resulting database depends entirely on the filtering applied afterward. When a regulator asks where a specific finding came from and how it was sourced, “we crawled the web” is a harder answer to defend than a registered, assessed source registry with known URLs and publication dates.<\/p>\n The third approach is to ingest from a defined, curated set of source feeds – public RSS, government publications, licensed aggregators, keyword-configured alert feeds from known publishers – and to store intelligence extracted from articles rather than the articles themselves. Named entities, crime classifications, PEP flags, sentiment, country attribution, a structured summary, a contextual excerpt, and a URL back to the original publisher, are stored instead. The full article text is processed transiently through an enrichment pipeline. What gets written to the database is the signal, not the content.<\/p>\n The legal distinction here is meaningful. The fact that an article exists, when and where it was published, and what risk signals it contains is not copyrightable. The article text is. A signal database and a content database are structurally different from a legal standpoint. When a regulator or auditor asks about provenance, every record traces to a known, registered source with a URL they can verify.<\/p>\n Because enrichment – entity disambiguation, crime taxonomy classification, deduplication across sources – is applied at ingestion rather than display, analysts review structured intelligence records rather than search results. That changes the analyst workload considerably. It also means the cost of running a screening query doesn’t scale with volume the same way a live search model does.<\/p>\n The honest weakness is coverage. No source registry, however large, captures everything. A name submitted today may have adverse media\u00a0the pipeline hasn’t reached. That gap is real. The right response is a conditional, licensed live search that fires only when the internal database returns no match. Furthermore, results are enriched and stored permanently so the same gap doesn’t recur. That is a targeted gap-fill, not a fallback to Architecture One.<\/p>\n The industry conflates two separate problems more often than it should. Data quality – curated sources, enrichment at ingestion, entity disambiguation before a record reaches an analyst – is the foundation. Screening intelligence is what runs on top of it: name matching, fuzzy logic, NLP for entity resolution, LLM-assisted inference for the complex cases where rule-based logic runs out of road. Both matter, and a platform can be strong at one while being weak at the other. Buying on the demo without understanding which layer you’re actually evaluating is how you end up with a tool that performs well in a proof of concept. Unfortunately, it struggles in production.<\/p>\n This post covers the data layer. The next one in this series covers screening and decisioning; how deterministic NLP and LLM inference work together to reduce false positives at query time, and where human-in-the-loop disposition fits. The third covers ongoing monitoring. You can find more of our thinking on adverse media screening in our blog archive<\/a>. If you want to be notified when those are published, follow KYC2020 here on LinkedIn<\/a>.<\/p>\n When a name comes in for screening, does your system query a live search engine? If yes, what is the legal basis for that API access and what is the contingency if it goes away?<\/p>\n Does your platform store full article text? If yes, how is copyright exposure managed?<\/p>\n Where does enrichment happen. At ingestion or at display time? That distinction determines whether your analysts are reading intelligence or sorting through search results.<\/p>\n How do you handle names with no internal database match? What triggers your gap coverage, and what is the per-call cost?<\/p>\n What is your false positive rate, measured how? A benchmark against manual Google search sets a low bar on purpose.<\/p>\n We run a signal database built from 1,372 registered source feeds across 25 geographic markets. Articles are enriched at ingestion through an 8-stage pipeline covering NER, crime classification, PEP detection, sentiment, and country attribution. We store signals and contextual excerpts, not full article text – analysts have a direct link to the original publisher source for the full article. For gap coverage we offer a conditional live search that fires on configurable trigger conditions and permanently enriches the database with what it finds, so the same name doesn’t trigger a repeat call. Screening intelligence runs on top of that data layer and is the subject of the next post.<\/p>\n If you are rebuilding an internal Adverse Media Screening capability or want to understand what your current vendor is actually doing under the hood, we are happy to talk through it. \u00a0Contact us here<\/a>\u00a0or reach us at sales@kyc2020.com<\/a><\/p>\n KYC2020 is an AML-first compliance platform for KYC. ISO27001 and SOC 2 Type II certified. Trusted by 250+ regulated institutions for Sanctions, PEP, and Adverse Media Screening across USA, Canada, & EMEA.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" A compliance partner’s pipeline went dark overnight when Google took out their scraping vendor. It pushed us to write something we’d been meaning to publish – a clear breakdown of how adverse media data is actually sourced, and why the architecture underneath your screening tool matters more than most buyers realize<\/p>\n","protected":false},"author":22,"featured_media":3331,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[90,24],"tags":[95,30,201,183,197,198,34,91,12,199],"class_list":["post-3323","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-adverse-media-screening","category-compliance-trends-topics","tag-adverse-media-screening","tag-aml-compliance","tag-data-architecture","tag-false-positives","tag-financial-crime","tag-google-searchguard","tag-kyc","tag-negative-news-screening","tag-regtech","tag-serpapi"],"acf":[],"yoast_head":"\nArchitecture One: Search at Screening Time<\/h2>\n
Architecture Two: Continuous Web Scraping into a Database<\/h2>\n
Architecture Three: Structured Feed Ingestion into a Signal Database<\/h2>\n
Data First. Screening Second.<\/h2>\n
Questions Worth Asking Your Vendor<\/h2>\n
Where KYC2020 Fits<\/h2>\n
\n