S&P Global Market Intelligence Saves Time and Money While Scaling Machine Learning

S&P Global provides essential intelligence. The company enables governments, businesses and individuals with the right data, expertise and connected technology so that they can make decisions with conviction. From helping its customers assess new investments to guiding them through sustainability and energy transition across supply chains, S&P Global unlocks new opportunities, solves challenges and accelerates progress for the world. The organization is widely sought after by many of the world's leading organizations to provide credit ratings, benchmarks, analytics and workflow solutions in the global capital, commodity and automotive markets. With every one of its offerings, S&P Global helps the world's leading organizations plan for tomorrow, today.

A division of S&P Global, S&P Global Market Intelligence integrates financial and industry data, analytics, research and news to help corporations identify risk and reward opportunities. RiskGauge™, an S&P Global Market Intelligence product, allows customers to assess and monitor credit risk of their counterparty exposures. These insights are especially vital for clients that have exposure to small and midsize enterprises, which — unlike large public companies — have less financial information readily available to fully understand financial and business risks. 

To efficiently store, mine and curate massive amounts of data about these millions of private companies, S&P Global Market Intelligence uses Snowflake. Building a scalable ML pipeline with Snowpark helps S&P Global Market Intelligence deliver timely, reliable risk reports and expand coverage to include even more small to midsize enterprises.

To build its risk reports and analysis, S&P Global Market Intelligence uses advanced ML models to source terabytes of data from millions of enterprises’ websites. Initially, S&P Global Market Intelligence stored raw web crawler data in object storage and used multiple data science technologies for data cleaning and model hosting. However, S&P Global Market Intelligence quickly abandoned this approach due to concerns about data movement, runtime performance, infrastructure costs and complexity. 

“Our technology workhorses have to be really performant,” says Ganesh Nagarathnam, Head of AI Engineering & Architecture for Data Science in S&P Global Enterprise Data Organization. “We do not want to move data into one place for compute then have results in a separate space. We need one unified engine. Data ingestion, data mining and data curation happen in a single ecosystem.” 

For scalability, S&P Global’s Market Intelligence team turned to Snowflake. “I wanted to bring the data and compute together to enable mining within hours — not days — due to the sheer size of the information being collected,” says Moody Hadi, Head of Risk Solutions New Product Development at S&P Global Market Intelligence. “We selected Snowflake and Snowpark for the scale to massively process big data and produce insights synchronized with the way companies generally update their public information.”

With Snowflake, S&P Global Market Intelligence benefits from a fully managed service, which has allowed the team to scale resources efficiently without manual configurations or downtime while also enhancing both performance and availability for data processing. "Snowflake provides us with a unified platform that allows us to easily scale up and down without changing the code, providing the speed we need for AI-related projects while keeping our compute costs within budget." Hadi says.

S&P Global Market Intelligence now loads both structured and unstructured web-crawled data into Snowflake and applies business attributes and firmographic mining models built with Snowpark. These AI custom models then curate the business data, ultimately feeding S&P Global Market Intelligence’s credit models within their RiskGauge™ reports.

Handling terabytes of data weekly, S&P Global Market Intelligence needed a platform that could process large data sets efficiently. With Snowpark, the team collocates data and compute more easily for faster parallel processing across multiple algorithms and data mining models, including custom models, natural language toolkits and named entity recognition. Running advanced ML workloads in parallel helps accelerate extraction of organizational data, such as business name, snippets of business activities, company location details, industry classification, announcements and news sentiment.

Snowpark Container Services helps S&P Global optimize data costs by selectively using graphics processing units (GPUs) for compute-intensive use cases. “Snowpark and Snowflake effortlessly handles all our big data processing, which includes 15 heavy ML models all competing with each other,” Hadi says.

While using other data science and AI platforms often added significant overhead costs and processing delays, switching to Snowflake has simplified the team’s entire workflow while consolidating storage and compute in a single unified environment. The team enjoys streamlined operations with automated platform management and seamless upgrades — and no longer has to deal with complex data transfers. These improvements reduce downtime, manual configuration and cumbersome maintenance while allowing the team to enhance performance, decrease complexity and gain more control over their infrastructure. 

S&P Global Market Intelligence’s data mining engines have performed consistently well with Snowflake. For example, a pattern-based algorithm that extracts business locations processes several million webpages in just minutes, while a natural language processing model extracts business names and attributes from millions or URLs in minutes. 

With Snowflake, S&P Global Market Intelligence keeps pace with evolving market data while maintaining proper historical context. According to Nagarathnam, “A huge benefit of Snowflake is having the entire history of audits — including data that’s been changed or deleted — so we can compare the probability of default scores to previous calculations.”

S&P Global Market Intelligence uses Snowflake to store, mine and curate data from at least 200 million website URLs and 5 million small to midsize enterprises. Doubling coverage by year’s end will provide corporate clients with even richer credit insights. “Since Snowflake is so scalable, we can use the same pipeline we built to gather data from another 5 million companies,” Hadi says. “This will help us achieve our goal to cover 100% of U.S. small businesses for credit analysis. Combining our AI capabilities on entities that have different levels of data veracity allows us to triangulate information on them in an efficient manner.”

Mining additional business attributes — web traffic and corporate change data are next — and combining them with other data sets will help S&P Global Market Intelligence deliver even more value to its customers. “We augment primary information extracted from websites, so customers know they’re getting the best quality data from us. That’s the key differentiator,” Nagarathnam says.