It seems inevitable today that everyone’s had at least one interaction with a price comparison site. From cars and apartments to hotels and flights, there’s a price comparison site, also called a comparison shopping engine (CSE), dedicated to finding you the best price.
In this article we’ll discuss what sets CSEs apart from other price aggregators, how they make their money, and where they get their data from. Finally, we’ll get to the heart of what fuels price comparison websites’ competitive edge.
As there are so many businesses in this industry, to help illustrate our points we’ll specifically be using travel price comparison sites to illustrate our examples, although anything discussed here applies generally to the CSE industry as a whole.
What Is a Price Comparison Website?
On paper, price comparison sites (or CSEs as they’re officially known) specialize in presenting you with products or services at the lowest price, based on your needs or parameters.
Price comparison sites aren’t online marketplaces, they’re middlemen. They make money from the companies whose services and products they list advertisements for.
Skyscanner, for example, defines itself as “a comparison website” that generates revenue “through a combination of advertising and commission from our partners”. It does this for flights, hotels, and car rental companies.
A CSE doesn’t make any purchases on your behalf, it only puts you into contact with the company offering you the service, whether it’s an airline, hotel, car rental company, or travel agent.
However, this begs the question: What’s the difference between a price comparison site and an online marketplace?
Price Comparison Sites vs Online Marketplaces
To the consumer, the difference between a price comparison site and an online marketplace as it relates to price aggregation is not a meaningful one. When you’re trying to organize a holiday, the fact that Booking.com and Skyscanner have two different business models — the former an online travel agent (OTA) and the latter a CSE — doesn’t matter to you. As long as you’re able to compare prices between hotels you’re happy.
A CSE’s business model is exclusively price aggregation. By contrast, an online marketplace may offer price comparisons, but that is only part of its business model. The fact that many, if not most, CSEs and online marketplaces are vertically integrated adds to this confusion. The same company may own several CSEs and OTAs that cover every aspect of travel from flights to hotels and rentals across every continent.
Let’s take a side-by-side look at the differences between the two companies we mentioned to illustrate the differences.
| Feature | Booking.com | Skyscanner |
|---|---|---|
| Business Model | OTA | CSE |
| Core Function | Transaction platform | Comparison engine |
| Who Owns the Transaction? | Booking.com | OTA, airline, hotel, etc. |
| Revenue Model | Commissions: hotels, airlines, and hosts pay per completed booking | Referral: partners pay per click/redirect (CPC model) |
| Incentives | Maximize bookings completed on own platform | Maximize number of referrals to partners |
Examples of Price Comparison Sites by Industry
To give you a selection of CSEs by industry, some of which might be familiar to you.
- Travel
- Skyscanner
Google Flights/Google Hotels
Trivago
- Skyscanner
- Retail & ecommerce
- Google Shopping
- Idealo
- PriceRunner
- Finance & insurance
- CompareTheMarket
- NerdWallet
- Policygenius
- Consumer electronics & specialized goods
- PriceGrabber
- Shopzilla
- Kelkoo
Examples of Sites That Aren’t CSEs
To make the distinction clear, here are online marketplaces that are not CSEs but do engage in price aggregation.
- Travel: Booking.com
- Retail & ecommerce: Amazon
- Finance & insurance: LendingTree
- Consumer electronics & specialized goods: Newegg
The key difference is that the transaction is happening on their platform.
As we’ve touched on, price comparison sites make money for every user that finds a partner through its site. Let’s explore that more.
How Do Price Comparison Sites Make Money?
CSEs rely on collecting lots of data on their respective industries to be able to offer as accurate a price comparison to users as possible. They theoretically have a vested interest in maximizing consumer’s trust in the company and charge OTAs and other partners for every click, similar to many other platforms that use CPC (cost-per-click) pricing models.
When you set your trip parameters into Skyscanner, for example, you won’t necessarily find the cheapest flights automatically. Instead, you’ll find the “Best” results listed by default. These are the partners that have won Skyscanner’s CPC bid. This is where the money is.
When you move on to booking a hotel, you’ll find the following categories:
- Recommended
- Top reviews
- Lowest price
- Most stars
- Nearest first
Skyscanner’s “Recommended” section, which is selected by default, sorts “deals available on each property by price and our potential revenue.” However, because their business model relies on user trust, you can of course sort by lowest price.
As much as CSEs have a vested interest in presenting partners, they also have an incentive to present the most complete picture of available offerings. How do CSEs collect all their data though?
How Do Price Comparison Sites Get Their Data?
Both hotels and OTAs are incentivized to buy into price comparison platforms, but to set themselves apart from their competitors, price comparison sites need to go beyond relying solely on partners.
Some airlines, hotels, car rental companies, and more, benefit from additional visibility and lose nothing by offering CSEs their data. For example, structured data markup is an internet standard for embedding metadata in a website’s HTML to make it easier for search engines to understand context. This includes everything from articles and reviews to product descriptions and prices.
Although not consistently applied, it can make it easier for CSEs to automate crawling websites and collecting the relevant data through web scraping. However, price comparison sites can’t rely on markup that makes it easy to scrape. We have finally arrived at the purpose of this article.
Web Scraping: CSEs’ Competitive Edge
If the “completeness” of a price comparison site’s offerings is the determining factor in whether users trust them, then CSEs have to invest heavily in finding every product or service and collecting every listing they have on offer.
Let’s continue using hotels as an example. Skyscanner benefits not only from working with OTAs like Booking.com but also from finding lesser-known hotels and hostels that might offer its users a “unique” experience they may not find anywhere else.
Moreover, because the travel industry is so fiercely competitive, price fluctuations are nearly constant. Whether it’s to outbid a competitor or to ensure 100% capacity, airlines and hotels constantly update their prices.
This means that a CSE, Skyscanner in this case, doesn’t only have to ensure it knows of the existence of every flight or hotel room, but also how much it costs right now. Furthermore, deals can be based on many factors, including but not limited to time of day, geographic location, and current occupancy.
Indeed, we have evidence that companies like Skyscanner engage in large-scale web scraping and screen scraping as the result of an unsuccessful injunction filed by Ryanair against a number of OTAs and CSEs.
Every price aggregator, but especially CSEs, set themselves apart from their competitors with large-scale web scraping operations that search for as many relevant listings as possible, and scrape them as often as possible, specifically price scraping, as listings are often priced dynamically to maximize revenue and occupancy.
Another conclusion we can draw from Ryanair’s injunction is that not every company wants their data to be scraped. As a consequence, many sites will employ anti-bot measures, which further requires CSE’s web scraping to be able to bypass those measures.
Of course, as several lawsuits have shown, some CSEs seek a competitive advantage through unfair business practices. In 2022, the Australian justice system fined Trivago AU$44.7 million for “misleading consumers”, and CompareTheMarket, an insurance CSE, was fined £17.9 million by British authorities for breaching competition law in 2020.
Conclusion
At the end of the day, a CSE’s advantage is driven by the completeness of their information. This is driven by an intense data collection process that serves as its driving force.
All price aggregators, but especially CSEs set themselves apart by having more listings, more often, and more accurate than their competition. Whether they gather that data from partners directly or through a sophisticated web scraping operation — often both — price comparison sites rely on a steady flow of data.
Key Takeaways:
- Price aggregation is a CSE’s entire business model.
- Having more listings improves a CSE’s position, as it negotiates from a position of strength, and makes it more attractive to partners and consumers.
- CSEs have to find a balance between improving user trust and maximizing revenue by promoting partners, which sometimes conflict.
- For price comparison websites, large-scale web scraping provides a competitive edge.
Data and the increasing speed with which it can be acted upon will continue to develop. The implementation and adoption of dynamic pricing will only continue to expand as online marketplaces and individual businesses continue to seek their own competitive advantages.
Furthermore, they’ll be even more incentivized to make sure their websites aren’t easy targets for web scrapers and data collectors. As a result, CSEs have their own vested interest in investing in advanced data collection.
