Is Your Web Safe Against Web Crawling and Scraping?

Web crawling refers to the process of systematically browsing the web by using automated bots(so-called ‘crawlers’) and collecting information from the web. Since manually extracting vast amounts of online data is practically impossible, crawlers are used to systematically navigate web pages and gather information automatically. Unlike web scraping, which focuses on extracting specific data from a webpage, web crawling is about building an entire system for discovering and collecting information across multiple web pages. Crawlers are commonly used by search engines, such as Google, to index content and help users find relevant information efficiently.

Crawlers visit and navigate through millions of web pages, continuously accessing links and loading pages in an automated cycle. During this process, the crawler bot sends HTTP requests to retrieve the HTML content of a page, extracts the necessary data, and stores it in a database for further processing. This mechanism is primarily used by search engines like Google and Naver, which index web pages to provide relevant search results. When a user searches for a keyword, the search engine’s crawling algorithm enables it to display a list of web pages along with their content, making information more accessible.

* Common Misconception About Web Crawling : We often come across phrases like ‘Stock Market Crawling’, ‘Linked IN Crawling’, or ‘E-commerce Price Crawling’, which suggest methods for extracting data from specific services. However, most of these discussions actually describe web scraping techniques rather than true web crawling. In reality, many of these guides rely on CSS Selectors to extract data from HTML pages, which is a fundamental method of web scraping rather than crawling. These techniques typically involve making one-time HTTP requests to retrieve specific information, lacking the systematic, large-scale data collection that defines true web crawling.

Web scraping refers to the process of extracting specific information from a web page and storing it for further use. A scraper sends an HTTP request to load a webpage, analyzes its HTML structure, and extracts targeted data, which is then stored in a database. Unlike web crawling, which systematically explores and collects data on a large scale, web scraping focuses on gathering specific pieces of information from a narrower dataset. For example, scraping can be used to extract specific news articles from a news site or automatically collect stock prices and real estate listings from financial platforms.

The key difference between web scraping and web crawling lies in their data collection approach. Web scraping and crawling both use automated bots to efficiently collect web data, and the two terms are often used interchangeably. However, they have clear differences in purpose and method.

Web crawling involves following links across websites, analyzing structures and indexing data to build search engines or databases. Its primary focus is on systematically navigating web pages to gather large-scale information. In contrast,web scraping is designed to extract specific data from targeted web pages for analysis and utilization. It is commonly used for collecting product details, pricing information or other structured content. In summary, web crawling is a broad exploration process for large-scale data collection, whereas web scraping is a focused process for extracting precise information.

Both web scraping and crawling operate on a simple principle. Since web pages deliver content through HTML documents, scraping and crawling begin by sending an HTTP request to a server or browser to retrieve an HTML page. Once the HTML document is received, the process involves analyzing the DOM(Document Object Model) to extract relevant data. If the goal is to pull specific pieces of information, it is classified as scraping. On the other hand, if the focus is on structuring and indexing large amounts of data, the process is categorized as crawling.

Static web pages maintain a pre-rendered HTML structure, meaning their content is fully generated before being sent from the server to the client. Because of this, accessing HTML content is nothing complicated—simply by visiting the URL, the entire page structure could be retrieved. Static scraping/crawling involves loading an HTML page via a URL and extracting relevant data using libraries such as Requests and BeautifulSoup. This process allows the HTML document to be parsed and structured, making it possible to extract text, images, media, and other content efficiently.

Dynamic web pages are characterized by content that is generated in real-time on the client side. These pages cannot be fully accessed by simply visiting a URL, as the data is dynamically rendered in live-time, making it harder to collect the information directly. Dynamic web pages are often used for sites where data changes continuously, as they help reduce server load.

Web scraping and crawling are powerful tools for automatically collecting data, but it is critical to consider ethical and legal issues before implementing them.

Firstly, the purpose of data collection must be legitimate and legal. Data should never be used in malicious or unethical ways. When using collected data for commercial purposes, it is essential to ensure that copyright laws are not violated. Additionally, compliance with the Robots.txt(Robots Exclusion Standard) is a critical requirement. The robots.txt file is a protocol set by website owners to indicate the level of access bots are allowed to have, and it specifies whether crawling is permitted. Therefore, it is important to check a webpage's robots.txt file and comply with the specified rules before proceeding with scraping or crawling.

Despite these ethical and legal considerations, there is no clear legal standard that defines whether scraping and crawling publicly available data is legal or illegal.

Let’s examine the legality of web scraping and crawling through an actual legal dispute.

A recent lawsuit between Company A and the real estate platform Company B highlights the legal complexities of web crawling. Company B crawled property listings posted by Company A and posted them on its own platform, using Company A's data for commercial purposes. In response, Company A filed a lawsuit, claiming that Company B's actions infringed upon its database rights. The domestic court ruled in favor of Company A, determining that Company B had violated Company A's rights as the creator of the database. This case illustrates how web scraping and crawling can potentially infringe upon intellectual property rights, particularly database protection and copyright laws.^[1]

On the other hand, some legal precedents have recognized the legality of crawling publicly available information. For instance, in the case between LinkedIn and hiQ in the United States, the act of collecting publicly available profile information was deemed legal, and this case illustrates how data collection from publicly available sources can be allowed in some jurisdictions. This demonstrates that the legality or illegality of data collection can vary depending on the jurisdiction, the specific case, and the interpretation of the law by the judges or legal professionals involved.^[2]

As a result, service providers are increasingly focusing on the implementation of security measures against scraping and crawling, rather than relying on legal post-action steps, as the legality of scraping and crawling can be subject to interpretation of local laws and specific circumstances.

While the security measures to prevent web scraping and web crawling utilize various approaches to protect data, each method has its own bypass techniques and vulnerabilities. So, what fundamental measures should be prioritized to prevent web scraping and crawling?

Most existing prevention methods focus on data access control within the web browser, and since bypassing these controls is not overly difficult, it becomes impossible to prevent data extraction once a hacker gains access to the DOM (Document Object Model) of the web browser. Traditional web services have treated web browsers as safe zones, which has led to a focus on securing the end-to-end (E2E) process, from the web server to the browser. However, this approach overlooks the potential for data extraction from within the web browser's internal DOM.

To effectively prevent scraping and crawling, a Zero-Trust principle is required, where every request and interaction with the web browser is continuously verified, regardless of the source or internal environment. This would reinforce security within the browser itself, making it significantly impossible for attackers to access and extract data from the DOM.

Therefore, by adopting a Zero Trust-based security approach, even within the DOM area of the web browser, users should not be trusted to access the data. If the user fails to pass the authentication, the data should remain obfuscated or encrypted. Only after successfully authenticating through a secure process can the data be accessed, and this access should be limited to specific URLs and time periods. Through this method, the data of web-based services can be protected from web scraping and crawling attacks, ensuring a higher level of security.

To fundamentally prevent web scraping and crawling, protecting the DOM is essential. Due to the nature of web browsers, data within the DOM exists in plaintext, making it easily accessible to scrapers and crawlers who can indiscriminately collect this data. Therefore, data in the DOM should be encrypted and obfuscated to ensure ongoing data security. Additionally, obfuscated data should be unreadable if extracted without proper authorization.

This method applies to both static scraping, where data is directly extracted from HTML, and dynamic crawling, where the DOM structure generated by JavaScript is analyzed. By obfuscating or encrypting the DOM data, both static and dynamic scraping and crawling can be effectively prevented.

Blocking developer tools focuses on preventing scraping and crawling through API and network request information and DOM structure analysis. Traditional browser access controls are often ineffective, as crawlers can easily bypass them by manipulating header information or user-agent strings, which fails to prevent static or dynamic crawling. Therefore, a more robust approach is needed, one that blocks access to developer tools entirely, thus preventing network request analysis and HTML structure analysis.

Additionally, to prevent the use of developer tools on unspecified browsers and operating systems, it's crucial to block access to developer tools across all browsers and OS environments, ensuring that scraping and crawling attempts are thwarted from any platform.