Discover the Automated Web Crawlers using Web Server Log Files Clustering to Prevent DDOS Attacks

Web mining, also known as Weblog Crawling, is the process of extracting striking patterns and designs from searches on a Web-accessible list. Web mining is in fact the application of data mining techniques to explore patterns of the web. Exploring the use of the Web is a way to find users who are looking for specific goals on the Internet. Some users may be looking for text data while others may want to receive audio data from the Internet. One of the problems is Internet Security Webpages. DDOS1 attacks are a type of attack on a Web server that sends a large amount of requests at one time to a Web server. The problem is most often due to the recent use of the automatic web crawler method for DDOS attack. Web crawlers are pages designed by robots and sent to the desired destination. The server also has no way of detecting these pages and pages posted by the user.The solution to detect these attacks is to check and analyze the log file of the server. The first step of any web mining project is known as preprocessing. This process is performed on the log file that is stored by the web server. One way to diagnose and prevent the log file server from working with data mining techniques.Clustering analysis is a widely used data mining algorithm, which is the process of dividing a series of data into a number of clusters, each of which has a high similarity to other data in the same cluster, but other data Different in other clusters. In this thesis, with the clustering of log file data, we will detect and discover auto web crawlers. The proposed approach is that the behavior of the users and any behavior contrary to the suspicious behavior are identified and efforts made to penetrate into the system are clustered and announced to the server. The server also adds this method to its intrusion detection patterns, detects the behavior of web crawlers, and blocks requests for this template.