through the daily log analysis in statistics, we can see what web page by the spider’s favorite, and can understand that spider crawling behavior in these pages, for example whether the spider to grab a few not included value pages, or repeated grasping some pages, we all know that this will affect the weight of our web site other pages transfer. For example, a small part of the skin site than name column column usually crawl diligently, included health is also good, so Xiaobian now regularly in the name column add >
through the analysis of the web log, we can well determine the health of our website, and can also see the records of some spiders crawl records and user behavior, so that the data is undoubtedly a key upgrade website for us, when can let us see the defects of the website, and correct. However, Xiao Bian today to share with you is a spider crawling behavior, hope to help you master.
, can be very good to see what we are directory by the spider like, spiders often grab included, and give the weight ranking. And we can use these data to find a lot of problems, for we want to highlight the column can through the internal structure to the point, so as to effectively increase the column weight and the degree of spiders to crawl. For those we don’t like spiders crawl page, also can be shielded.
grew up in the title refer to the data we can see the following information:
tip: if you want long-term development of a website, small series suggest that regular websites report on our website, this development will have a very good help.
above three load self Shanghai encyclopedia.
Through the analysis of Web log
1, the average number of crawled pages = total volume / grab access times
from these data we can be very good to see the love of spiders in Shanghai on our site activity, whether the affinity data and our website content capture the depth of effective. When the total access frequency, the spider stay of our website, and web crawling degree is high, these are all can be seen from our website whether by the spider love. And the residence time in our single page spider’s length can also be seen in our article page is affected by the spider love.
(a), the number of spiders visit the residence time and the number of grab.
2, a single page crawl every time / each stay stay = crawl
3, the average residence time = total residence time / access number
(two) Statistics website directory spiders crawl.
(three) web pages to crawl.