HTTP is the foundation of data communication for the World Wide Web, where hypertext documents include hyperlinks to other resources that the user can easily access. The Hypertext Transfer Protocol (HTTP) is an application protocol for the distribution of hypermedia. Hypermedia contrasts with multimedia, in that it is interactive and frequently contains hyperlinks to other hypermedia information systems.
Google utilizes this hypertext protocol in part of it's process to index and record the content existing on the World Wide Web. Google's Googlebot crawler, for example, scans the robots.txt file when indexing websites to check for special instructions on what sections it should ignore.
Google's exclusion protocol is quickly becoming the Internet standard. In July 2019 Google proposed making their Robots Exclusion Protocol (REP) the official standard by releasing their parsing application to the public and have now made their REP open source. Any URI based transfer protocol can use Google REP it's not limited to HTTP anymore and can be used for Constrained Application Protocol (CoAP) and FTP as well.
Frequently, these fail to give direct crawling instructions, creating a
'hole' in the Application-Layer Protocol Negotiation (ALPN) which can be
exploited. Part of the ALPN process includes Transport Layer Security (TLS)
(formally Secure Sockets Layer (SSL)), which are designed to provide
communication security. The protocol is subject to
abatement, delaying or excluding the target during the process.
A service of CleanSearch LLC https://cleansearch.com
CleanSearch® Customer Support
2385 NW Executive Center Dr. Suite 100
Boca Raton, FL 33431