Crawling l Indexing l Ranking
a.) Crawling: The Google search starts with “Crawling”. Google uses web crawlers to discover information publicly. The crawlers crawl the World Wide Web to discover pages. These pages exist to identify the best web pages to be evaluated for the query. The crawlers travel via are website links. These website links bind together pages in a website across the web. Then secondly, create a pathway for the crawlers to reach the trillions of interconnected website pages that exist. Computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. The most popular web crawler is “GoogleBot”. Googlebot is Google’s web crawling robot which finds and retrieves pages on the web and hands them off to the Google indexer. It’s easy to imagine Googlebot as a little spider scurrying across the strands of cyberspace. But in reality Googlebot doesn’t traverse the web at all. It functions much like your web browser, by sending a request to a web server for a web page, downloading the entire page, then handing it off to Google’s indexer. b.) Indexing: The web is like an ever-growing public library with billions of books and no central filing system. Google essentially gathers the pages during the crawl process and then creates an index, so we know exactly how to look things up. Much like the index in the back of a book, the Google index includes information about words and their locations. When you search, at the most basic level, algorithms look up your search terms in the index to find the appropriate pages. Indexing is process of adding information about a web page to a search engine’s index. The index is the database with which Google store and retrieve data when a user types a query into the search engine. The search process gets much more complex from there when you search for “dogs”. You don’t want a page with the word “dogs” on it hundreds of times. You probably want pictures, videos or a list of breeds. Google’s indexing systems note many different aspects of pages, such as when they were published, whether they contain pictures and videos, and much more. c.) Ranking: Before Google decides which web pages to show from the index and in what order, search engines apply algorithms to help rank those web pages. So the third step is “Ranking” the data. Google ranks pages according to their relevance and importance. As SEO’s this is the area we are most concerned with and the part that allows us to show clients tangible progress. Once a keyword is entered into a search box, google will check for pages within their index that are a closest match. A score will be assigned to these pages based on an algorithm consisting of hundreds of different ranking signals. These pages (or images & videos) will then be displayed to the user in order of score. So in order for your site to rank well in search results pages, it’s important to make sure google can crawl and index your site correctly. Otherwise they will be unable to appropriately rank your website’s content in search results. Google is one of the most dominant forces on the Internet. So superficially, Google search seems so simple, type what you want to know into the search box, and poof! The results display. But this quick search is result of a complex set of processes behind the scene that help us identify the most relevant data. So, next time when you have to find the best chicken biryani recipe, then think about google as well which has its own complex recipe that helps us serve the best and most relevant data in few seconds.