In 1994, a Stanford PhD student named Sergey Brin wrote a paper describing a spider program called "Backrub" that crawled the web and analyzed link relationships. It was not yet called Google. It was not yet a company. It was a research project trying to solve a single problem: how do you find the most useful document in a universe of documents that no single person has read? That question - how do machines discover, evaluate, and file away the entire web - is still the foundation of everything you will do in SEO.
Before you optimize a single word on your site, you need to understand the three-stage process that determines whether search engines know your pages exist at all.
Stage One: Crawling
Search engines deploy software programs called crawlers - also called spiders or bots - that move through the web continuously, following links from one page to the next. Think of them less like librarians cataloguing books and more like postal workers walking a neighborhood they have never mapped before. They start with streets they already know, notice new addresses along the way, and gradually build a record of what exists where.
The implication for your site is direct: if a page has no links pointing to it from anywhere else, the crawler may never find it. A page that exists only as a URL you know but have never linked to from another page is invisible to the entire system. This is called an orphan page, and it is more common than most site owners realize.
Every search engine also assigns what is called a crawl budget to each website - a rough limit on how many pages it will visit in a given period. Large, fast-loading sites with strong authority get larger budgets. New or slow sites get smaller ones. If your site has hundreds of pages but most of them load slowly or have very thin content, crawlers may spend their entire budget on your worst pages and never reach your best ones.
You influence crawling in two practical ways. First, through your internal linking structure - the links you create between your own pages, which give crawlers a path to follow. Second, through a file called robots.txt, which lives at the root of your domain and tells crawlers which sections they are and are not allowed to visit. A misconfigured robots.txt that accidentally blocks your most important pages is one of the most common and consequential technical SEO mistakes beginners make.
Stage Two: Indexing
Being crawled and being indexed are not the same thing. A crawler can visit your page, note its existence, and still decide not to add it to the search engine's database. The index is a structured record of every page the search engine considers worth storing. It is built from a much smaller subset of everything that gets crawled.
During indexing, the search engine processes your page's content - parsing your HTML, reading your text, attempting to understand what topic the page covers, and filing it alongside other pages that cover similar territory. Modern indexing also involves rendering, which means executing the JavaScript and CSS on your page to see how it actually looks when displayed, the same way a human browser would.
One indexing concept worth understanding early is canonicalization. If your content appears at multiple URLs - a print version, a mobile version, a paginated version - the search engine has to decide which one is the "real" version to store. The canonical URL is that master version. When the search engine gets this wrong because the site owner has not provided guidance, it can split the authority of a page across multiple URLs, weakening all of them.
Stage Three: Ranking
Once a page is in the index, it becomes eligible to appear in search results. Ranking is the process of deciding which eligible pages to show - and in what order - when a user enters a specific query. This is where the hundreds of signals you have probably heard about come into play: relevance, authority, page experience, and more. But none of those signals matter if the page never made it through crawling and indexing first.
The pipeline runs in sequence. Crawl, then index, then rank. Most SEO advice lives in the ranking conversation, but a significant portion of visibility problems are actually crawl or indexing problems in disguise. Fixing those upstream issues is often faster and more impactful than any optimization applied to content that the search engine has never properly read.
Key Point: Ranking is the last step in a three-stage pipeline. If your pages are not being crawled efficiently or not being indexed properly, no amount of content optimization will move them in search results. Diagnose upstream before optimizing downstream.