What is the difference between how Google crawls my site and how an LLM uses it?

Googlebot works like a librarian: it crawls your HTML, renders the page, and categorises it into a massive index that returns a list of links for each query. An LLM instead converts text into tokens mapped into a high-dimensional vector space and generates an original, synthesised answer by predicting the next word, citing sources as it goes. One retrieves documents for you to read, the other reads the material and writes the answer itself.

Is traditional SEO dead now that AI tools answer questions directly?

No, because modern LLMs cannot function without the structured data layer that traditional SEO built. If Googlebot cannot crawl and index your site, Perplexity's retrieval pipeline has no data to fetch and ChatGPT has no facts to ground its synthesis. Traditional SEO gets your data discovered and indexed, while Generative Engine Optimisation structures it so the LLM selects your brand as the source of truth.

How do answer engines like Perplexity get current information about my business?

They use Retrieval-Augmented Generation, or RAG, because pre-trained models have a knowledge cutoff. When a user asks a question, the system queries a traditional search index in real time, extracts the top results, strips the HTML noise and injects the raw text into the LLM's context window. That means your visibility in these engines still depends on being crawlable and well indexed.

Crawlers vs LLMs: The Architecture of Retrieval

The internet is undergoing a structural fracture. For twenty-five years, digital visibility was dictated by a single mechanical process: crawl the site, index the site, rank the site. Today, that paradigm is being overlaid not replaced by generative synthesis.

To control how your brand is cited in this new era, you must first understand the mechanical differences between how a traditional search spider (like Googlebot) and a Large Language Model (like ChatGPT, Claude or Perplexity) ingest, process, and output your data.

Comic illustration of a mechanical crawler spider versus a luminous reading octopus

1. The Traditional Crawler: Indexing the Web

Googlebot is fundamentally a librarian. Its architecture is built on discovery, crawling, and indexing.

The Mechanics:
When Googlebot arrives at your domain, it executes a highly linear sequence. It requests the HTML payload, downloads CSS, and utilises a headless chromium rendering engine to process JavaScript and construct the Document Object Model (DOM). It evaluates the page based on structured hierarchies (H1s, H2s), internal link flow (PageRank), and keyword density.

The Output: A Lookup Table
Googlebot does not "understand" your content; it categorises it. It maps your pages into a massive, highly optimised database (the Index). When a user inputs a query, the search engine retrieves the most statistically relevant documents from this index and presents them as a list of blue links. The cognitive load of reading, synthesising, and concluding is left entirely to the human user.

2. The Generative Engine: Synthesising the Web

LLMs (ChatGPT, Gemini, Claude) and Answer Engines (Perplexity) operate on a fundamentally different mathematical foundation. They do not present lookup tables; they synthesise answers.

The Mechanics: Vector Space and RAG
Generative engines ingest data in two distinct phases:

Pre-training (The Base Knowledge)
Crawlers like OpenAI's GPTBot scrape the internet to build the LLM's base weights. During this phase, text is converted into tokens and mapped into a high-dimensional vector space. Concepts that are semantically related are positioned physically closer together in this mathematical space (Vector Proximity).
Retrieval-Augmented Generation (RAG)
Because pre-trained models have a knowledge cutoff, engines like Perplexity use real-time fetching agents. When a user asks a question, the system queries a traditional search index, extracts the top real-time results, strips the HTML noise, and injects the raw textual data into the LLM's context window.

The Output: Token Prediction
An LLM does not retrieve a document. It calculates the mathematical probability of the next word (token) based on the context window and its internal vector weights. It reads the source material and generates a completely original, synthesised answer, dynamically citing the source data as it generates.

Minilab Dual-Pipeline Architecture Diagram

The Dual-Pipeline Reality

The hyperbole claiming "SEO is dead" demonstrates a fundamental misunderstanding of generative architecture. Modern LLMs cannot function without the structured data layer that traditional SEO built. If Googlebot cannot crawl and index your site, Perplexity's RAG pipeline has no data to retrieve, and ChatGPT has no facts to ground its synthesis.

Traditional SEO ensures your data is discovered and indexed. Generative Engine Optimisation (GEO) ensures your data is structured densely enough (via semantic entity mapping and JSON-LD schema) that the LLM's mathematical weights select your brand as the definitive source of truth over a competitor.

At Minilab, we build the bridge. We make sure that you're ranking in search, and then we architect your transition to LLMs. You cannot afford to ignore either.

Whether the bridge is working is measurable: our guide to tracking brand visibility across AI answer engines covers how to baseline and monitor both pipelines.

The Architecture of Retrieval: Traditional Crawlers vs. Generative LLMs.