Stop Obsessing Over llms.txt: A Practical Guide to Generative Engine Optimization (GEO) That Actually Works

Let me start with the most obvious shift I’ve noticed around me over the past couple of days.

After I published a keyword research tutorial last week, several people in my DMs asked me the exact same question: “I found the keywords and built the site, so how do I get AI to mention me in its answers?” In the past, when building websites, everyone obsessed over Google rankings; now, more and more people are realizing that what truly drives sign-ups and orders is ChatGPT and Doubao—it’s whether the AI calls out your name when a user directly asks, “Which one should I buy?”

There is a name for this: GEO, or Generative Engine Optimization.

Coincidentally, a landmark event happened just yesterday. DeepSeek launched its proprietary Search API on October 17, and when I dug into the official documentation, I found that it assigns two scores to every webpage: one called rerank_score, graded from 0 to 3 across relevance, authority, and timeliness; and another called authority, graded from 0 to 4. The documentation even explicitly states the thresholds: a rerank_score of 1.5 qualifies as a high-quality result, and an authority score of 3 counts as highly authoritative (Source: DeepSeek Official Documentation, api-docs.deepseek.com/zh-cn/guides/search/). Furthermore, according to an official announcement relayed by Huxiu, it specifically filters out SEO spam, GEO manipulation, and low-quality AIGC content (Reportedly, via Huxiu).

Think about what this signal means. AI search is no longer just serving up whatever it scrapes; it has started scoring and curating content.

So what should you do? I looked around online, and out of ten tutorials teaching GEO, eight tell you right off the bat to set up an llms.txt file and pile on schema markup. Honestly, after carefully digging through official documentation and several rigorous studies, my conclusion is quite counterintuitive: those two things are basically useless for the vast majority of people. What you can actually do today—and what is proven by data to work—are five other things entirely.

In this article, I will break them down for you one by one, with sources cited for every conclusion. Self-reported metrics are labeled “According to the author,” and unverified claims are marked “Reportedly”—I don’t make up numbers. This post is quite long, so I suggest bookmarking it first and following along step by step.

I. First, Understand the Real Relationship Between GEO and SEO

Many people treat GEO as something brand new that completely overturns SEO and requires starting from scratch. It’s not.

In Google’s own official guide released in May 2026, their exact words are that optimizing for generative AI search is “still SEO” (Source: Google Official AI Optimization Guide). That same guide also specifically listed a string of “things you don’t need to do,” which made me laugh when I read it, because it called out by name: llms.txt, chopping content into bite-sized chunks, rewriting content specifically for AI, artificially manufacturing mentions of your brand, and obsessing over schema.

In other words, the exact things a bunch of people are bending over backward to do are precisely what the official guidelines say you don’t need to do.

Then what actually works? A GEO paper by researchers from Princeton and other institutions (KDD 2024, arXiv:2311.09735) ran empirical tests and came to a very specific conclusion: adding statistics, including quotations, and clearly citing sources are the three most effective tactics, boosting visibility in AI citations by 30 to 40 percentage points. To be clear, 30 to 40 is the ceiling, not the average, so don’t expect that for every single post.

Even more interesting is another set of data. Websites originally ranked 5th saw an improvement of roughly 100% after implementing these changes, whereas sites originally ranked 1st actually dropped by 20% to 30%. What does this tell us? It shows that small and mid-tier sites benefit the most from GEO. Ranking lower and being relatively unknown is the exact reason you should do this, not an excuse to skip it. Also, one more thing: keyword stuffing is basically useless, so save your breath.

So in a nutshell: GEO is not the opposite of SEO; it is the fundamentals of SEO plus one extra layer—making sure AI can recognize you and wants to cite you. The five actions below revolve entirely around this principle.

II. Action 1: `robots.txt` — Let the AI Crawlers In First

This is the foundation of all foundations. For AI to cite you, its crawlers first have to be able to get in and read your site. Plenty of people build great websites only to lock the crawlers out without even realizing it.

First, understand that there isn’t just one type of AI crawler—there are three, each serving a completely different purpose:

The first type is search crawlers. These directly determine whether you can be cited by AI and must be allowed: OAI-SearchBot, PerplexityBot, Claude-SearchBot, as well as Googlebot and Bingbot.

The second type is user-triggered crawlers. These step in when a user clicks your link inside an AI interface and the AI fetches it in real time: ChatGPT-User, Perplexity-User, and Claude-User. It is also recommended to allow these.

The third type is training crawlers. These scrape your content to train their models: GPTBot, ClaudeBot, and Google-Extended. Blocking these will not affect your ability to be cited in search, so if you care about copyright, feel free to block them. (Source: Official crawler documentation from OpenAI, Perplexity, Anthropic, and Google)

There is a very common misconception here that I’ve seen trip up a lot of people: thinking that blocking Google-Extended will remove your site from Google’s AI Overviews. Wrong. AI Overviews rely on Googlebot, not Google-Extended. If you block Google-Extended, you will still show up anyway—you’re just blocking it in vain.

Here is a ready-to-copy snippet built around the logic of “allow all search crawlers, decide on training crawlers as needed”:

Plaintext

User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Training crawlers: block if you mind, won't affect citations
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /

Is allowing them in robots.txt the end of the story? There are two more pitfalls—step into either of them, and everything above goes to waste.

Pitfall 1: Cloudflare. If your domain is on Cloudflare, new domains have AI crawler management enabled by default, as Cloudflare updated its default policies twice—in July 2025 and September 2026. No matter how open your robots.txt is, if Cloudflare blocks them at the outer layer, the crawlers still can’t get in. Go check your AI Crawl Control dashboard yourself (Source: Cloudflare Official Documentation).

Pitfall 2: Rendering methods. Crawlers like GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript. That means if your content is rendered purely on the front end and relies on JS to load, they can’t see it at all—all they scrape is an empty shell (According to Vercel research). The fix is to switch to SSR (Server-Side Rendering) or SSG (Static Site Generation) so that key content is already present in the raw HTML. This is especially critical for anyone building sites with pure client-side rendering in React or Vue.

By the way, Chinese models like DeepSeek and Doubao haven’t publicly disclosed their crawler user-agents yet, so I have to mark this part as “officially undisclosed” and can’t give you exact names.

Finally, back to llms.txt. Why do I say it’s basically useless? I’m not just guessing. Ahrefs analyzed 137,000 domains and found that 97% of llms.txt files received zero requests—no AI came to read them at all (Ahrefs study); SE Ranking studied 300,000 domains and found no correlation between having it and getting cited; and Google flat-out stated they don’t use it. The takeaway in one sentence: documentation sites can add it if they want to keep things neat, but if you’re counting on it to bring you AI citations, forget about it.

III. Action 2: Page Structure — Built for Humans, Built for AI

A lot of people overcomplicate this step, spending money to pile on schema structured data as if it were some secret weapon.

Ahrefs conducted a rigorous difference-in-differences study where they added schema to 1,885 pages, and the result was virtually zero increase in AI citations (Ahrefs study). They explained the reason clearly, too: when AI fetches a page in real time, it only reads the visible HTML on the page. The invisible markup buried in your code is, more often than not, completely ignored.

So what actually works? Visible structure—which, put simply, means writing your articles clearly:

Keep a clean heading hierarchy—H1, H2, H3 nested properly without making a mess. At the very beginning of each section, directly answer the question that section is meant to address without beating around the bush. Stick to one topic per paragraph instead of cramming five different points into one. Whenever you can use a table or a list, do it—AI loves scraping structured information like that. Also, make sure critical information is written out in text rather than locked inside images, because AI won’t read text embedded in graphics (referencing Vercel’s practices).

Should you still do schema? Yes, but treat it as basic hygiene. Tag the essentials like Organization, Product, SoftwareApplication, and Article—it won’t hurt you, just don’t treat it as a magic bullet that will send your traffic to the moon.

Have you noticed that the requirements in this section are almost identical to “writing an article that is pleasant for a human to read”? That is no coincidence.

IV. Action 3: Write Truly Citable Facts

What kind of content is AI willing to cite? As the paper mentioned earlier showed, content with statistics, quotations, and sources. When it comes down to your actual writing, it boils down to four things:

Use specific numbers—don’t write “a lot,” write “37,900 times.” Pair your data with source links so it can be verified. Display the “last updated” date so it knows the information is fresh. Most valuable of all is firsthand experience—things you built and tested yourself that nobody else has.

In its guide, Google specifically highlighted the term “non-commodity content,” which simply means content others don’t have. Vercel shared a super handy self-check criterion that I now use to hold my own writing accountable: after finishing a section, ask yourself, “Could a competitor easily copy this tomorrow?” If the answer is yes, dig one layer deeper until you hit something they can’t replicate.

Here’s another thing that’s easy to forget: content needs maintenance. I recommend reviewing your posts every 30, 90, and 180 days to update outdated info while refreshing the lastmod date in your sitemap so AI knows the page is alive.

V. Action 4: Show Up Where AI Actually Looks for Citations

This step is what I consider the most critical—and most easily overlooked—move for a new site.

Simply writing great content on your own site isn’t enough, because much of what AI cites isn’t your official website at all—it’s the places where other people are talking about you.

Profound analyzed 680 million AI citations (Profound study) and found that ChatGPT loves citing Wikipedia the most, while Perplexity favors Reddit. There is a specific metric floating around that people love using to mislead others, so let me clarify it for you: the viral claim that “Reddit accounts for 46.7%” refers only to its share within the top 10 sources—if you calculate its share of all citations, Reddit is actually just 6.6%. That’s nearly a sevenfold difference, so always check how the numbers are defined.

More importantly, Chinese AI engines have completely different preferences from overseas ones. A paper from Fudan University (arXiv:2610.11932) tested this and found that Ernie Bot favors Baijiahao, while Doubao favors Toutiao—essentially, each model favors content from its own parent ecosystem. Interestingly, SMZDM (“What’s Worth Buying”) was cited by all 10 tested engines, making it a universal favorite.

So the takeaway is straightforward: if you’re targeting overseas markets, build a real presence on Reddit, YouTube, G2, and Wikipedia; if you’re targeting China, publish content on Zhihu, CSDN, SMZDM, and Toutiao.

Here is a brutal comparison. That same study mentioned someone who spun up a self-hosted WordPress site, tirelessly published 40 articles, and got zero citations after a week. Why? Because a brand-new site has zero trust in the eyes of AI. If a new site wants to be cited, it must first establish a real, discussed presence on third-party platforms. This isn’t taking a shortcut—it’s letting AI get to know you elsewhere first.

Another set of data backs up this direction. Ahrefs studied 75,000 brands and found that the correlation between brand mentions and AI Overview visibility is 0.664, whereas the correlation for backlinks is only 0.218 (Ahrefs study). In plain English: having other people mention your name matters way more than painstakingly trading backlinks.

For execution, you can borrow Tally’s playbook: build “X vs. Y” comparison pages and “X Alternatives” pages, since those are the exact two types of queries users love asking AI; run your own subreddit; and have the founder personally jump into relevant threads to answer questions. Once you show up in enough comparisons, AI simply can’t ignore you when making recommendations.

Let me add one more note about founders personally answering questions: it looks unglamorous, but it’s actually one of the highest-ROI actions you can take. Think about it—when a real human founder jumps into a relevant question on Reddit or Zhihu, thoughtfully answers someone’s query, and shares real-world use cases of their product, that piece of content is a double win for AI: it counts as both a genuine mention on a third-party platform and firsthand insight no one else could provide. Buying ten sponsored posts might not work as well as the founder thoughtfully answering a few questions themselves. Don’t be shy about doing this—nobody knows your product better than you do.

VI. Action 5: Run a Test Once a Month

The first four steps are about execution. This step is about testing. Without testing, you have no idea whether your efforts are working and are just flying blind—which won’t cut it.

The testing method isn’t complicated; here is a workflow you can follow right away:

First, prepare 15 to 20 prompts broken into three categories. Brand queries, asking directly about your name, like “How good is [Tool X]?”; category queries, asking about your niche, like “What are the best [Category X] tools?”; and comparison or alternative queries, like “Which is better, X or Y?” or “What are the best alternatives to X?”

Next, ask every single question across ChatGPT (make sure web search is turned on), Perplexity, Gemini, and Google AI Mode—and if you operate in China, add Doubao, DeepSeek, Kimi, and Yuanbao. One crucial detail: use an incognito window or log out before asking, otherwise the results will be skewed by your own chat history. Run each prompt 2 to 3 times, because AI responses fluctuate from run to run.

In case that feels abstract, let me use a fictional AI tool site as an example—let’s call it NoteFlow, an AI meeting transcription tool. Your prompt list could look like this:

Brand queries: e.g., “Is NoteFlow any good?”, “How much does NoteFlow cost?”, “Is NoteFlow safe?”

Category queries: e.g., “What are the best AI meeting transcription tools?”, “Recommended free audio-to-text meeting tools”, “Best meeting minutes software for teams”.

Comparison and alternative queries: e.g., “NoteFlow vs. Otter: which is better?”, “Affordable alternatives to Otter”, “Are there any meeting transcription tools that support Chinese?”

For your own site, just swap out NoteFlow for your brand name and swap the category for yours, and you’ll have twenty-something questions ready in no time.

Log the results in a spreadsheet with these columns: Were you mentioned? What position were you ranked in? Was the description accurate? Which URLs were cited? Who were the competitors mentioned? This spreadsheet is vital because when you run the test in month two, you need to compare it side by side with month one to see whether your actions actually pushed your rankings up.

Why go through the trouble of testing each engine one by one? Because the source overlap across different AI engines is only around 0.2—meaning performing well in ChatGPT doesn’t mean you’ll perform well in Perplexity, as they pull from almost completely different sets of sources. You have to test them separately.

Official platforms have also started rolling out data, so don’t let it go to waste: Bing Webmaster Tools launched an AI Performance report; Google Search Console has a generative AI report, though it currently only shows impressions; and in GA4, you can create a custom “AI Channel Group” to track traffic—links coming from ChatGPT automatically carry utm_source=chatgpt.com, making them very easy to spot. For a free tool, you can use HubSpot’s AI Search Grader to get an initial baseline.

There is also a super clever, low-tech trick borrowed from Tally: add a single question directly into your sign-up flow asking users, “Which AI recommended us to you?” Hearing it straight from the user’s mouth is more accurate than any analytics dashboard.

VII. 12 Real-World Case Studies — Judge for Yourself

After talking about all these methods, do they actually work? I dug up a batch of public case studies and organized them into a list for you. Self-reported numbers are labeled “According to the author,” and unverified ones are marked “Reportedly.” Pay special attention to the last one—it’s a bucket of cold water.

Tally: ChatGPT drives ~25% of new sign-ups, helping ARR grow from $2M to $3M in four months (Reportedly, via Foundation Inc.)
Vercel: ChatGPT drives ~10% of new sign-ups (Vercel Official Blog)
Ahrefs: AI search accounts for just 0.5% of traffic, yet drives 12.1% of sign-ups (Ahrefs Official)
@meepodev: A brand-new site on day 3 after launch, with zero promotion other than submitting for indexing, got 30+ sign-ups and 80+ unique visitors—mostly from ChatGPT—targeting niche image-tool keywords (According to the author)
@byCanen: One site was cited tens of thousands of times by AI and landed its first $350 GEO listing order; another product saw nearly a quarter of its 2,500+ sign-ups come from ChatGPT (According to the author)
@HiFrey: For the URL shortener TrueURL, traffic from ChatGPT has already surpassed Bing (According to the author)
@lianbozhou: On a new site with no formal promotion targeting old keywords, a user from ChatGPT bought the highest-tier plan on their very first order (According to the author)
@Ruiqin_yuan: A small utility site logged 60 sessions in a week—29 came from ChatGPT, while only 4 came from Google (According to the author)
Pinateca: A free press release syndicated across ~800 sites brought only 3 visits in 5 days; during the same period, 3 out of 6 sign-ups came from ChatGPT (Relayed by @NianBing85)
Nuwtonic: Riding the GEO wave, its monthly visits reportedly grew 10x in a short span (Broken down by @FanShifu; the 10x claim is unverified, and the product itself is a GEO tool)
Skincare Brand Case: After customers asking ChatGPT were directed to competitors and quoted outdated prices, someone packaged “AI Visibility Audits” into a paid service charging $300 per order (Relayed by @xiaoshan0413; original Reddit thread unverified)
@johnzhou123: The sobering counterexample: a tool site launched on Product Hunt and ground out 20 backlinks, yet got only 212 pageviews in 7 days, and the single download across the entire site was clicked by the founder during testing (According to the author)

I want you to take away two things from this list. First, the conversion rate from AI citations is vastly higher than its share of overall traffic—Ahrefs getting 12.1% of sign-ups from 0.5% of traffic is the classic example, because people asking AI “which one should I buy” already have their wallets out. Second, that final sobering data point is the real starting line for most people. Launching on Product Hunt and building 20 backlinks doesn’t automatically mean people will use your product. Don’t just stare at screenshots flexing wins; survivorship bias will mislead you.

VIII. Pitfalls You Are Guaranteed to Encounter — Cleared Out in Advance

Let me round up all the pitfalls mentioned earlier along with a few new ones so you can dodge them:

Blindly worshiping llms.txt and schema: We covered these thoroughly earlier—do them as basic hygiene, but don’t treat them as a silver bullet.
Confusing training crawlers with search crawlers: If you accidentally block the search crawlers that should be allowed, you can kiss any chance of getting cited goodbye.
Allowing crawlers in robots.txt while WAF or Cloudflare blocks them at the outer layer: That’s the same as not allowing them at all.
Relying solely on JS to render core content: Crawlers will only see an empty shell.
Mass-generating pages or buying fluff PR articles to “poison” the AI: I need to emphasize this one. Journalists at Xinhua News Agency ran an experiment where they invented a completely nonexistent water cup brand named “Quanjiade” (a homophone in Chinese for “completely fake”) and fed articles about it onto the web; within hours, AI models started recommending it as a real product (Xinhua News report). Does that sound like a loophole you can exploit? Stop right there. Google has already explicitly classified scaled content abuse as a violation, and as mentioned earlier, DeepSeek specifically filters out GEO manipulation. Tricks like that might fool AI in the short term, but the moment platforms tighten enforcement, your site will be the first to get wiped out—it’s simply not worth the risk. Building genuine content is the only sustainable path.
Misreading statistical definitions: The Reddit example earlier (46.7% vs. 6.6%) is a classic lesson in getting led astray by flashy numbers.
Falling for cherry-picked screenshots or agencies promising “guaranteed rankings” or “guaranteed AI recommendations”: AI answers change every single day—anyone guaranteeing results is just scamming you.
Publishing content only on your own website without showing up on third-party platforms: For a new site, doing this is basically talking to yourself.
Copy-pasting the exact same playbook between domestic and overseas markets: Different AI engines have different preferences and pull from different content ecosystems—you cannot mix them up.
Finally, don’t put GEO on a pedestal: Data from Diib shows that currently only 2.9% of websites rank ChatGPT among their top five traffic sources (Reportedly). It is a growing new opportunity well worth investing in today, but it is not the entirety of your traffic yet. Treat it seriously as an emerging acquisition channel, and you’ll keep a level head.

IX. A Self-Audit Checklist — Check Off Each Box

To wrap things up, I’ve condensed every action in this guide into a 10-point checklist—run through it against your own website and fill in whatever is missing:

Are search-type AI crawlers allowed in your robots.txt?
Is Cloudflare or your WAF blocking AI crawlers at the outer layer?
Is your core content rendered via SSR/SSG so it can be read without executing JavaScript?
Have you registered for Bing Webmaster Tools and Google Search Console to view AI-related performance data?
Is your page structure clean—with proper heading hierarchy, direct answers at the start of sections, and plenty of tables and lists?
Have you created “X vs. Y” comparison pages and “X Alternatives” pages?
Does your content include specific numbers, cited sources, and last-updated dates?
Do you have genuine mentions on third-party platforms like Reddit, Zhihu, or SMZDM?
Have you set up an AI Channel Group in GA4 to track traffic coming from ChatGPT?
Do you test your visibility across multiple AI engines once a month using a fixed set of prompts?

Final Thoughts

After writing all of this, it really boils down to one sentence: GEO is first and foremost about nailing the fundamentals of SEO, plus getting genuine mentions on third-party platforms, and finally testing and iterating every single month. There are no secret hacks—and don’t fall for anyone selling them.