Why This Chapter Matters
A webpage can look excellent to a human visitor and still remain difficult for machines to understand.
The title may be visually prominent but coded as an ordinary paragraph.
Important information may appear only after JavaScript runs.
An author's name may be visible but not connected with an author profile.
The publication date may appear on screen but have no machine-readable date format.
A product may show its price, availability and manufacturer, but these elements may exist only as unrelated pieces of text.
A report may be available through several URLs without any indication of which version is authoritative.
A website may therefore appear complete while remaining structurally unclear.
Search engines and AI systems do not understand a webpage exactly as a human does.
A person can look at a page and infer:
• this is the main heading,
• this person is the author,
• this number is the price,
• this paragraph is the answer,
• this table compares alternatives,
• and this organisation published the information.
Machines need many of these relationships to be represented more explicitly.
They examine:
• HTML structure,
• headings,
• metadata,
• links,
• canonical URLs,
• structured data,
• page source,
• sitemaps,
• crawl permissions,
• server responses,
• and relationships between entities.
This does not mean programmers should create one version of the website for people and another version for machines.
That approach may create inconsistency and mistrust.
The better approach is to build one clear, semantic and well-structured website whose meaning is visible to everyone.
Humans should see useful content.
Search engines should discover it.
AI systems should interpret it.
Assistive technologies should navigate it.
Future programmers should maintain it.
The terms AI-readable, machine-readable, search-friendly and semantically structured are sometimes used as though they refer to one technical feature.
They do not.
No single file, plugin or code block makes a website understandable to machines.
Machine readability emerges from many connected decisions:
• using the right HTML elements,
• exposing important content in the delivered HTML,
• assigning stable URLs,
• identifying duplicate pages,
• adding accurate metadata,
• marking recognised entities,
• maintaining sitemaps,
• controlling crawler access,
• improving server performance,
• and preserving consistency between visible content and machine-readable information.
This chapter brings these decisions together.
It is among the most important chapters in this handbook because every earlier chapter depends upon it.
Author identity becomes more useful when the author is represented as a machine-readable entity.
Dates become more reliable when they use valid date formats.
Internal links become more meaningful when they point towards canonical pages.
Case studies become easier to interpret when their organisation, location, dates and subjects are structured.
FAQs become useful to machines when the questions and answers exist visibly in the HTML.
Documents become discoverable when their metadata and permanent URLs are maintained.
The programmer is therefore building more than a website.
The programmer is building the layer through which the organisation’s knowledge becomes visible, interpretable and retrievable across many systems.
This requires technical discipline.
It also requires honesty.
Structured data must describe the content that actually exists.
Metadata must not make claims that the page cannot support.
Sitemaps must not contain broken or unwanted URLs.
Canonical tags must not point randomly towards unrelated pages.
Crawler controls must not accidentally hide important knowledge.
An llms.txt file must not be treated as a magical guarantee that an AI system will use the website.
Technical visibility cannot compensate for weak, inaccurate or untrustworthy knowledge.
It can only help genuine knowledge become easier to find and understand.
________________________________________
Rule 14.1; Use Semantic HTML to Express Meaning
The Rule
Use HTML elements according to the meaning and function of the content, not merely according to how they look.
Why This Rule Exists
HTML is not only a visual layout language.
It provides a vocabulary for describing the structure of a webpage.
For example:
• <header> identifies introductory page or section content.
• <nav> identifies navigation.
• <main> identifies the primary page content.
• <article> identifies a self-contained knowledge asset.
• <section> groups related content.
• <aside> contains supporting information.
• <footer> identifies concluding or supplementary information.
• <figure> connects media with a caption.
• <table> represents tabular relationships.
• <time> represents a date or time.
• <address> represents relevant contact information.
When these elements are used properly, machines receive useful structural clues.
Assistive technologies also benefit because visitors can move through the page more intelligently.
Visual appearance should be controlled through CSS.
Meaning should be expressed through HTML.
What Most Programmers Do
Generic <div> elements are used for almost everything.
A heading may be created as:
<div class="big-bold-title">Food Cost Assessment Framework</div>
It looks like a heading but does not behave like one.
A clickable element may be created from a styled rather than a real button or link.
A collection of numbers may be visually arranged in rows without using a table.
The page may appear correct while carrying very little structural meaning.
Better Practice
Use the element that matches the content.
Poor:
<div class="page">
<div class="title">Food Cost Assessment Framework</div>
<div class="menu">Home | Articles | Tools</div>
<div class="content">
This framework explains...
</div>
</div>
Better:
<header>
<h1>Food Cost Assessment Framework</h1>
</header>
<nav aria-label="Primary navigation">
<a href="/">Home</a>
<a href="/articles/">Articles</a>
<a href="/tools/">Tools</a>
</nav>
<main>
<article>
<p>This framework explains...</p>
</article>
</main>
The improved version helps the browser, assistive technologies, search engines and AI systems recognise the role of each element.
Semantic HTML should not be added merely to satisfy a checklist.
The structure must reflect the actual page.
Programming Impact
This requires:
• semantic page templates,
• correct heading hierarchy,
• meaningful form controls,
• accessible navigation,
• proper table structures,
• media captions,
• and periodic HTML validation.
________________________________________
Rule 14.2; Ensure Essential Knowledge Exists in the Delivered HTML
The Rule
The main content, headings, links, dates and essential page meaning should be available in the HTML delivered to the visitor, rather than depending entirely upon scripts that run later.
Why This Rule Exists
JavaScript can create rich and responsive experiences.
However, problems arise when the page initially contains almost no useful content and depends completely upon client-side scripts to construct everything.
A browser may eventually display the page correctly.
But some crawlers, preview tools, assistive systems or low-powered devices may not process the scripts in the same way.
Scripts may also:
• fail,
• time out,
• be blocked,
• load slowly,
• or depend upon external services.
The important distinction is not:
JavaScript or no JavaScript?
The correct question is:
Can the website’s essential knowledge still be discovered and understood if JavaScript is delayed or fails?
Interactive filters, charts and calculators may require scripts.
The page title, primary content, author, dates, answer text and important links generally should not disappear without them.
What Most Programmers Do
The server returns an almost empty document:
<body>
<div id="app"></div>
<script src="/assets/app.js"></script>
</body>
Everything else is generated inside the visitor’s browser.
This may work for some applications but is risky for important public knowledge.
Better Practice
Use one of the following where appropriate:
• server-side rendering,
• static-site generation,
• pre-rendering,
• hybrid rendering,
• or progressively enhanced HTML.
For example, deliver the main article directly:
<main>
<article>
<h1>Can 100% Red Rice Be Used to Make Pasta?</h1>
<p>
Yes. Pasta can be made entirely from red rice flour, although its
processing behaviour differs from wheat pasta.
</p>
</article>
</main>
JavaScript may later add:
• a comparison tool,
• expandable explanations,
• user voting,
• or personalised recommendations.
The knowledge itself already exists.
Programming Impact
This requires:
• rendering strategy decisions,
• meaningful server responses,
• progressive enhancement,
• script-failure testing,
• and inspection of the raw page source rather than only the browser display.
________________________________________
Rule 14.3; Build a Clear and Consistent Metadata System
The Rule
Every important page should provide accurate, page-specific metadata that reflects its visible content.
Why This Rule Exists
Metadata gives machines a concise description of the page.
Important elements may include:
• page title,
• meta description,
• canonical URL,
• language,
• author,
• publication date,
• modification date,
• social-sharing image,
• content type,
• and indexing instructions.
Metadata is not a substitute for page content.
It is a summary and identification layer.
A generic title such as:
Hello Kisan
does not explain whether the page contains:
• a news article,
• a cost calculator,
• an internship manual,
• a case study,
• or a technical report.
A more useful title might be:
Ginger Powder Cost Assessment Framework | Hello Kisan
Similarly, every page should not reuse the same promotional description.
What Most Programmers Do
Metadata is copied from the homepage.
Hundreds of pages share the same title or description.
Dates are missing.
Social-sharing previews use unrelated images.
Staging environments accidentally publish instructions such as:
<meta name="robots"
content="noindex,nofollow">
which may remain after launch.
Better Practice
Generate metadata from structured CMS fields.
Example:
<head>
<title>
Ginger Powder Cost Assessment Framework | Hello Kisan
</title>
<meta
name="description"
content="A practical cost framework for producing heat-pump-dried
ginger powder from Meghalaya ginger processed in Jaipur."
>
<meta name="author" content="Mukesh Gupta"?>
<link
rel="canonical"
href="https://example.org/food-costing/ginger-powder/"
>
<meta
property="og:title"
content="Ginger Powder Cost Assessment Framework"
>
<meta
property="og:url"
content="https://example.org/food-costing/ginger-powder/"
>
</head>
Metadata should be:
• accurate,
• concise,
• unique where appropriate,
• and generated from the same source of truth as the visible page.
Programming Impact
This requires:
• CMS metadata fields,
• automatic fallbacks,
• duplicate-title reporting,
• social-preview testing,
• language metadata,
• and safeguards against accidental noindex instructions.
________________________________________
Rule 14.4; Use Structured Data to Describe Recognised Entities
The Rule
Use relevant structured data to identify the people, organisations, articles, products, events, documents and other entities genuinely represented on the page.
Why This Rule Exists
Visible text may tell a human:
• this is an article,
• this person wrote it,
• this organisation published it,
• and this date represents the latest update.
Structured data expresses these relationships in a more predictable machine-readable form.
Schema.org provides a shared vocabulary for describing many common entities. Its terms are available in machine-readable formats, including JSON-LD.
Possible types include:
• Organization
• Person
• Article
• NewsArticle
• Report
• Product
• Service
• Event
• FAQPage
• HowTo
• VideoObject
• ImageObject
• Dataset
• BreadcrumbList
• WebSite
• WebPage
The objective is not to apply as many schema types as possible.
The objective is to represent the page truthfully.
What Most Programmers Do
A structured-data plugin is installed.
The same markup is added to every page.
A normal article may be described as a product.
A promotional paragraph may be marked as a review.
Questions hidden from visitors may be inserted as FAQ structured data.
The organisation may claim ratings that do not exist.
Better Practice
Choose a primary entity for each page.
Then connect related entities.
For example:
• the page is an Article,
• written by a Person,
• published by an Organization,
• about Food Cost Assessment,
• and connected with a downloadable DigitalDocument.
Only include information that can be supported by the visible page or reliable organisational records.
Programming Impact
This requires:
• structured content types,
• entity relationships,
• schema mapping,
• validation,
• and monitoring when page templates change.
________________________________________
Rule 14.5; Prefer JSON-LD for Structured Data Where Appropriate
The Rule
Use JSON-LD as the primary method for implementing structured data unless the project has a specific reason to use another supported format.
Why This Rule Exists
JSON-LD is a JSON-based format for expressing linked data and relationships between entities. It is designed to integrate with web systems without requiring the visible HTML markup to carry every structured-data property directly.
It can be placed in the page source using:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "The Hidden Cost of Cheap Spices",
"datePublished": "2026-07-21",
"dateModified": "2026-07-21",
"author": {
"@type": "Person",
"name": "Mukesh Gupta",
"url": "https://example.org/authors/mukesh-gupta/"
},
"publisher": {
"@type": "Organization",
"name": "Hello Kisan",
"url": "https://example.org/"
}
}
</script>
This block tells machines:
• what kind of knowledge asset the page contains,
• its title,
• its dates,
• its author,
• and its publisher.
JSON-LD is especially useful when generated from structured CMS records.
If the author changes in the CMS, both the visible page and JSON-LD should update from the same record.
What Most Programmers Do
Structured data is manually copied into each template.
Visible information changes, but JSON-LD does not.
The page may show one author while the code identifies another.
Dates may be fabricated automatically every time the page loads.
Better Practice
Generate JSON-LD from the same database fields used to build the visible page.
Example in pseudocode:
const articleSchema = {
"@context": "https://schema.org",
"@type": "Article",
headline: article.title,
datePublished: article.publishedAt,
dateModified: article.updatedAt,
author: {
"@type": "Person",
"@id": article.author.canonicalUrl,
name: article.author.name
},
publisher: {
"@type": "Organization",
"@id": site.organizationUrl,
name: site.organizationName
},
mainEntityOfPage: article.canonicalUrl
};
Validate that required values exist before publishing the block.
Do not output empty, false or invented properties merely to complete a template.
Programming Impact
This requires:
• JSON-LD generators,
• entity IDs,
• schema validation,
• shared CMS data,
• automated tests,
• and synchronisation between visible and machine-readable content.
________________________________________
Rule 14.6; Give Important Entities Stable Machine-Readable Identities
The Rule
Important people, organisations, projects, topics, products and documents should have permanent URLs that function as stable identities.
Why This Rule Exists
Machines understand knowledge better when they can recognise that repeated references point to the same entity.
Suppose an author appears as:
• Mukesh Gupta
• Mr. Mukesh Gupta
• M. Gupta
• Founder, Hello Kisan
• Editor, Hello Kisan
A human may infer that these refer to one person.
A machine may not always make that connection confidently.
A stable author URL provides a persistent identity:
https://example.org/authors/mukesh-gupta/
The same URL can be used:
• on the author profile,
• in article structured data,
• in project records,
• in case studies,
• and in APIs.
In JSON-LD, this identity can be represented through @id:
{
"@type": "Person",
"@id": "https://example.org/authors/mukesh-gupta/#person",
"name": "Mukesh Gupta"
}
The objective is not to create artificial technical complexity.
It is to give each important entity one durable digital identity.
What Most Programmers Do
Names are stored as plain text in every content record.
If a spelling changes, hundreds of pages become inconsistent.
No stable relationship exists between the person's biography and their work.
Better Practice
Create reusable entity records for:
• people,
• organisations,
• departments,
• projects,
• subjects,
• products,
• services,
• programmes,
• and important documents.
Reference those records rather than repeatedly typing names.
Programming Impact
This requires:
• canonical entity URLs,
• permanent database identifiers,
• reusable entity records,
• relationship tables,
• and consistent @id values in structured data.
________________________________________
Rule 14.7; Provide Canonical URLs
The Rule
Every indexable page should identify its preferred permanent URL, particularly when the same or substantially similar content can be reached through several URLs.
Why This Rule Exists
One page may accidentally become available through many addresses:
https://example.org/article
https://example.org/article/
https://www.example.org/article/
https://example.org/article?source=newsletter
https://example.org/category/article
Machines may treat these as separate pages.
This can fragment signals, create duplicate indexing and make it unclear which URL should be cited or shown.
A canonical URL indicates the organisation’s preferred version.
Google describes redirects and rel="canonical" as strong canonicalisation signals, while sitemap inclusion is a weaker supporting signal. It also recommends self-referential canonicals and consistency between internal links, sitemaps and canonical declarations.
What Most Programmers Do
Canonical tags are omitted.
Or every page points to the homepage:
<link rel="canonical"
href="https://example.org/">
Another common error is that filtered, translated or paginated pages all point towards an unrelated page.
Better Practice
Place a self-referencing canonical on the preferred page:
<link
rel="canonical"
href="https://example.org/food-costing/ginger-powder/"
>
Where a duplicate must remain available, point it towards the genuine preferred equivalent.
Also ensure that:
• internal links use the canonical form,
• XML sitemaps contain canonical URLs,
• redirects lead towards canonical URLs,
• and structured data uses canonical identifiers.
Canonical tags are signals, not magical commands.
They should reflect a consistent technical architecture.
Programming Impact
This requires:
• URL normalisation,
• canonical-generation rules,
• redirect strategy,
• parameter handling,
• duplicate-page reporting,
• and canonical validation.
________________________________________
Rule 14.8; Prevent Duplicate and Near-Duplicate Pages
The Rule
Do not allow the same knowledge to appear through uncontrolled combinations of URLs, filters, print views, tags, archives and tracking parameters.
Why This Rule Exists
Duplicate pages often arise unintentionally.
A CMS may create:
• category archives,
• tag archives,
• author archives,
• date archives,
• print pages,
• mobile pages,
• filtered views,
• search-result pages,
• tracking URLs,
• and downloadable HTML copies.
Some duplication is operationally necessary.
The problem arises when machines cannot determine:
• which version is authoritative,
• which version should be indexed,
• and whether the pages are meaningfully different.
Near-duplicate pages create similar problems.
An organisation may publish the same article separately under several departments with only the heading changed.
This fragments knowledge and complicates maintenance.
What Most Programmers Do
Every CMS-generated URL is allowed to be indexed.
Tracking parameters create unlimited variants.
The same content is copied into several sections.
Better Practice
For each duplicate pattern, choose the appropriate treatment:
• permanent redirect,
• canonical URL,
• noindex,
• parameter control,
• consolidation,
• or genuine differentiation.
Example:
HTTP/1.1 301 Moved Permanently
Location: https://example.org/knowledge/soil-health/
Do not use robots.txt as a substitute for canonicalisation. A blocked URL may still be known or indexed without its content being crawled.
Programming Impact
This requires:
• URL-pattern audits,
• redirect maps,
• parameter rules,
• archive decisions,
• duplicate-content detection,
• and regular index reviews.
________________________________________
Rule 14.9; Maintain robots.txt Carefully
The Rule
Use robots.txt to manage crawler access to appropriate areas without accidentally blocking important public knowledge.
Why This Rule Exists
The robots.txt file usually appears at the root of a host:
https://example.org/robots.txt
It provides crawler instructions for paths that should or should not be requested.
A simple example might be:
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Disallow: /temporary/
Allow: /
Sitemap: https://example.org/sitemap.xml
This may help keep crawlers away from:
• administrative interfaces,
• internal search results,
• temporary environments,
• duplicate filter paths,
• or technically unhelpful areas.
However, robots.txt is not an access-security mechanism.
A blocked URL may still be visited by humans.
Confidential information must be protected through authentication and authorisation, not merely through crawler instructions.
The file must also be maintained for each relevant host or subdomain.
What Most Programmers Do
A staging instruction is copied to the live website:
User-agent: *
Disallow: /
The entire website becomes unavailable to compliant crawlers.
Another mistake is blocking CSS, JavaScript or image resources required to understand and render the page.
Better Practice
Keep the file short and intentional.
Example:
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /search/
Disallow: /preview/
Disallow: /*?sort=
Disallow: /*?session=
Sitemap: https://example.org/sitemap-index.xml
Before every deployment, test:
• public articles,
• author pages,
• document pages,
• structured media,
• and sitemap URLs.
Do not assume that all AI-related crawlers have the same purpose or user-agent name.
Crawler policy should be an organisational decision supported by technical implementation.
Programming Impact
This requires:
• environment-specific files,
• deployment checks,
• crawler-policy documentation,
• resource-access testing,
• and monitoring of accidental site-wide blocking.
________________________________________
Rule 14.10; Create and Maintain XML Sitemaps
The Rule
Provide accurate XML sitemaps containing the canonical URLs of important public content.
Why This Rule Exists
A sitemap helps search engines discover pages and understand which URLs the organisation considers important. It can also communicate modification information and specialised data for images, videos or news.
A basic sitemap may look like:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>
https://example.org/food-costing/ginger-powder/
</loc>
<lastmod>2026-07-21</lastmod>
</url>
</urlset>
Large websites may use a sitemap index:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex
xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.org/sitemaps/articles.xml</loc>
<lastmod>2026-07-21</lastmod>
</sitemap>
<sitemap>
<loc>https://example.org/sitemaps/documents.xml</loc>
<lastmod>2026-07-20</lastmod>
</sitemap>
<sitemap>
<loc>https://example.org/sitemaps/authors.xml</loc>
<lastmod>2026-07-18</lastmod>
</sitemap>
</sitemapindex>
Google notes that the <lastmod> value is useful only when it is consistently accurate. It ignores sitemap <priority> and <changefreq> values.
What Most Programmers Do
The sitemap includes:
• broken URLs,
• redirected pages,
• noindex pages,
• duplicate URLs,
• temporary pages,
• and inaccurate modification dates.
Some systems update every <lastmod> value whenever the sitemap is generated, even when the page content has not changed.
Better Practice
Include only:
• public,
• canonical,
• indexable,
• successful,
• and meaningful URLs.
Separate sitemaps by content type where useful.
For example:
• articles,
• authors,
• documents,
• videos,
• products,
• case studies,
• and language versions.
Remove deleted pages promptly.
Use genuine content-modification dates.
Programming Impact
This requires:
• automated sitemap generation,
• content-status filters,
• accurate modification records,
• sitemap-index support,
• and monitoring for invalid URLs.
________________________________________
Rule 14.11; Treat llms.txt as an Emerging Discovery Aid, Not a Guarantee
The Rule
Consider providing an llms.txt file as a concise, curated guide to the website’s most important knowledge, while recognising that it remains optional and emerging.
Why This Rule Exists
The llms.txt proposal suggests placing a Markdown-style file at:
https://example.org/llms.txt
Its purpose is to provide AI systems and agents with a concise orientation to a website and links to important resources.
The original proposal was introduced in 2024 as a way of helping language models locate concise, useful content at inference time.
As of July 2026, Chrome’s Lighthouse documentation describes llms.txt as an emerging, optional convention rather than a required web standard.
It must not be confused with robots.txt.
robots.txt communicates crawler-access instructions.
llms.txt is intended as a curated knowledge guide.
It does not grant access.
It does not protect content.
It does not guarantee that any particular AI system will read or use the file.
What Most Programmers Do
Some treat llms.txt as a replacement for:
• structured data,
• XML sitemaps,
• good HTML,
• internal links,
• or crawl management.
Others automatically list every URL, turning it into a duplicate sitemap.
Better Practice
Keep it concise and curated.
Example:
# Hello Kisan – Future of Food
> Hello Kisan publishes practical knowledge for farmers,
> food entrepreneurs and mindful consumers.
## Core Knowledge
- [Food Product Cost Assessment Engine]
(https://example.org/food-costing/):
Frameworks and worked examples for manufactured food products.
- [Cooked Food Cost Assessment]
(https://example.org/cooked-food-costing/):
Ingredient, conversion and business-operation costing.
- [Urban Farming Knowledge Centre]
(https://example.org/urban-farming/):
Practical resources for household and rooftop farming.
## Organisation
- [About Hello Kisan]
(https://example.org/about/):
Mission, editorial purpose and institutional background.
- [Authors and Experts]
(https://example.org/authors/):
People responsible for the organisation's knowledge.
Links should point towards canonical pages.
The file should be updated whenever the organisation’s knowledge architecture changes substantially.
Do not fill it with promotional slogans.
Programming Impact
This requires:
• a root-level llms.txt,
• Markdown or plain-text delivery,
• editorial curation,
• canonical links,
• update ownership,
• and periodic validation.
________________________________________
Rule 14.12; Keep Visible Content and Machine-Readable Data Consistent
The Rule
Machine-readable information should accurately represent what visitors can see and verify on the page.
Why This Rule Exists
A page may display:
Written by Mukesh Gupta
while its structured data identifies another person.
A product may visibly show no reviews while JSON-LD claims:
"ratingValue": "4.9",
"reviewCount": "864"
An article may show an original publication date of 2022 while its code generates today’s date on every visit.
These inconsistencies damage trust.
They also make it difficult for machines to determine which information is reliable.
Structured data should reveal relationships more clearly.
It should not invent a different reality.
What Most Programmers Do
Visible templates and structured-data templates are maintained separately.
One is updated while the other remains stale.
Placeholder ratings, prices, dates or organisations survive into production.
Better Practice
Use one structured source of truth.
For example:
CMS Article Record
│
├── Visible author name
├── Author profile link
├── Article JSON-LD author
├── Social metadata author
└── API author record
All outputs should derive from the same relationship.
Add automated checks for contradictions.
For example:
if (page.reviewCount === 0) {
omitAggregateRating();
}
if (!page.visibleAuthor) {
blockArticleSchemaPublication();
}
Programming Impact
This requires:
• shared data models,
• template consistency,
• production validation,
• placeholder detection,
• and automated structured-data tests.
________________________________________
Rule 14.13; Use Correct HTTP Status Codes and Redirects
The Rule
The server should accurately communicate whether a resource exists, has moved, is unavailable or has failed.
Why This Rule Exists
Machines depend heavily upon server responses.
Common status codes include:
• 200 — successful response,
• 301 — permanently moved,
• 302 or 307 — temporarily redirected,
• 404 — not found,
• 410 — deliberately removed,
• 429 — too many requests,
• 500 — server error,
• 503 — temporarily unavailable.
A page displaying “Not Found” while returning 200 OK creates a soft error.
A deleted article redirected automatically to the homepage may mislead visitors and machines.
A temporary maintenance page returning 200 may be indexed as though it were the actual content.
What Most Programmers Do
Every request returns 200.
Deleted URLs redirect to the homepage.
Redirect chains develop:
Old URL
→ Second URL
→ Third URL
→ Current URL
Better Practice
Return the status that reflects reality.
Example permanent redirect:
HTTP/1.1 301 Moved Permanently
Location: https://example.org/new-permanent-url/
Example removed resource:
HTTP/1.1 410 Gone
Content-Type: text/html; charset=utf-8
Where a replacement exists, explain it clearly.
Avoid long redirect chains.
Update internal links so visitors go directly to the final destination.
Programming Impact
This requires:
• status-code rules,
• redirect management,
• broken-link monitoring,
• custom error pages,
• and server-log analysis.
________________________________________
Rule 14.14; Improve Server Performance and Core Web Vitals
The Rule
Deliver important pages quickly, responsively and with minimal visual instability.
Why This Rule Exists
Machine readability is not separate from technical performance.
A page that responds slowly, repeatedly fails or requires excessive processing becomes harder to crawl and frustrating to use.
Core Web Vitals currently focus upon:
• Largest Contentful Paint, which represents loading performance;
• Interaction to Next Paint, which represents responsiveness;
• Cumulative Layout Shift, which represents visual stability.
Current guidance considers performance good when, at the 75th percentile of page loads, LCP is within 2.5 seconds, INP is 200 milliseconds or less, and CLS is 0.1 or less.
These numbers are not the whole purpose.
The larger objective is to provide a stable and usable experience under real conditions, especially on mobile devices and slower networks.
What Most Programmers Do
Large scripts load on every page.
Images are delivered at full original size.
Fonts block rendering.
Advertisements or widgets push the content down after loading.
The server performs expensive database queries for simple pages.
Better Practice
Possible improvements include:
• server and page caching,
• image compression,
• responsive image sizes,
• lazy-loading non-critical media,
• reserving image dimensions,
• reducing unused JavaScript,
• code splitting,
• content delivery networks,
• database indexing,
• efficient font delivery,
• and removing unnecessary third-party scripts.
Example responsive image:
<img
src="/images/rooftop-garden-800.webp"
srcset="
/images/rooftop-garden-480.webp 480w,
/images/rooftop-garden-800.webp 800w,
/images/rooftop-garden-1280.webp 1280w
"
sizes="(max-width: 600px) 100vw, 800px"
width="800"
height="450"
alt="Vegetable grow bags arranged on a Jaipur rooftop"
loading="lazy"
>
Providing width and height helps reserve space and reduce layout movement.
Programming Impact
This requires:
• field performance measurement,
• caching,
• media optimisation,
• script budgets,
• database tuning,
• and performance testing across realistic mobile conditions.
________________________________________
Rule 14.15; Keep the Page Source Clean and Understandable
The Rule
Avoid unnecessary code, duplicated markup, hidden content and script clutter that obscures the page’s essential meaning.
Why This Rule Exists
A page source does not need to be visually beautiful to a programmer.
It does need to be coherent.
Problems arise when a simple article generates:
• thousands of unnecessary HTML elements,
• multiple conflicting metadata blocks,
• repeated navigation,
• hidden copies of the article,
• obsolete scripts,
• inline styling everywhere,
• and several contradictory schema blocks.
Such clutter complicates:
• maintenance,
• debugging,
• rendering,
• accessibility,
• and machine interpretation.
Clean source does not mean removing useful structure.
It means ensuring that each element has a purpose.
What Most Programmers Do
Page builders and plugins add layers of wrappers.
Old features remain loaded even after they are no longer used.
Several plugins generate competing structured data.
Hidden text is inserted for search engines.s
Better Practice
Periodically inspect:
• raw HTML source,
• rendered DOM,
• network requests,
• structured-data blocks,
• headings,
• canonical tags,
• and indexing directives.
Remove:
• duplicate scripts,
• unused styles,
• placeholder metadata,
• hidden keyword blocks,
• duplicate content,
• and obsolete plugin output.
Example of unnecessary nesting:
<div>
<div>
<div>
<div class="heading">
Organic Farming Knowledge Centre
</div>
</div>
</div>
</div>
Better:
<h1>Organic Farming Knowledge Centre</h1>
Programming Impact
This requires:
• component discipline,
• dependency reviews,
• HTML audits,
• template simplification,
• and code-quality standards.
________________________________________
Rule 14.16; Allow Legitimate Crawlers While Blocking Junk Areas
The Rule
Make deliberate decisions about which public resources machines may access and which areas provide no public knowledge value.
Why This Rule Exists
Not every website path deserves crawling.
Low-value areas may include:
• internal search results,
• shopping-cart states,
• login screens,
• session URLs,
• empty tag archives,
• preview pages,
• administrative areas,
• duplicated filters,
• and infinite calendar pages.
Allowing uncontrolled crawling can waste server resources and create large numbers of weak or duplicate URLs.
At the same time, indiscriminate blocking can hide:
• articles,
• media,
• documents,
• structured resources,
• or JavaScript required for rendering.
AI-related crawlers may have different purposes, including search, retrieval, assistant responses or model development. Organisations may choose different policies for different crawler categories.
Those decisions should be documented rather than improvised by individual developers.
What Most Programmers Do
All crawlers are either allowed everywhere or blocked everywhere.
No one knows why the policy exists.
Better Practice
Create a crawler-policy register containing:
| Area | Public Value | Crawl Decision | Reason |
| Articles | High | Allow | Core public knowledge |
| Author Profiles | High | Allow | Expertise and identity |
| Internal Search Results | Low | Block or noindex | Duplicate combinations |
| User Accounts | None | Block | Private area |
| Preview Pages | None | Block | Unpublished content |
| Documents | High where public | Allow | Knowledge resources |
| Admin Area | None | Block and authenticate | Security |
Crawler controls, indexing controls and access security should remain separate concepts.
Programming Impact
This requires:
• crawler classification,
• robots policies,
• noindex rules,
• authentication,
• rate limiting,
• server monitoring,
• and documented organisational governance.
________________________________________
Rule 14.17; Provide Machine-Readable Dates, Languages and Relationships
The Rule
Dates, languages, authorship and content relationships should be represented in valid, predictable formats.
Why This Rule Exists
A human may understand:
Updated last Tuesday
A machine may need an exact date:
<time datetime="2026-07-21">21 July 2026</time>
Similarly, the page language should be declared:
<html lang="en">
A Hindi page may use:
<html lang="hi">
Language alternatives can be connected:
<link
rel="alternate"
hreflang="en"
href="https://example.org/en/urban-farming/"
>
<link
rel="alternate"
hreflang="hi"
href="https://example.org/hi/urban-farming/"
>
<link
rel="alternate"
hreflang="x-default"
href="https://example.org/urban-farming/"
>
These declarations help machines distinguish:
• human display text,
• actual dates,
• language versions,
• and equivalent pages.
What Most Programmers Do
Dates appear only as decorated text.
Language declarations remain unchanged across translated pages.
Alternative versions are unconnected.
Better Practice
Use standard date formats internally.
Generate human-readable presentation separately.
Maintain explicit relationships between language versions rather than guessing from URL patterns.
Programming Impact
This requires:
• ISO-formatted dates,
• language records,
• translation relationships,
• hreflang generation,
• and date validation.
________________________________________
Rule 14.18; Make APIs and Data Feeds Available Where They Serve a Real Purpose
The Rule
Provide structured APIs or feeds for knowledge that genuinely benefits from machine reuse, while protecting sensitive and proprietary data.
Why This Rule Exists
Some organisational knowledge is useful beyond individual webpages.
Examples include:
• public job listings,
• event schedules,
• document catalogues,
• agricultural price data,
• research datasets,
• product specifications,
• and public resource directories.
An API can allow authorised systems to retrieve structured records consistently.
For example:
{
"title": "Heat Pump Drying of Ginger",
"contentType": "TechnicalGuide",
"language": "en",
"published": "2026-07-21",
"author": {
"name": "Mukesh Gupta",
"url": "https://example.org/authors/mukesh-gupta/"
},
"canonicalUrl":
"https://example.org/food-processing/heat-pump-ginger/"
}
However, an API should not be created merely because it sounds advanced.
It requires:
• documentation,
• access policy,
• stability,
• security,
• versioning,
• and maintenance.
What Most Programmers Do
Either all data is trapped inside webpages, or an undocumented endpoint exposes more information than intended.
Better Practice
Provide feeds where machine reuse creates genuine value.
Possible formats include:
• JSON APIs,
• RSS or Atom feeds,
• CSV datasets,
• calendar feeds,
• and structured document indexes.
Define:
• public fields,
• access limits,
• update frequency,
• licence,
• and version policy.
Programming Impact
This requires:
• API design,
• authentication where needed,
• versioning,
• rate limits,
• documentation,
• and privacy review.
________________________________________
Rule 14.19; Validate Machine Readability Continuously
The Rule
Machine readability should be tested as part of regular development and publishing workflows, not checked only when the website launches.
Why This Rule Exists
A website changes constantly.
New templates are added.
Plugins are updated.
Metadata rules change.
Editors create new content types.
Redirects accumulate.
Structured data becomes incomplete.
Sitemaps begin containing errors.
A technically sound website can gradually become inconsistent.
Validation should therefore be continuous.
Testing should include:
• raw HTML availability,
• heading structure,
• canonical URLs,
• HTTP status codes,
• structured data,
• sitemap contents,
• robots rules,
• metadata,
• mobile performance,
• accessibility,
• and broken links.
What Most Programmers Do
The homepage is tested once.
Thousands of internal pages remain unchecked.
A template error may affect an entire section for months.
Better Practice
Build automated and manual checks into the workflow.
Example deployment tests:
describe("Article page", () => {
it("returns HTTP 200", async () => {
expect(response.status).toBe(200);
});
it("contains one H1", () => {
expect(document.querySelectorAll("h1")).toHaveLength(1);
});
it("contains a self-referencing canonical", () => {
expect(canonical.href).toBe(expectedCanonicalUrl);
});
it("contains visible article text", () => {
expect(article.textContent.length).toBeGreaterThan(500);
});
it("contains valid publication metadata", () => {
expect(articleSchema.datePublished).toMatch(
/^\d{4}-\d{2}-\d{2}/
);
});
});
Technical tests should be supplemented by real inspection.
A machine may confirm that structured data is syntactically valid while a human notices that it describes the wrong entity.
Programming Impact
This requires:
• automated test suites,
• crawl audits,
• structured-data validation,
• performance monitoring,
• deployment gates,
• and scheduled technical reviews.
________________________________________
Rule 14.20; Design for Search and AI Without Writing for Machines Alone
The Rule
Technical optimisation should improve the visibility of genuine knowledge without making the content unnatural, repetitive or misleading for human readers.
Why This Rule Exists
Whenever a new discovery system emerges, organisations attempt to manipulate it.
Earlier websites repeated keywords unnaturally.
Later, they produced shallow pages for every possible search phrase.
Now some organisations may be tempted to create artificial content designed only to influence AI answers.
This is a weak long-term strategy.
Search engines and AI systems change.
The organisation’s credibility remains.
The strongest machine-readable page is usually one that:
• answers a real question,
• identifies its source,
• explains its context,
• uses clear structure,
• links to evidence,
• carries accurate dates,
• and remains useful to people.
Technical elements help machines interpret that knowledge.
They do not replace it.
What Most Programmers Do
Machine visibility becomes a separate layer managed through hidden text, excessive markup or automatically generated pages.
Better Practice
Begin with the human purpose.
Then represent that purpose technically.
For example:
1. Write the genuine answer.
2. Structure it with meaningful headings.
3. identify the author and organisation.
4. Add accurate dates.
5. Connect supporting evidence.
6. assign a canonical URL.
7. include the page in the sitemap.
8. represent relevant entities through structured data.
9. test the delivered HTML.
10. monitor performance and accessibility.
This is not writing for humans instead of machines.
It is creating knowledge that both can understand.
Programming Impact
This requires:
• editorial and technical collaboration,
• quality controls,
• avoidance of automated content pollution,
• transparent metadata,
• and long-term knowledge governance.
________________________________________
Suggested Technical Architecture for Machine-Readable Knowledge
| Component | Purpose |
| Semantic HTML | Expresses page structure and meaning |
| Server-Rendered Content | Makes essential knowledge immediately available |
| Metadata | Summarises page identity and purpose |
| Canonical URL | Identifies the preferred permanent version |
| JSON-LD | Describes entities and relationships |
| Entity IDs | Connects repeated references to the same person, organisation or resource |
| XML Sitemap | Lists important canonical public URLs |
| robots.txt | Manages compliant crawler access |
| llms.txt | Provides an optional curated guide for AI-oriented discovery |
| Language Metadata | Identifies language and translated alternatives |
| HTTP Status Codes | Communicates resource condition accurately |
| Redirects | Preserves continuity when URLs change |
| Performance Layer | Improves delivery, rendering and interaction |
| API or Feed | Supports structured reuse where appropriate |
| Validation System | Detects technical errors and contradictions |
| Monitoring | Tracks crawl failures, broken URLs and performance |
________________________________________
Suggested CMS Fields for AI and Machine Readability
| Field | Purpose |
| Content Type | Article, Report, Person, Organisation, Product etc. |
| Canonical URL | Preferred permanent page address |
| Page Title | Browser and discovery title |
| Meta Description | Concise page summary |
| Primary Entity | Main subject represented by the page |
| Author | Responsible creator |
| Publisher | Publishing organisation |
| Publication Date | Original release |
| Modification Date | Genuine latest update |
| Language | Primary language |
| Alternate Languages | Equivalent translated pages |
| Indexing Status | Index, Noindex or Restricted |
| Sitemap Status | Include or Exclude |
| Structured Data Type | Relevant Schema.org type |
| Entity Identifier | Stable URL or machine-readable ID |
| Related Entities | People, projects, organisations and subjects |
| Social Image | Sharing preview |
| Robots Policy | Page-specific crawler instructions where required |
| Change Reason | Why machine-readable details were altered |
| Validation Status | Passed, Warning or Failed |
| Last Technical Review | Most recent inspection |
The programmer should avoid asking editors to write JSON-LD manually.
Editors should enter factual information through understandable CMS fields.
The system should transform those records into:
• visible content,
• metadata,
• structured data,
• sitemaps,
• feeds,
• and relationship links.
This reduces contradiction and improves maintainability.
________________________________________
A Practical Machine-Readable Article Template
The following simplified example demonstrates how several rules can work together:
<!doctype html>
<html lang="en">
<head>
<meta
name="viewport"
content="width=device-width, initial-scale=1"
>
<title>
The Hidden Cost of Cheap Spices | Hello Kisan
</title>
<meta
name="description"
content="How pesticide-intensive production may make spices
appear cheaper while creating hidden food-safety and
environmental costs."
>
<link
rel="canonical"
href="https://example.org/food/hidden-cost-cheap-spices/"
>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"@id":
"https://example.org/food/hidden-cost-cheap-spices/#article",
"headline": "The Hidden Cost of Cheap Spices",
"description":
"How pesticide-intensive production may create hidden costs.",
"datePublished": "2026-07-21",
"dateModified": "2026-07-21",
"inLanguage": "en",
"mainEntityOfPage":
"https://example.org/food/hidden-cost-cheap-spices/",
"author": {
"@type": "Person",
"@id":
"https://example.org/authors/mukesh-gupta/#person",
"name": "Mukesh Gupta",
"url":
"https://example.org/authors/mukesh-gupta/"
},
"publisher": {
"@type": "Organization",
"@id": "https://example.org/#organization",
"name": "Hello Kisan",
"url": "https://example.org/"
}
}
</script>
</head>
<body>
<header>
<a href="/" aria-label="Hello Kisan home">
Hello Kisan
</a>
</header>
<nav aria-label="Primary navigation">
<a href="/food/">Food</a>
<a href="/farmers/">Farmers</a>
<a href="/tools/">Tools</a>
<a href="/library/">Knowledge Library</a>
</nav>
<main>
<article>
<header>
<p>Food Safety and Agriculture</p>
<h1>The Hidden Cost of Cheap Spices</h1>
<p>
Written by
<a href="/authors/mukesh-gupta/">
Mukesh Gupta
</a>
</p>
<p>
Published
<time datetime="2026-07-21">
21 July 2026
</time>
</p>
</header>
<p>
India’s relationship with spices extends across centuries...
</p>
<section aria-labelledby="pesticide-cost">
<h2 id="pesticide-cost">
The Cost That Does Not Appear on the Packet
</h2>
<p>
Increased chemical use may raise output while creating...
</p>
</section>
<section aria-labelledby="references">
<h2 id="references">References</h2>
<ul>
<li>
<a href="https://official-source.example/report">
Official pesticide-residue monitoring report
</a>
</li>
</ul>
</section>
</article>
</main>
<footer>
<p>
Published by Hello Kisan – Future of Food.
</p>
</footer>
</body>
</html>
This example includes:
• semantic HTML,
• one clear main heading,
• visible authorship,
• machine-readable dates,
• canonical URL,
• JSON-LD,
• stable entity identities,
• descriptive links,
• and essential content in the original HTML.
It is not a complete production template.
It demonstrates the principle that human presentation and machine meaning should arise from the same knowledge record.
________________________________________
Technical Launch Checklist
Before publishing an important section of the website, the development team should verify the following.
Page Delivery
• The server returns the correct status code.
• Essential content exists in the delivered HTML.
• The page works when non-essential JavaScript fails.
• Mobile visitors can read and interact with it.
Page Identity
• The page has a unique and meaningful title.
• The meta description reflects the visible content.
• The canonical URL is correct.
• The language is declared.
• Publication and modification dates are accurate.
Semantic Structure
• The page contains one clear main heading.
• Heading levels follow a meaningful sequence.
• Navigation, article, section, table and media elements use appropriate HTML.
• Links and buttons use their correct elements.
Structured Data
• The schema type matches the visible content.
• The author and organisation records are correct.
• Dates, ratings and prices are genuine.
• JSON-LD is generated from current CMS data.
• Machine-readable information does not contradict the page.
Crawl and Discovery
• The page is not accidentally blocked.
• The page is included in the correct sitemap.
• The sitemap uses the canonical URL.
• Internal links point towards the canonical page.
• Deleted or moved URLs return appropriate responses.
Performance
• Images are appropriately sized.
• Layout dimensions are reserved.
• Unnecessary scripts are removed.
• Mobile field performance is monitored.
• Server failures and slow responses are investigated.
Maintenance
• Structured-data tests run after template changes.
• Redirects are periodically reviewed.
• Sitemaps exclude invalid URLs.
• robots.txt is checked after deployment.
• llms.txt, where provided, remains current.
• Technical ownership is assigned.
________________________________________
Chapter Summary
AI readability is not achieved by adding one file, one schema block or one plugin.
It emerges from the complete technical behaviour of the website.
Semantic HTML explains the role of each element.
Structured data identifies entities and relationships.
Canonical URLs establish authoritative page identities.
XML sitemaps help machines discover important resources.
robots.txt manages crawler access.
An optional llms.txt file may provide a curated orientation for AI systems, but it does not replace established web architecture.
Correct metadata summarises the page.
Server-rendered HTML exposes essential knowledge.
Status codes explain whether resources exist or have moved.
Performance determines whether pages can be delivered and used efficiently.
Clean source reduces confusion.
Stable entity identities connect people, organisations, projects and knowledge assets.
The programmer’s responsibility is not to trick machines into noticing the organisation.
It is to remove unnecessary technical barriers between the organisation’s genuine knowledge and the systems attempting to discover, interpret and retrieve it.
This requires honesty.
Structured data must match visible content.
Dates must be real.
Ratings must be genuine.
Canonical URLs must reflect actual page relationships.
Sitemaps must contain valid public resources.
Crawler policies must be deliberate.
AI-oriented files must not make unsupported claims.
For human visitors, this architecture produces faster, clearer and more accessible pages.
For search engines, it produces discoverable and consistently identified resources.
For AI systems, it produces knowledge that is easier to interpret, connect and retrieve.
For the organisation, it establishes a durable technical foundation through which decades of expertise can remain visible even as discovery technologies continue to change.
That is the central purpose of an Organisation’s Knowledge Visibility Ecosystem:
not merely to place information online, but to ensure that genuine organisational knowledge can be found, understood, connected and responsibly used by humans and machines alike.
