Digital marketing

How Does SEO Work? Search engine optimization improves the relationship between Website and search engine. Basic SEO practice is to identify queries (keywords) to which the pages on a site are relevant, and finding ways to improve the pages’ visibility in those queries on search engines (including a Website’s own site search).

 

Search Engine Optimization (SEO) is the practice of improving the relationship between Website and search engine. The first attempts to define what SEO is and how SEO works were published in the 1990s. Most of what we knew then is still relevant today, but there are more ways to improve the relationship between site and search than there were in the 90s. Optimization implies that we are seeking the best possible performance for a site in search results, meaning we want the most possible relevant referrals that help us achieve our goals for our sites.

SEO works through the following methods:

1. Improving the Crawlability of the Site
2. Managing Crawl
3. Managing the Indexability of the Site’s Content
4. Publishing Unique, Helpful, Informative Content
5. Attracting Helpful Links from Other Sites
6. Ensuring the Website is Organized Efficiently, Succinctly, and to provide fast, useful results to visitors
7. Ensuring the Website Complies with Search Engine Guidelines
8. Updating Content to Maintain Its Relevance to Searcher Interests
9. Measuring Performance to Determine Where Improvement Is Still Needed

Everything else that people associate with search engine optimization falls under these categories. Let’s take a look at some of the details for these categories.

1. Improving the Crawlability of the Site

It’s not as easy to make a site easy to crawl as you might desire. There is an immense difference between a 10-page Website and a 10,000,000-page Website. The search engine will determine how many pages it crawls and how often it crawls each page. That’s called crawl budget, and you have no way to control it.

The SEO specialist’s job is to give the search engine as many reasons to crawl a page as possible, and to ensure the page can be crawled. A Website’s crawlability is impacted by navigation, page construction, server performance, and other factors.

The SEO process improves crawlability by ensuring:

• Internal navigation (links) connects to all pages (URLs) published on a site
• Internal navigation (links) do NOT use the “rel=’nofollow’” attribute
• Published pages do NOT use the robots “nofollow” meta directive
• The “robots.txt” file only blocks pages (URLs) that should not be included in the index
• Links from other Websites point to as many URLs on a site as possible
• The server delivers content quickly and completely to crawlers
• Unwanted bots and crawlers are blocked and prevented from abusing server resources
• Pages are designed so that search algorithms can parse them (to find links and information)

2. Managing Crawl

Crawl management entails crawl monitoring. We need to confirm that the search engines are crawling the pages we want them to crawl, and we want to verify that problems are fixed. One of the most common complaints about Google is that it’s not indexing content. And yet, many people who experience this frustration never verify that Googlebot is really fetching their pages.

Your search engine optimization only works if you understand how the search engine works, at least in the most basic sense. Ignoring the fundamentals of crawler behavior costs you time and (your customer/employer) money.

We monitor crawl by:

• Monitoring live traffic
• Analyzing Web server logs
• Analyzing search engine indexing reports
• Analyzing search engine crawl error reports

All SEO specialists should be aware of and watch for these common crawling issues.

• Broken (dead) URLs (no content is served)
• The same content being served on two or more URLs (duplicate content)
• Improper or non-existing canonicalization for duplicate content
• Slow server response times (including timeout errors)
• Too many redirects from an original URL to a current URL (1 is optimal)

People still fear duplicate content. It’s not going to get you penalized. When a search engine finds duplicate content on a site, it decides which version of the content to display first in its search results. Your job as an SEO specialist is to ensure the search engine makes a good decision, not to waste time fussing over imaginary duplicate content penalties.

See my article Basic Crawl Management for more details.

3. Managing the Indexability of the Website

By indexability, I mean the pages on the site should not only be available to the search engine crawlers, they should be correctly coded and organized in a consistent manner. Whether you use structured markup is a business decision. The SEO decision is to ensure that every page make sense to both a human visitor and a multitude of search engine algorithms, each performing specialized tasks.

Merely publishing a Web document (page) does not ensure it is easy to index. Search engines attempt to index many different types of documents but when you embed executable code (PHP, JavaScript, or anything else) in those documents the situation becomes tricky. Also, if you use an interactive platform (coded in JavaScript) to dynamically retrieve and publish data to a URL, the search engine probably won’t see the content you’re fetching after the page is loading. That’s what indexability is all about.

SEO manages indexability by:

• Granting crawlers permission to fetch (crawl) pages
• Granting crawlers permission to index the content on pages
• Embedding as much indexable content in the fetched page as possible
• Limiting or removing duplicate content (publishing unique content on every URL)
• Using standard HTML markup as much as possible
• Only using special HTML markup (like rich snippets or structured data) recognized by search engines
• Ensuring that HTML code is used correctly
• Providing indexable links (the search engines can “see” and follow them)

The type of content you use on a page also matters for indexing. For example, images are indexed differently from text. It is customary to ensure that meaningful images are assigned descriptive “ALT=” text and helpful captions. Images don’t necessarily need captions, but they help. And if for some reason you can’t include captions, you can still embed caption-like content near the images.

Image galleries can be challenging, especially when the image owners (often artists or photographers) are reluctant to publish high quality images or visible text on a page. The SEO specialist must find the best possible compromise between the desired aesthetic appearance of the page and the necessary inclusion of indexable content.

Text that is published on a page via a push notification may not be indexed by search engines. The SEO specialist must find the best possible compromise between the desired functionality of the page and the necessary inclusion of indexable content.

Indexing requirements often conflict with aesthetic requirements

In many scenarios the SEO decision for a given page settles on compromises between conflicting needs. Search engine optimization only works when it supports the business decision. No site should be designed “for SEO”. The site should be designed to accomplish the publisher’s goals and to satisfy the visitor’s needs.

Sometimes, the SEO specialist should ignore popular formulaic approaches to organizing and publishing content. SEO works best when it strikes a balance between the desired appearance and functionality of a page and the desired crawling and indexing of the page.

See my article How to Manage Website Subduction to learn about and understand a common indexability problem on large Websites, especially blogs.

4. Publishing Unique, Helpful, Informative Content

It’s all about the content.

Search engines only want to show the best possible content to their users.

Effective SEO specialists work to provide good content to search engines. But given the billions of queries that searchers use every month, there is no single, universal way to identify “the best possible content”. The SEO specialist must work within a query space. A “query space” consists of all the similar or related queries for a given topic and all the relevant pages a search engine shows to searchers for those queries.

Hence, in a query space about the best horse saddles search engine optimization works to improve the usefulness, helpfulness, and thoroughness of a site’s content about horse saddles. The SEO specialist must reduce internal competition for keywords by ensuring that less relevant pages don’t overuse highly desired “targeted” keywords.

Content may be unique, helpful, and valuable regardless of how well written it is, how many words are on the page, or how well structured it is. If the content meets the searcher’s need better than any other content, it is good content.

Nonetheless, the search engines may have learned (through analysis of thousands or millions of pages in similar query spaces) to expect certain types of content or structure. In those situations the best content may be pushed down in the search results.

Search engine optimization specialists must seek a balance between framing all content in structured markup and allowing the creator to express ideas in a native style. Native style is NOT always trumped by structured markup. The goal of the content is to inform the visitors to the site. The goal of the SEO is to support the content.

Content quality is also important but it’s gravely misunderstood by (in my opinion) at least half the people in the SEO industry.

See my article titled How to Include High Quality Signals On Your Website for more information. You should also check out Quality Signals that May have Affected the March 2017 “Fred” Update.

5. Attracting Helpful Links from Other Sites

Search engines use links to find content on the Web and sometimes they use the anchor text provided by those links to understand or label the content the links point to. The King of the Link Value Hill, however, is Google’s PageRank algorithm. Similar algorithms are used by other search engines. PageRank seeks to assign qualitative value to a document through trusted links. Google engineers believe PageRank flows toward the more important (and therefore “higher quality”) pages on the Web. In practice, Web marketers have diluted the value that links provide to search engines through manipulative strategies.

But search engines have always used other signals to assess the quality and relevance of content. Certain Web marketers have promoted false ideas such as “links are the most important factor [in the search ranking algorithms]” and “all you need are links”. These false ideas, when followed, have led to hundreds of thousands or millions of Website penalties and algorithmic downgrades. You’re not optimizing for search when you set your site up for that kind of failure.

Link-based search marketing does not optimize for search. Truly optimal links are created without the knowledge or support of the site being linked to. Optimal links do not lead to penalties or algorithmic downgrades. Optimal links do not use obviously engineered anchor text that favors highly competitive search phrases. Virtually every practice associated with competitive link building for search leads to creating sub-optimal links.

Helpful, informative links provide descriptive cues to visitors about what they will find on a Website. The descriptive cues do not have to be included in the link anchor text. Helpful, informative links are always placed in a context that gives meaning and value to the links. Search engines may or may not analyze the context of the links.

The reason you want natural, helpful, informative links is that you don’t want to have to constantly replace links ignored by the search engines and you don’t want to suffer from penalties and downgrades. Link acquisition should happen without anxiety.

The natural growth of a Website’s backlink profile is a strong indication of the true competitive quality of the site’s content. Most Websites never earn very many links. Search engines understand this and award those sites with appropriate traffic based on other signals used by their algorithms.

However, in recent years marketers have become paranoid about the wrong kind of natural links. See my article How Do You Know If a Link Is Safe? for more information about what marketers are doing wrong. Another good article is Toxic Backlink Myths and How to Know Which Links are Really Bad.

6. Ensuring the Website is Organized Efficiently and Succinctly

SEO works on every aspect of Website design and construction, and that includes content organization. The SEO provider should understand how a typical visitor uses the site, and how they’ll find what they are looking for if they land on the wrong page.

Efficient use of content is necessary to create a positive experience for the visitor. Ideally they find what they are looking for quickly, they extract the information they need, and they take whatever actions they desire without obstruction. Anything that slows down the visitor journey is friction. Search engine optimization should work to reduce friction because, some people theorize, the search engine algorithms are better at ignoring high-friction pages. In fact, some algorithms eliminate friction altogether by displaying information directly in the search results.

We have more rules for what not to do with content today than we did in the 1990s. Efficient content organization includes reducing, minimizing, or eliminating obstructive page components that may negatively impact how search engines assess the quality of a page.

Efficient content organization seeks to reduce, minimize, or eliminate unnecessary on-page elements without compromising the creator’s free expression. SEO must support the style of the content creator, not dictate it. By allowing content creators to express themselves as they wish, search engine optimization specialists push themselves to expand their own skills beyond formulaic check list SEO.

Efficient content organization enhances the message without interfering with the content’s purpose. It’s not easy to explain this in simple, concise terms. A good rule of thumb is to ask yourself if you want to change content “for SEO” or “for presentation”. If you are only thinking of the SEO benefit of the proposed change then it’s the wrong change.

Presentation is more important than perceived rules of optimization strategies. In the long term, the search engines will identify and reward effective presentation styles and methods. We have seen time and time again that artificially structured content which follows SEO rules becomes obsolete as search engines learn to recognize the patterns.

There is no single, optimal way to organize content either on the page or on the Website. SEO specialists who require that content be formatted to specific rules are taking shortcuts and reducing opportunities for content to establish its unique value.

See my article Website Navigation and Website Navigation Theory for more information about how to organize a site efficiently for a good user experience.

7. Ensuring the Website Complies with Search Engine Guidelines

It blows me away how, even in 2021, many Web marketers still dismiss search engine guidelines out of hand. In fact, many people seeking help in online discussions ask questions that are quickly, directly answered by the search engine guidelines.

Search engine guidelines have evolved in part as a result of the long-running abuse of search algorithms by aggressive Web marketers. Most of the guidelines put into place by Bing, Google, and other search engines are reactions to deceptive marketing practices. The guidelines are as close as you will get to having a checklist of things that lead to penalties and algorithmic downgrades.

The time and effort marketers invest in pushing the content and links for a Website as close to the limit of what search engines allow in their guidelines are better spent creating unique, interesting, and helpful content. If the content were really that good to begin with there would be no desire to test the guidelines to find an advantage. Seriously, truly useful (“good”, “high quality”) content serves a purpose other than to make money for the publisher. Making money is fine – but put the needs of your audience first. They’ll respond to that consideration in ways that fake search engine optimization cannot replicate.

Search engine guidelines help us distinguish truly useful content from content that was created for the sake of earning traffic from search engines. When a Website is downgraded by search algorithms, penalized by spam teams, or deindexed completely, a good question to ask is which pages on the site are earning traffic from other sources and why? By removing the pages that cannot earn traffic without violating search engine guidelines the SEO specialist eliminates a major point of friction between the Website and the search engine.

Search Engine Guidelines explain what the friction points are between Websites and search engines. Creating friction between the Website and search is the exact opposite of optimization.

See my article Best Free Online SEO Tools and Guides for Beginners for links to search engine guidelines and tutorials.

8. Updating Content to Maintain Its Relevance to Searcher Interests

People who ask how SEO works often expect to be given a set of one-time strategies and tasks. “Set it and forget it” SEO does exist, but most Websites require at least some ongoing or periodic maintenance. And that work includes updating content to ensure it matches the queries and needs of contemporary searchers.

As search engines find and share content from a Website, they match that content to the queries their searchers use. Those queries change over time and so the content should change with the queries. That’s an often overlooked priority.

Web marketers have long desired to attract more visitors through a growing body of queries. Hence, they keep looking for new queries to add to their list of targets – but they seldom if ever update the targets they’ve already collected. Worse, many marketers now publish and absurd amount of frivolous content in the hope of pleasing search algorithms that, ironically, seek to ignore frivolous content. The SEO specialist must find a balance between growth in query-driven traffic and limiting content to just what is needed by the site and its visitors.

Creating content for the sake of matching queries is inefficient and leads to algorithmic downgrades. In some cases it may lead to spam penalties. A thought leader creates queries naturally by stimulating interest in new topics, leading people to search for more information about new subjects.

We use search engine optimization to monitor the queries that drive traffic to a page, to improve the quality of the content on the page to better match those queries, and to modify or expand content (where appropriate) to match overlooked queries. But that doesn’t mean search engine optimization is limited to chasing queries. Worse, if you spend all your time looking at “keyword data” from SEO tools, you’ll miss something important in your own site’s data.

The query language or search lexicon of a page often looks very different from the content creator’s assumptions about what phrases and words are relevant to the content.

We use keyword research to identify potentially useful queries for growth; but you must maintain the content you’ve already created. Keyword research is often abused by aggressive Web marketers who pursue an endless keyword strategy. Endless Keyword marketing continually adds keyword targets to a Website’s list of desired queries. This leads to an overabundance of content production and production standards suffer.

Proper keyword research identifies missed opportunities as opposed to new keywords that have no relevance to the site’s primary goal. Missed opportunities are most easily found in the query data provided by search engines rather than in SEO tools that aggregate keyword data from across the spectrum.

We also use keyword research to identify the types of queries users prefer in a specific market vertical. When the site owner has no real idea of what queries people use to find content like their own, the SEO specialist must use keyword discovery to identify queries that can be helpful to the site’s existing content. The content may need to be (almost always must be) adjusted to better match those queries.

See my articles Fundamental Principles of Keyword Analysis and Search Optimization and the Keyword Matrix for more details about keyword optimization.

9. Measuring Performance to Determine Where Improvement Is Still Needed

Monitoring Website performance is the most difficult challenge in search engine optimization. We have many metrics to choose from, but none of them are perfect. Worse, many tools that collect data do so imperfectly either because of the flawed assumptions of the tool creators or the natural limitations of the type of data being collected. Here are examples of common SEO metrics.

Direct visits consist of all page requests that are made without any referer data. Direct visits might represent people clicking on bookmarks, people clicking on links that open in new browser windows, people copying and pasting URLs into their browser address bars, and search (or non-search) referrals that don’t pass any referrer data.

Errors record page requests that failed. Server errors are identified by status codes. HTML markup errors are identified passively by crawlers. Major search engines report errors in their dashboards Search engine optimization works best when errors are identified and corrected.

Non-search referrals consist of direct visits and other site referrals; they count the number of landings that occur from non-search sources. These metrics may not be entirely reliable for various reasons. For example, some search referrals may be misidentified as direct visits.

Other site referrals are attributed to visitors clicking on links on other sites. Some spam tools are used to inflate these referral statistics with the goal of tricking the Website owner into visiting the sites mentioned in the referer field. Fake referral traffic is a widespread problem.

Page views count the number of times a specific URL was requested. Page views are easy to filter by type of URL and they do not depend on start and end times the way sessions and visits do. However, page views can be inflated by excessive repetitive requests from inappropriate sources.

Search referrals are usually divided into organic (unpaid) and PPC (paid or pay-per-click) referrals. Search referrals are easy to filter by search engine but require a little extra effort to distinguish between unpaid and paid visits. Search referral data can be misidentified and faked. Search referral data may be reported within different parameters by various analytics tools; none of the tools are necessarily wrong.

Sessions are used to represent visits and sometimes visitors, although a visitor may engage in multiple visits to a Website. One reason why sessions are a poor metric is that there is no reliable method for identifying when a session begins and ends. All session definitions are arbitrary.

Visitors may be robots (software) or people. A visitor may return to a Website may times or may “stay” on the site for long periods of time. A visitor may be a potential customer or just a random wanderer on the Web.

Visits are not exactly the same as sessions. A visit may consist of a single URL request or it may consist of hundreds of URL requests made over a period of several days. Like sessions our metrics for measuring visits are arbitrary.

There are many other metrics that various reporting tools use. None of these metrics is perfect. None of these metrics is even close to perfect. All we can ask of an SEO metric is that it be consistent.

Search engine optimization works best with consistent metrics because that allows us to identify growth in traffic patterns, anomalous issues, and seasonal patterns in searcher behavior. We also want to know which visitors are receiving errors, what kinds of errors, and if those errors are degrading Website performance, user experience, and conversions.

You can browse the SEO Metrics category on this site for many articles about measurement and analysis.

Conclusion

Despite what marketing claims you’ll see on the Web, there are no definitive guides to SEO. A “definitive” guide would have to cover everything and no guide to SEO covers everything. Worse, the longer these guides become the more difficult it is for readers to master the basics. Search engine optimization cannot be effectively reduced to a check list. You will always omit or misclassify something, according to someone else’s opinion.

What we need to understand is that search engine optimization begins with the simplest of foundations: Create a Website that is easy to crawl, index, and match to queries people use. From that foundation we build greater complexity into our optimization methods and practices.

SEO works best when it is tailored to the Website, rather than forcing a Website to comply with a specific list of SEO requirements. You’re not optimizing for search if you require every Website to do the exact same things.