What does HTTP 404 Error mean and how to use it properly

What is a 404 error

Error 404 (or “Page Not Found”) – is a standard HTTP status code that occurs when the server can’t find the document requested by a user. Basically, this is a page that doesn’t exist or broken.

How can this page look like:

• 404;
• 404 Error;
• 404 Not Found;
• Error 404;
• 404 Page Not Found;
• Error 404 Not Found;
• HTTP 404 Not Found.

What causes “404 Not Found” error and how to fix 

There are 4 reason why this http status code appears:

1. Your page was deleted from the website. In fact, it happens when an internal link led to nonexistent page. Such actions can ruin users’ experience and spoil website interlinking.
2. The user has typed the URL address incorrectly. The user tried to enter a direct link manually and made a typo.
3. Page was moved and the redirection was configured incorrectly. If the page has been deleted from the site server, this does not mean that no one will try to go to it. It will be located for some time in search engines. In addition, a link to a specific page of the site may be located in the tabs of the user’s browser.
4. The server malfunctions (it happens very rarely). Sometimes 404 error is located on the server due to its malfunctioning. The DNS cache stores the locations (IP addresses) of web servers with web pages that you recently viewed. You can clear it to ensure that only the current IP addresses are used. After completing the necessary steps, restart the browser and open the page to understand whether it was possible to restore access to it.

Ideally, there shouldn’t be any links that return 404 server response on the site (or external links). But let’s be realistic if the site has more than a thousand pages, the occurrence of such issues can’t be avoided. It can be a bit surprising though when you find a site with a small number of pages (which means that all broken pages can be traced manually) that contain three broken backlinks to articles in the “Latest News” section of the front page.

• Reload the page. The first thing came in mind. Sometimes web page can’t load properly because of Internet connection. Just press the “refresh” sign on your browser.
• Clear cache and cookies. It is easy to check. Just open a website or web page form another browser. HTTP 404 error only seems to appear on a certain computer, then the problem could lie with your browser. If the site works, you should delete cache and cookies from your browser. To find more information about browser cache and how to clear cache in your browser, you can follow this extensive guide on Google cache.

• Navigate through the website. Use a website search for needed web page,  just enter specific keywords to a certain page that you are looking for. If this does not work, you can contact the webmaster directly.

Significant increase of 404 errors

It’s one thing when a site is bypassed, and search engine robots report on several dozens of errors, but what if this number increases by hundreds or even thousands? A prompt intervention and solution should be initiated if the number of 404 pages exceeds 10% of the total number of pages on the website. According to Sitechecker’s research on “Most Common Website Errors”, 404 pages occupies 8,97% of all status codes on the website. There can be many reasons for the appearance of such a larger number of non-existent pages, but mostly they are related to problems with the code (for instance, these pages were created with the help of scripts and styles, and the styles are in the code of the template, and not in a separate file).

Another reason why robots bypass pages that have never been on the site is because of virus attacks (for example, placing doorway pages on the website). Even after its prevention and removal, robots will continue to bypass the created pages for some time afterward, and this can lead to the appearance of numerous 404 errors. But, thanks to various tools, you can easily track conversions on such links.

How to detect and track down broken links

1. Sitechecker 

One time audit in Sitechecker helps to find out all broken pages on the website and detect where links to them are placed.

Real time monitoring helps to detect broken pages that appear during last changes you have made on the website.

This feature saved our projects from decreasing traffic a few times. If you make changes on the website often, it is better to start monitoring website’s health.

2. Yandex.Metrica

To track broken links and redirections in Yandex.Metrica you need to go to “Visitor settings”. Just place this string in the counter code:

Also you should place this JS-code somewhere on the 404 page:

Here url – is the current address of the 404 code, and url_referrer is the address from which it was accessed. This way you can not only track all the broken links, but also the pages on which they are placed.

3. Google Analytics

To track the redirections using Google Analytics, you should place this code on a 404 page:

Where document.location.pathname + document.location.search is the URL of the page not found on the site and document.referrer is URL of the page which the user was redirected from.

4. Yandex.Webmaster

To find these pages follow this path.

Yandex.Webmaster > Indexing > Searchable pages > Excluded pages

Here you can select the pages that have the error using the “Error HTTP: 404” filter.

5. Google Search Console

Use this path to find out 404 codes that Google Search Console detected.

Google Search Console > Index > Coverage > Excluded

Thus, you will see the list of all URLs excluded from indexing. And 404 pages are among them.

Click on a type of error 404 and you will get on the list of URLs you want to audit.

Why can it be dangerous?

There is an opinion that pages with a 404 error directly influence the indexing of the site and its position in the search results negatively, but in most cases this statement is wrong. Knowing SERP features is important for better website optimization. What happens to such a page when the robot finds it? It removes it from the index (if it’s there, if not – the bot simply doesn’t scan it). This is natural because it is no longer informative and useful. But you need to remember the existence of SOFT errors.

SOFT 404 is not an official response from the server, but rather a tag, assigned to pages by search engines after a crawl.

Search engines make such decisions due to the following reasons:

1. There is little or no content on the page.
2. There is a redirection to the page, which does not meet the users’ requests.
3. The non-existent page doesn’t provide either a 404 or 410 server response.

What to choose: 301 or 404

After deleting the page, a new question arises: what is the next step? Do you configure a 404 server response or a 301 redirect? To determine this, each case has to be analyzed separately. For instance, a method that was previously successfully used for a webshop might not be a good choice for a site that offers services. But how to fix 404 error? Configure a 404 server response if, for some reason, a page is deleted completely from the site. 301 redirect is better when:

• The address of the page was changed (e.g. because of structural changes);
• A page is relevant to the users’ request;
• Redirection to another page with a similar content is possible (has the same theme or when going from an item page to a catalog);
• There are exist backlinks to the page.

301 redirect to the homepage is not the best solution for both users (will cause confusion) and SEO (for example, google search bots consider them as SOFT 404).

When your website has lots of 404 pages redirected to the home page via 301 redirects, Google treats such pages as soft-404s. That’s why you can find 404 errors graph in Google Search Console. Yoast recommends not to go in cycles with redirecting 404 pages. There’s nothing wrong with having a few 404 errors on your site, but you should always track them.

Yeah, it’s not a great practice (confuses users), and we mostly treat them as 404s anyway (they’re soft-404s), so there’s no upside. It’s not critically broken/bad, but additional complexity for no good reason – make a better 404 page instead.

— 🍌 John 🍌 (@JohnMu) January 8, 2019

Behavioral factors:

Remember the feeling when you visit a site in search of something, and instead of finding what you’re looking for you discover a 404 error page. What does the user usually do in this case? Closes the tab and searches for a new site of course. Naturally, the more pages on the site that show a 404 message, the more users experience this error, the less time they will spend on the site and more rejections will occur.

The search engine algorithms function in a way that special attention is paid to behavioral factors as they play a very important role in the ranking process. To avoid worsening behavioral factors, you should motivate users to stay on the site. To do this you should approach the issue of creating 404 page in an interesting and creative way.

Your own 404 error page

When making a website, you need to create your own 404 page. If you want users to stay on the site, you should think about creating and displaying your own 404 page once an error occurs. In most cases, webmasters do not pay much attention to this solution. To understand how to create perfect 404 page keep in mind objectives of this page:

• Attract the users’ attention;
• Explain what has happened;
• Suggest a solution the problem.

When speaking about your own 404 page, search engines recommend the following.

1. Google

• The page should be created in the same style (including navigation) as the main website;
• You should place the links to the most popular articles and a comment section on this page;
• Provide your users with the opportunity to report broken links.

2. Yandex

• Its design should look different from the other pages of the site;
• You should use different colors or even do not use graphics at all when creating a page.

3. General recommendations

• It should contain a link to the main page;
• You should inform the user in a polite way that the page he/she wanted to visit is not available.

The recommendations above are not unconditional rules for 404 pages. It all depends on the theme of the website and the target audience. For a web shop it is relevant to add a search line, and for site that offer services – contact information.

Examples of custom 404 pages

Lots of marketers are underestimating 404 Error Pages so it can make your customers more loyal to your business. Well-designed moments can build a brand. Consider the 404 pages as an opportunity. So use 404 Error page to remind your users what is brand is about and why they love you.

1. Hotjar

2. GitHub

3. Facebook

4. Ahrefs

5. Semrush

6. YouTube

7. Amazon

8. Zappos

9. Linkedin

10. Sitechecker.pro