In the world of SEO, redirect mapping plays a crucial role in ensuring a smooth user experience and maintaining organic traffic flow. However, manually mapping hundreds, thousands, or hundreds of thousands of redirects can be time-consuming and prone to errors. In this article, we will explore how you can automate the SEO redirect mapping process using two powerful tools: Screaming Frog and Python.
- What is SEO redirect mapping?
- The importance of SEO redirect mapping
- Manual vs automated redirect mappings
- Introducing Screaming Frog
- Overview of Python scripting for SEO
- Benefits of automating redirect mapping
- Setting up Screaming Frog
- Automate the redirect mapping (link to script)
- Analyzing the data
What is SEO redirect mapping?
SEO redirect mapping is the process of redirecting old URLs (Source URLs) to new ones (Destination URLs) while preserving SEO rankings and user experience.
When a website undergoes structural changes or content is moved to new locations such as a site/CMS migration, redirect mapping helps ensure that visitors and search engines are directed to the correct page. It involves creating a mapping between the old URLs and the new URLs, typically using HTTP status codes like 301 redirects.
The importance of SEO redirect mapping
Proper redirect mapping is crucial for several reasons.
Firstly, redirect mappings help maintain the SEO equity built up by the old URLs. By redirecting them to relevant new URLs, you prevent the loss of organic rankings and traffic. Old URLs can remain in Google’s index when they are properly redirected, and inversely old URLs will be removed from Google’s index if that are not redirected.
Additionally, redirect mapping ensures that users don’t encounter 404 status codes or dead ends when navigating your website from the SERP, a saved bookmark, or an email link. Redirect mappings contributes to a positive user experience, which can lead to higher engagement, conversions, and customer satisfaction.
Manual vs automated redirect mappings
Traditionally, redirect mapping has been done manually, requiring meticulous attention to detail and significant time investment. However, as websites grow larger and more complex, manual mapping becomes impractical.
Automation, on the other hand, offers efficiency, accuracy, and scalability. By automating the redirect mapping process, you can save time and reduce the risk of human error. There will typically be a manual effort involved when migrating specific pages, but automation will help in mapping a large % of pages quickly, especially if pages are templated (like many Ecommerce site pages are).
Introducing Screaming Frog
Screaming Frog is a powerful SEO tool that can crawl websites and gather valuable site data. It provides insights into a website’s structure, links, content, and more. When it comes to redirect mapping, Screaming Frog becomes invaluable. By crawling both the old and new versions of your website, it helps extract the necessary on-page, SEO elements that we’ll use for automating our redirect mapping.
Screaming Frog has a free version which allows you to crawl up to 500 URLs per crawl. If you’re migrating a medium to large site, the free tier will not suffice.
Overview of Python scripting for SEO
Python is a versatile programming language widely used in various domains, including web development and data analysis. By harnessing the power of Python, you can automate tasks and create custom scripts to streamline your SEO workflow. In the context of redirect mapping, Python allows you to interact with the data collected by Screaming Frog and generate the necessary redirect mappings.
Benefits of automating redirect mapping
Automating the redirect mapping process brings several benefits. Firstly, it saves time by eliminating the need for manual URL matching and rule creation. Secondly, it reduces the chances of errors, ensuring accurate redirects. Automation also enables scalability, making it possible to handle large-scale websites with ease. By leveraging the combined power of Screaming Frog and Python, you can supercharge your redirect mapping efforts.
Setting up Screaming Frog
Before diving into the automation process, you need to run 2 crawls:
(1) A crawl against all of the legacy site URLs
(2) A crawl against all of the new site URLs.
The site crawls should contain the standard SEO element fields, the required fields for the automation are Title 1, H1-1, and H2-1. Inside the Internal tab, filter by HTML pages and export each crawls as an .xlsx file.
Note: this article does not go into detail on the steps to gathering the necessary URLs for a site migration. This assumes you have all the valuable URLs gathered.
Automate the redirect mapping (link to script)
Now that you have your (2) site crawls, navigate to the SEO URL Redirect Mapper and perform the following steps.
- Upload the legacy site crawl.
- Upload the new site crawl.
When both site crawls are uploaded, you will see the message “Reading site crawls …” this means that your files were uploaded and are formatted correctly.
The script will start to run iterations against both files. Mappings are attempted using 5 different methods:
- Mapping URLs based on the URL paths.
- Mapping URLs based on the URL slug (last directory in a URL).
- Mapping URLs based on the <title> tags
- Mapping URLs based on the <h1> tags.
- Mapping URLs based on the <h2> tags.
The logic here, is that we want to see where the strongest similarities exist when comparing against different site-attributes. Especially since most URL paths don’t stay the during a site move, comparing <title> tags, and heading tags can be a more accurate way of mapping sites together.
Analyzing the data
After the script has finished running, download the file provided. The xlsx file will contain 5 worksheets, one for each match method.
The file will only contain similarity scores >= 0.80, but I only focus my attention on scores >= 0.90 (matches with scores below 0.90 require deeper review, since accuracy can vary between URL matches).
It is best to review each worksheet and see where the most accurate matches (Similar >= 0.90) exist. From this point, you can incorporate the best matches from the file to a redirect mapping document.