Presence in Search Engines (SEs) is a key way of making visible products and services to users. For some of our customers, we had to allocate resources and special teams devoted to the manual task of ensuring the proper presence in Search Engine Results Pages (SERPs). This work, although crucial, consumed time and resources that could be used for other tasks.
At Ensolvers, we love to automate processes as much as possible, and that was also the case. In this article we are going to describe how the SERP assessment was automated by using common tools and tech stacks.
In this article we are going to describe a real solution implemented for one of our clients and we are going to center on Google as a primary SE since that was the concrete requirement we had to fulfill. Google already provides an API for checking the indexing status of a particular URL, which might be seen as an intuitive way of accomplishing what we needed. However, this API comes with two drawbacks: (1) it has daily limits and (2) its results are not precise - indexed pages are returned as non-indexed and vice-versa.
So, we decided to go for a different approach: simulate a user looking for a page using automation tools.
In this case, we've used the Cypress testing and automation tool, which basically simulates a browser environment and allows it to execute a search in a very similar way that a user would do it. To get started, we set up a dedicated Cypress project for our SERP testing. Then, we started writing the automation/simulation.
The test we have implemented can use a dataset of pre-generated URLs for dynamic parameterization or can be modified to fetch URLs during runtime. The choice between these methods depends on testing requirements and the need for adaptability in the testing process.
The test is simple in essence: it waits for https://google.com to load, it enters site:<url> as the query, waits for the result page to load and then it assesses if the requested URL is present in the result page.
Since Cypress can emulate browsers and user interactions in headless mode, it can be easily integrated into continuous integration pipelines to ensure that tests are run automatically as part of our development workflow. This means we can identify and address any SERP visibility issues as soon as they arise, on a daily basis.
The test results are shared with the SEO team with a Slack integration so they are constantly aware. Upon execution, the test outcome, including success or failure status and any captured screenshots, is instantly communicated to the designated communication channel. This real-time notification ensures the SEO team is promptly informed about the webpage's visibility in the Search Engine Results Page (SERP). Any failures trigger immediate attention for quick issue resolution. The integration streamlines communication, allowing the team to stay proactive in addressing and improving search engine visibility.
Implementing Cypress to automate SERP presence verification has proven to be a strategic investment for our team. By freeing up resources previously dedicated to manual tasks, we can now allocate them to other, more challenging initiatives regarding SEO and SERP optimization. As a result of this implementation, the team is notified proactively, without human intervention, if the presence of an existing page in SERPs is compromised or needs to be improved.
