How to spy on your competitor’s product and content strategy via sitemap monitoring

Introduction

Sitemaps play a very important role in ranking websites. Just like how a map gives us direction to our destination, a sitemap gives a peek into the pages available for crawling to search engines. A sitemap, usually in the XML format, mostly contains information like the URL of the page, and metadata like change frequency, priority, etc. But how can we use sitemap monitoring to capitalize on our competitor’s product and content strategy?

For that, we first need to find the sitemap of our competitor’s website and use it with tools like Hexowatch. Since a sitemap gives insights into all the pages of our competitor’s website, we can keep a watch on what is being added, removed, or updated. The first section of this article focuses on how you can find the sitemap of a website. Once you have that, the next step is to have sitemap monitoring in place using Hexowatch.

Why sitemap monitoring?

So, why bother monitoring a website’s sitemap? It is just a plain old XML file, isn’t it? Well, for starters, imagine having first-hand knowledge of your competitor’s content marketing strategy, or their PR initiatives, or just about any other updates. Let’s imagine that you’re running a tech company A with a good userbase, and you have a competitor B which is growing fast. Company B has been on a rampage releasing new content frequently and pushing out updates to their platform, whereas your company is following a traditional approach of releasing bi-weekly updates.

After the first quarter financial results are out, you realize that a major chunk of users from your platform shifted to Company B’s platform. The main problems cited by the users are the lack of updated content and new features on your platform. What if you had the power to get to know of these changes or updates as soon as they’re done. Once you get these updates, you can alter your strategies to closely match with theirs and give them a run for their money. This is where sitemap monitoring comes in handy.

How to find the sitemap of a website

For this article, we will be using DigitalOcean, and their sitemap to perform sitemap monitoring. Broadly speaking, there are three methods to find the sitemap of a website.

1 – Hitting the direct URL of the sitemap

A lot of websites store their sitemap under the root directory itself. To access them, it is as easy as typing in the website name and adding the suffix ‘/sitemap.xml’. If this gives you an XML document with a lot of entries, you’re at the right place. If you get a 404 error, you need to try something else.

2 – Using the robots.txt file to find the URL for the sitemap

Just as you did before, try adding the suffix ‘/robots.txt’ to the website you want to search for and see if you get something similar to this.

The first sitemap entry gives you the URL to their sitemap. We can use this inside Hexowatch for sitemap monitoring.

3 – Using Google’s Advanced Search to filter out XML results

If both of the above methods didn’t work out, then you will have to use the Advanced Search option on Google and filter out the results by their file type to XML.

How to monitor any sitemap to get alerts using Hexowatch

Hexowatch comes with a handy sitemap monitoring tool that allows you to get real-time alerts. This happens whenever there is a new addition to the sitemap or whenever there is any update or deletion of the existing entries. In today’s tutorial, we will take a look at how we can achieve this using the sitemap from Digital Ocean in just 3 steps.

1 – Log in to the Hexowatch Dashboard

Visit the dashboard of your Hexowatch account. Once you log in, this would be the first screen that you see.

2 – Select Sitemap Monitoring

Click on the ‘Add URL’ option on the left-hand side. This will open up a new menu, giving you a lot of options to choose from. You have to scroll down a little bit to view the option. Clicking on that will take you to another page, where you have to fill in a couple of fields.

3 – Fill in required details

To perform sitemap monitoring, you need to fill in just a single field, which is the complete URL of the sitemap. You can add a name to keep track of it, but this, by default uses the same URL itself. In the below screenshot, we have the name ‘Digital Ocean Sitemap’. Then we need to select the frequency to keep checking the sitemap. Here, we have set it to 4 hours. You can select a value between 3 hours and 3 months for this.

The next thing to do is set the server location, the default can be chosen here. After that, you have to select the notification channel. By default, it is the email address of your Hexowatch account, but you can integrate other channels in the Settings menu.

Lastly, you can add custom webhooks or schedules, but for now, we can leave them blank. Hit the ‘Start Monitoring’ button to add them to your monitored URLs.

If you click on the ‘Monitored URLs’ option, you can see the new sitemap on the list of URLs along with some basic details. Clicking on it will give you a comprehensive view of all the events that have taken place for every check.

That’s it, folks! With Hexowatch, it is very straightforward to implement sitemap monitoring on your competitor’s website. Be it for spying on their content marketing strategies or for their next PR initiative, Hexowatch can notify you of any changes through the multitude of available notification channels.

---