Why Monitor SaaS Performance?

We use a lot of SaaS applications at Catchpoint. We also monitor SaaS performance. I can’t think of any application I use on a regular basis that isn’t SaaS. Being a technology company and having a distributed workforce—relying on SaaS applications makes sense.

SaaS is easy. Fill out a form, provide a credit card, and voila—you’re up and running. This ease-of-use, or ease-of-registration we might say, can make it challenging for corporate IT to manage and track all the applications in use.

And you might run into a situation where a 100-person company is using 200 or 300 SaaS applications, many of which provide similar functionality. Not only does corporate IT have to support employees using approved SaaS apps, but they must also manage vendor relationships, and field calls about unapproved SaaS apps.

When an employee can’t log into a system or gets an error message with an application, they contact the IT helpdesk. This kicks off a series of troubleshooting steps to identify if the problem is with the vendor or internal.

Understanding performance and availability of your SaaS Applications

Just because you don’t manage the SaaS application doesn’t mean you don’t need to monitor it. It’s vital to monitor the uptime and performance of business-critical SaaS applications to keep vendors upheld to their promised service levels.

An SLA (service level agreement) is a legal agreement that holds the vendor accountable for service degradations that affect end users. If you don’t have them now, reach out to the appropriate party and get a copy of the SLAs for all relevant SaaS applications.

What are the uptime, availability, and response objectives?

Uptime/availability are synonymous and describe the percentage of time an application was reachable in monthly or quarterly terms. Check for how vendors define uptime and handle planned or unplanned downtime in your SLA.

The most common uptime objective is 99.9% monthly availability. That’s 43.2 minutes of downtime per month before an SLA breach.

While availability is important, latency is often just as important to users as downtime. Users have higher expectations today for how long a site should take to load. A site that loads slowly might be considered unusable, unreliable, and unavailable to end users. It would be advantageous for vendors to include response times in their SLAs yet finding this is rare.

What does the SLA exclude?

Most of the time, vendors aren’t going to take responsibility for outages caused by their vendors or third parties. For example, if you’re monitoring Office365 and experience an outage or latency, you might contact Microsoft about not meeting their SLA. But, Microsoft may show you monitoring data that proves one of their third parties is responsible for the outage or latency issue.

Issues with third parties are common and can take many forms:

• An outage with an ISP
• Issues with a public DNS resolver
• Analytics or other tags on the page

Nobody wants to be held responsible for other people’s problems. SLAs also exclude disasters—like hacker attacks or weather-related problems such as hurricanes and storms. If a disaster causes downtime, your vendor likely won’t be held responsible for that downtime by the SLA.

How do you monitor SLAs?

Before monitoring SLAs, you probably want to determine which of your SaaS applications are business-critical. If your company has 100s of SaaS applications in use, you might not want to monitor all of them. Which ones are the most essential?

• Collaboration and communication applications like G-Suite, Office 365, Zoom, or Slack
• Financial applications like Oracle or Workday
• Customer support systems such as Salesforce or ServiceNow

The definition of what is business-critical will be different on a company to company basis. A social media application used by every employee likely isn’t business-critical as employees can still do their job while an application used by a few customer service employees to respond to customer inquiries would be. Determine your criteria and then develop a plan to monitor those applications.

There are many monitoring and observability tools in today’s market, but these tools don’t provide the vantage points necessary for monitoring SaaS applications. This leads to an over-reliance on end users to indicate a problem exists.

You need to monitor your vendors from as many vantage points as possible. Server and network monitoring tools are great for systems in your data center but don’t deliver much visibility of complex cloud environments controlled by third parties and scattered around the globe.

What happens if your vendor doesn’t meet the SLA?

Your SLA will define the repercussions for when vendors don’t meet their objectives. These consequences can be fines—but reimbursement takes the form of  “free service” or discounted service for a defined period of time.

SaaS performance monitoring in action

We deployed Catchpoint Enterprise Nodes to our key offices – Bangalore, Boston, New York, and our CEOs home office in LA to track performance of business-critical SaaS applications that we use like Salesforce, Office 365, and Zoom to help our Corporate IT team know whether reported issues are local or with the SaaS provider.

With this setup, we discovered a DNS issue with our Bangalore ISP provider. DNS resolutions took over 4 seconds, directly impacting page load times. When users expect entire web pages to load in under four seconds, it’s brutal when DNS resolution takes this long. We saw this response time spike for all tests we were running which led us to investigate our internal network and not the SaaS provider.

We looked to see if the problem was across all offices or just some. It was clear our India office was having a problem.

A quick change to a public DNS provider led to improved performance for our employees in Bangalore.

Takeaways

It’s essential that you monitor your SaaS vendors from as many vantage points possible. You can’t rely on cloud-only synthetic monitoring.

Make sure you’re familiar with your vendor SLAs so that you can enforce them with your own monitoring data. Also, make sure you can view your vendor’s monitoring data so that you can compare performance metrics.

Alerts will help your vendors stay on top of performance issues, and they’ll also help you and your team stay on top of your vendors should they breach their agreement.