Skip to main content

Observations on Smoke Tests – Part 2

April 13, 2018

In my last blog post, I talked about the value of performing application smoke testing and some of the shortcomings to be aware of. In this post, I want to offer a brief comparison between the two main types of smoke test scanners: cloud-based and desktop-based.

There are a variety of scanning tools in the market today, from commercial to open source. Some are intended only for identifying a particular vulnerability or class of vulnerabilities, such as weak encryption settings for SSL/TLS. Other scanners are designed for comprehensive, deep-dive web application assessments or for ongoing application vulnerability management. Most commercial application scanners can be divided into two categories according to the environment from which they execute: cloud-based and desktop-based. Both have pros and cons. 

Advantages of Cloud-based vs. Desktop-based:

  1. Multi-user management.  Most cloud-based scanners allow multiple users to share the same license. Once the scanner admin receives the license, they simply invite other users to create/register for their accounts via email. In addition, cloud-based scanners often enable admins to define user groups with different permission sets, and access may be logged. These convenience features make cloud-based scanners attractive in terms of user provisioning and management.
  2. The ability to run multiple tests at the same time. Due to the inherent limitations of local hardware resources, running multiple tests simultaneously from the desktop is less feasible than from the cloud. Desktop-based scanners are still highly capable given enough CPU horsepower and memory, but default to consuming most of the machine’s available resources. Cloud-based scanners, on the other hand, tend to have vast processing and memory capabilities. The obvious benefit is that it offloads much of the work from the local machine.
  3. More reliable and user friendly. This is one of my favorites. After initiating the scan, security professionals have the freedom to use their machine for other tasks, or even reboot their computer. It is still recommended to monitor the scan in case of unexpected situations. The primary benefit is that a power outage, memory corruption or system fault on the local machine won’t interrupt or affect the scan.
  4. Dashboard overview. Cloud-based scanners commonly furnish a summary or overview screen that provides scan statistics and progress on one page. This is useful for monitoring the progress of one or more scans in one convenient view. The following screenshot shows a dashboard view, including a list of recent scans, vulnerabilities found, to-do issues, and charts showing severities and trends. This is a very straightforward way to visualize findings over time. 

Smoke Tests
Figure 1: Dashboard of one cloud-based scanner

Disadvantages of Cloud-based vs. Desktop-based:

  1. Lack of details during the scan. While running the scan, some cloud-based scanners show only a progressing bar (number of completed requests or estimated requests to be completed). Most desktop-based scanners, on the other hand, show requests currently under analysis or sequence, URLs being crawled, number of requests sent, and real-time bandwidth usage. These details give users more accurate information on the scan health and progress rate and indicate which requests the user should take a closer look at later.
  2. Slower to complete. Based on my experience, compared to desktop-based, cloud-based scanners take more time to complete a single scan on average. For example, I ran four identical tests from both environments, using the same tool vendor, and the results showed that the cloud scanner took at least 30 percent more time (depending on if the website had a lot of dynamic pages) to complete the process. 
  3. Less flexibility to change a pre-scheduled test. Some cloud-based scanners do not allow the user to change the scheduled testing window once the test begins. Desktop scanners generally have no such limit. For instance, say the user schedules the testing window from 8:00 a.m. to 5:00 p.m. and then runs the scan. Then if the user needs to adjust the time window to be 9:00 a.m. to 6:00 p.m., they have to either manually start/stop at the right time, or stop and configure another scan. Unlike scan policies, I believe testing windows should be changeable, even after a scan begins.

In this blog post I gave a brief overview of the pros and cons of performing web application scans from desktop vs. cloud environments, in addition to some of the factors to consider when choosing between these platforms. There are plenty of tools and features to select from, but in the end your choice should align with your overall desired results. 

    Raina Chen

By: Raina Chen

Security Consultant, Application Security

See More

Related Blogs

April 11, 2018

Quick Tips for Building an Effective AppSec Program – Part 1

An application security (AppSec) program can be defined as the set of risk mitigating controls and business functions that support the discovery, reme...

See Details

March 14, 2018

Observations on Smoke Tests – Part 1

Smoke testing in the traditional definition is most often used to assess the functionality of key software features to determine if they work or perfo...

See Details

May 10, 2018

Observations on Smoke Tests – Part 3

While attending one of our technology partner’s security training courses, the instructor presented on their product’s various features and capabiliti...

See Details

How Can We Help?

Let us know what you need, and we will have an Optiv professional contact you shortly.

Privacy Policy

Related Insights

September 20, 2017

Cloud Security Architecture

Learn how our experts formulate an actionable strategy with key stakeholders and help implement your cloud security program across the enterprise.

See Details

April 27, 2018

Application Security Program Management

Learn how to implement and maintain an effective AppSec program.

See Details

July 21, 2015

Application Security Solutions

Learn how Optiv can help with web, email and application protection.

See Details

Stay in the Know

For all the latest cybersecurity and Optiv news, subscribe to our blog and connect with us on Social.


Join our Email List

We take your privacy seriously and promise never to share your email with anyone.

Stay Connected

Find cyber security Events in your area.