The Hawthorne Effect of Penetration Testing
August 08, 2014
Back in the 1920s, there was a study done to test room illumination and its effects on the productivity of workers. What the testers expected to see was the level of illumination at which the work was optimal. What came out of the research, however, was something quite interesting. No matter what level of illumination, productivity of the workers increased. After the study, worker output began to decline again. Why was that? Because the subjects knew when the study was taking place, and therefore knew individuals would be watching them. They didn't know the specifics of the study, but they knew their level of work effort was being monitored. Years later this came to be known as the Hawthorne effect.
You may be asking yourself, what does this have to do with present day penetration testing? Let’s discuss the similarities. When you hire someone to do a penetration test, there are probably several people who are told about it. If onsite testing is to be performed, then a certain level of access needs to be granted and the consultants will need a work station. In other words, news will likely get out that a penetration test is being performed, alerting those individuals who are defending your organization (if they were not already notified). Even if you ask people not to tell anyone, they might anyway because they don't want to see their friends in other departments “fail”. This can do a couple of things, either the penetration testers are immediately “the enemy” or every single blip that no one cared about in the past is now the pen testers’ “fault.” I've been on plenty of engagements where we are still in the initial throws of a morning meeting, laptops still in bags and a network admin pokes their head in to say “Hey we noticed something, is it you guys?” This type of behavior may be a result of the Hawthorne effect. Many people in the office know penetration testing is occurring, so they are “on guard” for everything, even things that simply don't exist, because they believe their job is being “tested”. Much like the workers mentioned in the example earlier, they believe they are being monitored. Although this is awesome for the organization as the level of network monitoring increases during the testing, I would be willing to bet that a week after the pen testers leave, that same level is not adhered to.
When we engage a client, we always recommend that they not tell anyone about the upcoming pen test. We often hear, “Well, we have to tell .” My answer is, “No, you don't.” The only people that truly need to understand that a penetration test is occurring are the individuals who asked for it, and the person signing the statement of work (if they differ). Though you may feel the need to provide notice as a professional courtesy, there is no true “need to know” there.
Before you completely write me off, let me explain a bit more. We don't look at a penetration test as something working against a system or set of systems. We treat it as a readiness campaign for your organization. Our methodology is to test in two phases. The first portion, usually a day or two, is done as passively as possible. This is beneficial because most of this phase of testing is not caught unless the organization is extremely mature. What’s even better is that we can use this phase to help tune your defenses to catch the techniques we use, many of which your adversaries are using maliciously. The second portion of the testing is to identify vulnerabilities for all systems in scope. Since this usually employs automated tools, every bell and whistle in your arsenal should be going off. However, this provides no true test of the organization because an attacker will usually follow the first approach, but never the second. If your organization can't catch the first part of the attack, then catching the second means nothing.
A true pen test includes not only the systems involved, but all of your technologies, people and processes. It's about if your configurations are holding up, if your SIEM is alerting, and if your defenses are working symbiotically and as planned so that the individuals monitoring have actionable alerts. You can leverage a penetration test to provide information across a very wide spectrum, and to learn more than just if your systems are vulnerable to the latest exploit. I believe this provides an unbiased view of the readiness of your organization and is a better use of testing time and your budget. By not telling anyone about it, you've just increased the value-add of your penetration test.