It is time for the second installment of our efforts to reproduce original fuzzing research on modern systems. If you haven’t yet, please read the first part. This time we tackle fuzzing on Windows by reproducing the results of “An Empirical Study of the Robustness of Windows NT Applications Using Random Testing” (aka ‘the NT Fuzz Report’) by Justin E. Forrester and Barton P. Miller, published in 2000.

The NT Fuzz Report tested 33 applications on Windows NT and an early release copy of Windows 2000 for susceptibility to malformed window messages and randomly generated mouse and keyboard events. Since Dr. Miller published the fuzzer code, we used the exact same tools as the original authors to find bugs in modern Windows applications.

The results were nearly identical: 19 years ago 100% of tested applications crashed or froze when fuzzed with malformed window messages. Today, 93% of tested applications crash or freeze when confronted with the same fuzzer. Among the the applications that did not crash was our old friend, Calculator (Figure 1). We also found a bug (but not a security issue) in Windows.

A Quick Introduction to Windows

So what are window messages and why do they crash programs?

Windows applications that display a GUI are driven by events: a mouse move, a button click, a key press, etc. An event-driven application doesn’t do anything until it is notified of an event. Once an event is received, the application takes action based on the event, and then waits for more events. If this sounds familiar, it’s because the architecture is making a comeback in platforms like node.js.

Window messages are the event notification method in Windows. Each window message has a numeric code associated with a particular event. Each message has one or more parameters, by convention called lParam and wParam, that specify more details about the event. Examples of such details include the coordinates of mouse movement, what key was pressed, or what text to draw in a window. These messages can be sent by the program itself, by the operating system, or by other programs. They can arrive at any time and in any order, and must be handled by the receiving application.

Security Implications

Prior to Windows Vista it was possible for a low-privilege process to send messages to a high-privilege process. Using the right combination of messages, it was possible to gain code execution in the high-privilege process. These “shatter attacks” have been largely mitigated since Vista with UIPI and by isolating system services in a separate session.

Mishandling of window messages is unlikely to have a security impact on modern Windows systems for two reasons. First, window messages can’t be sent over the network. Second, crashing or gaining code execution at the same privilege level as you already have is not useful. This was likely apparent to the authors of the NT Fuzz Report. They do not make security claims, but correctly point out that crashes during window message handling imply a lack of rigorous testing.

There are some domains where same-privilege code execution may violate a real security boundary. Some applications combine various security primitives to create an artificial privilege level not natively present in the operating system. The prime examples is a browser’s renderer sandbox. Browser vendors are well aware of these issues and take steps to mitigate them. Another example is antivirus products. Their control panel runs with normal user privileges but is protected against inspection and tampering by other parts of the product.

Testing Methodology

We used the same core fuzzing code and methodology described in the original NT Fuzz Report to fuzz all applications in our test set. Specifically, in both SendMessage and PostMessage modes, the fuzzer used three iterations of 500,000 messages with the seed 42 and three iterations of 500,000 messages using the seed 1,337. We saw results after executing just one iteration of each method.

Fuzzing using the “random mouse and keyboard input” method was omitted due to time constraints and the desire to focus purely on window messages. We encourage you to replicate those results as well.

Caveats

Two minor changes were necessary to use the fuzzer on Windows 10. First was a tiny change to build the fuzzer on 64-bit Windows. The second change was enabling the fuzzer to target a specific window handle via a command line argument. Fuzzing a specific handle was a quick solution to the problem of fuzzing Universal Windows Platform (UWP) applications. The window message fuzzer is oriented to fuzzing windows belonging to a specific process, but UWP applications all display their UI via the same process (Figure 2). This meant that the fuzzer could not target the main window of UWP applications.

While modifying the fuzzer, a serious flaw was identified: the values selected for the two primary sources of randomized input, the lParam and wParam arguments to SendMessage and PostMessage, are limited to 16-bit integers. Both of the arguments are 32-bit on 32-bit Windows, and 64-bit on 64-bit Windows. The problem occurs In Fuzz.cpp, where the lParam and wParam values are set:

     wParam = (UINT) rand();
     lParam = (LONG) rand();

The rand() function returns a number in the range [0, 2¹⁶], greatly limiting the set of tested values. This bug was purposely preserved during evaluation, to ensure results were accurately comparable against the original work.

Tested Applications

The NT Fuzz Report tested 33 programs. This reproduction tests just 28 because only one version of each program is used for testing. The Windows software ecosystem has changed substantially since 2000, but there is also a surprising amount of conservation. The Microsoft Office suite features the same programs as the original tests. Netscape Communicator evolved into what is now Firefox. Adobe Acrobat was renamed to Adobe Reader, but is still going strong. Even Winamp made a new release in 2018, allowing for a fair comparison with the original NT Fuzz Report. However, some legacy software has gone the way of the last millenium. Find below the list of changes, and why:

• CD Player ⇨ Windows Media Player: The Windows Media Player has subsumed CD Player functionality.
• Eudora ⇨ Windows Mail: Qualcomm now makes basebands, not email clients. Because Eudora is no longer around, the default Windows email client was tested instead.
• Command AntiVirus ⇨ Avast Free Edition: The Command product is no longer available. It was replaced with Avast, the most popular third-party antivirus vendor.
• GSView ⇨ Photos: The GSView application is no longer maintained. It was replaced with Photos, the default Windows photo viewer.
• JavaWorkshop ⇨ NetBeans IDE: The JavaWorkshop IDE is no longer maintained. NetBeans seemed like a good free alternative that fits the spirit of what should be tested.
• Secure CRT ⇨ BitVise SSH: Secure CRT is still around, but required a very long web form to download a trial version. BitVise SSH offered a quick download.
• Telnet ⇨ Putty: The telnet application still exists on Windows, but now it is a console application. To fuzz a GUI application, we replaced telnet with Putty, a popular open-source terminal emulator for Windows.
• Freecell & Solitaire were run from the Microsoft Solitaire Collection application in the Windows App Store.

The specific application version appears in the results table. All fuzzing was done on a 64-bit installation of Windows 10 Pro, version 1809 (OS Build 17763.253).

Results

As mentioned in the NT Fuzz Report, the results should not be treated as security vulnerabilities, but instead a measure of software robustness and quality.

“Finally, our results form a quantitative starting point from which to judge the relative improvement in software robustness.”

From “An Empirical Study of the Robustness of Windows NT Applications Using Random Testing” by Justin E. Forrester and Barton P. Miller

The numbers are not particularly encouraging, although the situation is improving. In the original NT Fuzz Report, every application either crashed or froze when fuzzed. Now, two programs, Calculator and Avast Antivirus, survive the window message fuzzer with no ill effects. Our praise goes to the Avast and Windows Calculator teams for thinking about erroneous window messages. The Calculator team gets additional kudos for open sourcing Calculator and showing everyone how a high-quality UWP application is built. See Table 1 for all of our fuzzing results, along with the specific version of the software used.

Program	Version	SendMessage	PostMessage
Microsoft Access	1901	crash	crash
Adobe Reader DC	2019.010.20098	crash	ok
Calculator	10.1812.10048.0	ok	ok
Windows Media Player	12.0.17763.292	crash	crash
Visual Studio Code	1.30.2	crash	ok
Avast Free	19.2.2364	ok	ok
Windows Mail	16005.11231.20182.0	crash	crash
Excel	1901	crash	ok
Adobe FrameMaker	15.0.2.503	crash	crash
Freecell	4.3.2112.0	crash	crash
GhostScript	9.26	crash	ok
Photos	2019.18114.17710.0	crash	crash
GNU Emacs	26.1	crash	crash
IE Edge	44.17763.1.0	crash	crash
NetBeans	10	crash	crash
Firefox	64.0.2	crash	crash
Notepad	1809	crash	ok
Paint	1809	crash	crash
Paint Shop Pro 2019	21.1	crash	crash
Powerpoint	1901	crash	ok
Bitvise SSH	8.23	crash	crash
Solitaire	4.3.2112.0	crash	crash
Putty	0.70	freeze	freeze
VS Community 2017	15.9.5	crash	crash
WinAmp 5.8	5.8 Build 3660	crash	ok
Word	1901	crash	ok
Wordpad	1809	crash	crash
WS_FTP	12.7.0.1903	crash	crash

Table 1: The results of replicating the original NT Fuzz Report on Windows 10. After 19 years, very few applications properly handle malformed window messages.

A Bug in Windows?

Unfortunately our curiosity got the better of us and we had to make one exception. One common problem seemed to plague multiple unrelated applications. Some debugging showed the responsible message was WM_DEVICECHANGE. When the fuzzer sent that message, it would even crash the simplest application possible — the official Windows API HelloWorld Sample (Figure 3).

Using the HelloWorld sample we quickly realized that the problem only affects 32-bit applications, not 64-bit applications. Some rapid debugging revealed that the crash is in wow64win.dll, the 32-to-64-bit compatibility layer. My quick (and possibly wrong) analysis of the problem shows that the wow64win.dll!whcbfnINDEVICECHANGE function will treat wParam as a pointer to a DEV_BROADCAST_HANDLE64 structure in the the target program. The function converts that structure to a DEV_BROADCAST_HANDLE32 structure for compatibility with 32-bit applications. The crash happens because the wParam value generated by the fuzzer points to invalid memory.

Treating wParam as a local pointer is a bad idea, although it was probably an intentional design decision to make sure removable device notifications work with legacy 32-bit Windows applications. Regardless, it certainly feels wrong that it is possible to crash another application without explicitly debugging it. We reported the issue to MSRC, even though no security boundary was being crossed. They confirmed the bug is not a security issue. We hope to see a fix for this admittedly obscure problem in a future version of Windows.

Conclusion

Window messages are an under-appreciated and often ignored source of untrusted input to Windows programs. Even 19 years after the first open-source window message fuzzer was deployed, 93% of tested applications still freeze or crash when run against the very same fuzzer. The fact that some applications gracefully handle these malformed inputs is an encouraging sign: it means frameworks and institutional knowledge to avoid these errors exist in some organizations.

There is also much room for improvement in window message fuzzing — the simplest method possible crashes 93% of applications. There may even be examples where window messages travel across a real security boundary. If you explore this area further, we hope you’ll share what you find.