2 <title>Writing Conformance tests</title>
4 <sect1 id="testing-intro">
5 <title>Introduction</title>
7 With more The Windows API follows no standard, it is itself a de facto
8 standard, and deviations from that standard, even small ones, often
9 cause applications to crash or misbehave in some way. Furthermore
10 a conformance test suite is the most accurate (if not necessarily
11 the most complete) form of API documentation and can be used to
12 supplement the Windows API documentation.
15 Writing a conformance test suite for more than 10000 APIs is no small
16 undertaking. Fortunately it can prove very useful to the development
17 of Wine way before it is complete.
21 The conformance test suite must run on Windows. This is
22 necessary to provide a reasonable way to verify its accuracy.
23 Furthermore the tests must pass successfully on all Windows
24 platforms (tests not relevant to a given platform should be
28 A consequence of this is that the test suite will provide a
29 great way to detect variations in the API between different
30 Windows versions. For instance, this can provide insights
31 into the differences between the, often undocumented, Win9x and
35 However, one must remember that the goal of Wine is to run
36 Windows applications on Linux, not to be a clone of any specific
37 Windows version. So such variations must only be tested for when
38 relevant to that goal.
43 Writing conformance tests is also an easy way to discover
44 bugs in Wine. Of course, before fixing the bugs discovered in
45 this way, one must first make sure that the new tests do pass
46 successfully on at least one Windows 9x and one Windows NT
50 Bugs discovered this way should also be easier to fix. Unlike
51 some mysterious application crashes, when a conformance test
52 fails, the expected behavior and APIs tested for are known thus
53 greatly simplifying the diagnosis.
58 To detect regressions. Simply running the test suite regularly
59 in Wine turns it into a great tool to detect regressions.
60 When a test fails, one immediately knows what was the expected
61 behavior and which APIs are involved. Thus regressions caught
62 this way should be detected earlier, because it is easy to run
63 all tests on a regular basis, and easier to fix because of the
64 reduced diagnosis work.
69 Tests written in advance of the Wine development (possibly even
70 by non Wine developers) can also simplify the work of the
71 future implementer by making it easier for him to check the
72 correctness of his code.
77 Conformance tests will also come in handy when testing Wine on
78 new (or not as widely used) architectures such as FreeBSD,
79 Solaris x86 or even non-x86 systems. Even when the port does
80 not involve any significant change in the thread management,
81 exception handling or other low-level aspects of Wine, new
82 architectures can expose subtle bugs that can be hard to
83 diagnose when debugging regular (complex) applications.
91 <sect1 id="testing-what">
92 <title>What to test for?</title>
94 The first thing to test for is the documented behavior of APIs
95 and such as CreateFile. For instance one can create a file using a
96 long pathname, check that the behavior is correct when the file
97 already exists, try to open the file using the corresponding short
98 pathname, convert the filename to Unicode and try to open it using
99 CreateFileW, and all other things which are documented and that
100 applications rely on.
103 While the testing framework is not specifically geared towards this
104 type of tests, it is also possible to test the behavior of Windows
105 messages. To do so, create a window, preferably a hidden one so that
106 it does not steal the focus when running the tests, and send messages
107 to that window or to controls in that window. Then, in the message
108 procedure, check that you receive the expected messages and with the
112 For instance you could create an edit control and use WM_SETTEXT to
113 set its contents, possibly check length restrictions, and verify the
114 results using WM_GETTEXT. Similarly one could create a listbox and
115 check the effect of LB_DELETESTRING on the list's number of items,
116 selected items list, highlighted item, etc.
119 However, undocumented behavior should not be tested for unless there
120 is an application that relies on this behavior, and in that case the
121 test should mention that application, or unless one can strongly
122 expect applications to rely on this behavior, typically APIs that
123 return the required buffer size when the buffer pointer is NULL.
128 <sect1 id="testing-wine">
129 <title>Running the tests in Wine</title>
131 The simplest way to run the tests in Wine is to type 'make test' in
132 the Wine sources top level directory. This will run all the Wine
136 The tests for a specific Wine library are located in a 'tests'
137 directory in that library's directory. Each test is contained in a
138 file (e.g. <filename>dlls/kernel/tests/thread.c</>). Each
139 file itself contains many checks concerning one or more related APIs.
142 So to run all the tests related to a given Wine library, go to the
143 corresponding 'tests' directory and type 'make test'. This will
144 compile the tests, run them, and create an '<replaceable>xxx</>.ok'
145 file for each test that passes successfully. And if you only want to
146 run the tests contained in the <filename>thread.c</> file of the
147 kernel library, you would do:
149 <prompt>$ </>cd dlls/kernel/tests
150 <prompt>$ </>make thread.ok
154 Note that if the test has already been run and is up to date (i.e. if
155 neither the kernel library nor the <filename>thread.c</> file has
156 changed since the <filename>thread.ok</> file was created), then make
157 will say so. To force the test to be re-run, delete the
158 <filename>thread.ok</> file, and run the make command again.
161 You can also run tests manually using a command similar to the
164 <prompt>$ </>../../../tools/runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c
165 <prompt>$ </>../../../tools/runtest -p kernel32_test.exe.so thread.c
166 thread.c: 86 tests executed, 5 marked as todo, 0 failures.
168 The '-P wine' options defines the platform that is currently being
169 tested. Remove the '-q' option if you want the testing framework
170 to report statistics about the number of successful and failed tests.
171 Run <command>runtest -h</> for more details.
176 <sect1 id="cross-compiling-tests">
177 <title>Cross-compiling the tests with MinGW</title>
179 <title>Setup of the MinGW cross-compiling environment</title>
181 The most daunting problem while trying to cross-compile the Wine
182 tests is the setup of the MinGW cross-compiling environment. Here
183 are some instructions for different Linux distributions and *BSD
184 systems to help with this problem.
187 <title>Debian GNU/Linux</title>
189 On Debian all you need to do is type <command>apt-get install
194 <title>Red Hat Linux like rpm systems</title>
196 This includes Fedora Core, Red Hat Enterprise Linux, Mandrake,
197 most probably SuSE Linux too, etc. But this list isn't exhaustive;
198 the following steps should probably work on any rpm based system.
202 Download the mingw-binutils and mingw-gcc srpm's from
203 <ulink url="http://ftp.redhat.com/pub/contrib/libc6/SRPMS/">
204 http://ftp.redhat.com/pub/contrib/libc6/SRPMS/</ulink>.
207 Download the mingw srpm from
208 <ulink url="http://ftp.redhat.com/pub/contrib/libc6/noarch/SRPMS/">
209 http://ftp.redhat.com/pub/contrib/libc6/noarch/SRPMS/</ulink>.
212 Build the mingw and mingw-binutils rpm's (<command>rpmbuild
213 --rebuild $SRPM</>) and install them.
216 After the above step you can build the mingw-gcc rpm too.
224 The *BSD systems have in their ports collection a port for the
225 MinGW cross-compiling environment. Please see the documentation
226 of your system about how to build and install a port.
231 <title>Compiling the tests</title>
233 Having the cross-compiling environment set up the generation of the
234 Windows executables is easy by using the Wine build system.
237 If you had already run <command>configure</>, then delete
238 <filename>config.cache</> and re-run <command>configure</>.
239 You can then run <command>make crosstest</>. To sum up:
241 <prompt>$ </><userinput>rm config.cache</>
242 <prompt>$ </><userinput>./configure</>
243 <prompt>$ </><userinput>make crosstest</>
250 <sect1 id="testing-windows">
251 <title>Building and running the tests on Windows</title>
253 <title>Using pre-compiled binaries</title>
255 Unfortunately there are no pre-compiled binaries yet. However if
256 send an email to the Wine development list you can probably get
257 someone to send them to you, and maybe motivate some kind soul to
258 put in place a mechanism for publishing such binaries on a regular
263 <title>With Visual C++</title>
265 Visual Studio 6 users:
266 - MSVC headers may not work well, try with Wine headers
267 - Ensure that you have the "processor pack" from
268 <ulink url="http://msdn.microsoft.com/vstudio/downloads/tools/ppack/default.aspx">http://msdn.microsoft.com/vstudio/downloads/tools/ppack/default.aspx</>
269 as well as the latest service packs. The processor pack fixes <emphasis>"error C2520: conversion from unsigned
270 __int64 to double not implemented, use signed __int64"</>
277 Run msvcmaker to generate Visual C++ project files for the tests.
278 'msvcmaker' is a perl script so you may be able to run it on
281 <prompt>$ </>./tools/winapi/msvcmaker --no-wine
285 If the previous steps were done on your Linux development
286 machine, make the Wine sources accessible to the Windows machine
287 on which you are going to compile them. Typically you would do
288 this using Samba but copying them altogether would work too.
291 On the Windows machine, open the <filename>winetest.dsw</>
292 workspace. This will load each test's project. For each test there
293 are two configurations: one compiles the test with the Wine
294 headers, and the other uses the Visual C++ headers. Some tests
295 will compile fine with the former, but most will require the
299 Open the <menuchoice><guimenu>Build</> <guimenu>Batch
300 build...</></> menu and select the tests and build configurations
301 you want to build. Then click on <guibutton>Build</>.
304 To run a specific test from Visual C++, go to
305 <menuchoice><guimenu>Project</> <guimenu>Settings...</></>. There
306 select that test's project and build configuration and go to the
307 <guilabel>Debug</> tab. There type the name of the specific test
308 to run (e.g. 'thread') in the <guilabel>Program arguments</>
309 field. Validate your change by clicking on <guibutton>Ok</> and
310 start the test by clicking the red exclamation mark (or hitting
311 'F5' or any other usual method).
314 You can also run the tests from the command line. You will find
315 them in either <filename>Output\Win32_Wine_Headers</> or
316 <filename>Output\Win32_MSVC_Headers</> depending on the build
317 method. So to run the kernel 'path' tests you would do:
319 <prompt>C:\></>cd dlls\kernel\tests\Output\Win32_MSVC_Headers
320 <prompt>C:\dlls\kernel\tests\Output\Win32_MSVC_Headers></>kernel32_test path
326 <title>With MinGW</title>
328 This needs to be documented. The best may be to ask on the Wine
329 development mailing list and update this documentation with the
330 result of your inquiry.
336 <sect1 id="testing-test">
337 <title>Inside a test</title>
340 When writing new checks you can either modify an existing test file or
341 add a new one. If your tests are related to the tests performed by an
342 existing file, then add them to that file. Otherwise create a new .c
343 file in the tests directory and add that file to the
344 <varname>CTESTS</> variable in <filename>Makefile.in</>.
347 A new test file will look something like the following:
349 #include <wine/test.h>
350 #include <winbase.h>
352 /* Maybe auxiliary functions and definitions here */
356 /* Write your checks there or put them in functions you will call from
363 The test's entry point is the START_TEST section. This is where
364 execution will start. You can put all your tests in that section but
365 it may be better to split related checks in functions you will call
366 from the START_TEST section. The parameter to START_TEST must match
367 the name of the C file. So in the above example the C file would be
368 called <filename>paths.c</>.
371 Tests should start by including the <filename>wine/test.h</> header.
372 This header will provide you access to all the testing framework
373 functions. You can then include the windows header you need, but make
374 sure to not include any Unix or Wine specific header: tests must
378 You can use <function>trace</> to print informational messages. Note
379 that these messages will only be printed if 'runtest -v' is being used.
381 trace("testing GlobalAddAtomA");
386 Then just call functions and use <function>ok</> to make sure that
387 they behaved as expected:
389 ATOM atom = GlobalAddAtomA( "foobar" );
390 ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar" );
391 ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR" );
393 The first parameter of <function>ok</> is an expression which must
394 evaluate to true if the test was successful. The next parameter is a
395 printf-compatible format string which is displayed in case the test
396 failed, and the following optional parameters depend on the format
401 <sect1 id="testing-error-messages">
402 <title>Writing good error messages</title>
404 The message that is printed when a test fails is
405 <emphasis>extremely</> important.
408 Someone will take your test, run it on a Windows platform that
409 you don't have access to, and discover that it fails. They will then
410 post an email with the output of the test, and in particular your
411 error message. Someone, maybe you, will then have to figure out from
412 this error message why the test failed.
415 If the error message contains all the relevant information that will
416 be easy. If not, then it will require modifying the test, finding
417 someone to compile it on Windows, sending the modified version to the
418 original tester and waiting for his reply. In other words, it will
422 So how do you write a good error message? Let's start with an example
423 of a bad error message:
425 ok(GetThreadPriorityBoost(curthread,&disabled)!=0,
426 "GetThreadPriorityBoost Failed");
430 thread.c:123: Test failed: GetThreadPriorityBoost Failed
434 Did you notice how the error message provides no information about
435 why the test failed? We already know from the line number exactly
436 which test failed. In fact the error message gives strictly no
437 information that cannot already be obtained by reading the code. In
438 other words it provides no more information than an empty string!
441 Let's look at how to rewrite it:
445 rc=GetThreadPriorityBoost(curthread,&disabled);
446 ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d",
447 rc,GetLastError(),disabled);
451 thread.c:123: Test failed: rc=0 error=120 disabled=0
455 When receiving such a message, one would check the source, see that
456 it's a call to GetThreadPriorityBoost, that the test failed not
457 because the API returned the wrong value, but because it returned an
458 error code. Furthermore we see that GetLastError() returned 120 which
459 winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of
460 the problem is obvious: this Windows platform (here Windows 98) does
461 not support this API and thus the test must be modified to detect
462 such a condition and skip the test.
465 So a good error message should provide all the information which
466 cannot be obtained by reading the source, typically the function
467 return value, error codes, and any function output parameter. Even if
468 more information is needed to fully understand a problem,
469 systematically providing the above is easy and will help cut down the
470 number of iterations required to get to a resolution.
473 It may also be a good idea to dump items that may be hard to retrieve
474 from the source, like the expected value in a test if it is the
475 result of an earlier computation, or comes from a large array of test
476 values (e.g. index 112 of _pTestStrA in vartest.c). In that respect,
477 for some tests you may want to define a macro such as the following:
479 #define eq(received, expected, label, type) \
480 ok((received) == (expected), "%s: got " type " instead of " type, (label),(received),(expected))
484 eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );
490 <sect1 id="testing-platforms">
491 <title>Handling platform issues</title>
493 Some checks may be written before they pass successfully in Wine.
494 Without some mechanism, such checks would potentially generate
495 hundred of known failures for months each time the tests are being run.
496 This would make it hard to detect new failures caused by a regression.
497 or to detect that a patch fixed a long standing issue.
500 Thus the Wine testing framework has the concept of platforms and
501 groups of checks can be declared as expected to fail on some of them.
502 In the most common case, one would declare a group of tests as
503 expected to fail in Wine. To do so, use the following construct:
506 SetLastError( 0xdeadbeef );
507 ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0" );
510 On Windows the above check would be performed normally, but on Wine it
511 would be expected to fail, and not cause the failure of the whole
512 test. However. If that check were to succeed in Wine, it would
513 cause the test to fail, thus making it easy to detect when something
514 has changed that fixes a bug. Also note that todo checks are accounted
515 separately from regular checks so that the testing statistics remain
516 meaningful. Finally, note that todo sections can be nested so that if
517 a test only fails on the cygwin and reactos platforms, one would
526 <!-- FIXME: Would we really have platforms such as reactos, cygwin, freebsd & co? -->
527 But specific platforms should not be nested inside a todo_wine section
528 since that would be redundant.
531 When writing tests you will also encounter differences between Windows
532 9x and Windows NT platforms. Such differences should be treated
533 differently from the platform issues mentioned above. In particular
534 you should remember that the goal of Wine is not to be a clone of any
535 specific Windows version but to run Windows applications on Unix.
538 So, if an API returns a different error code on Windows 9x and
539 Windows NT, your check should just verify that Wine returns one or
542 ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...);
546 If an API is only present on some Windows platforms, then use
547 LoadLibrary and GetProcAddress to check if it is implemented and
548 invoke it. Remember, tests must run on all Windows platforms.
549 Similarly, conformance tests should nor try to correlate the Windows
550 version returned by GetVersion with whether given APIs are
551 implemented or not. Again, the goal of Wine is to run Windows
552 applications (which do not do such checks), and not be a clone of a
553 specific Windows version.
556 FIXME: What about checks that cause the process to crash due to a bug?
561 <!-- FIXME: Strategies for testing threads, testing network stuff,
562 file handling, eq macro... -->
566 <!-- Keep this comment at the end of the file
569 sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")