2 <title>Writing Conformance tests</title>
5 Written by &name-francois-gouget; <email>&email-francois-gouget;</email>
8 Note: This part of the documentation is still very much a work in
9 progress and is in no way complete.
12 <sect1 id="testing-intro">
13 <title>Introduction</title>
15 With more The Windows API follows no standard, it is itself a defacto
16 standard, and deviations from that standard, even small ones, often
17 cause applications to crash or misbehave in some way. Furthermore
18 a conformance test suite is the most accurate (if not necessarily
19 the most complete) form of API documentation and can be used to
20 supplement the Windows API documentation.
23 Writing a conformance test suite for more than 10000 APIs is no small
24 undertaking. Fortunately it can prove very useful to the development
25 of Wine way before it is complete.
29 The conformance test suite must run on Windows. This is
30 necessary to provide a reasonable way to verify its accuracy.
31 Furthermore the tests must pass successfully on all Windows
32 platforms (tests not relevant to a given platform should be
36 A consequence of this is that the test suite will provide a
37 great way to detect variations in the API between different
38 Windows versions. For instance, this can provide insights
39 into the differences between the, often undocumented, Win9x and
43 However, one must remember that the goal of Wine is to run
44 Windows applications on Linux, not to be a clone of any specific
45 Windows version. So such variations must only be tested for when
46 relevant to that goal.
51 Writing conformance tests is also an easy way to discover
52 bugs in Wine. Of course, before fixing the bugs discovered in
53 this way, one must first make sure that the new tests do pass
54 successfully on at least one Windows 9x and one Windows NT
58 Bugs discovered this way should also be easier to fix. Unlike
59 some mysterious application crashes, when a conformance test
60 fails, the expected behavior and APIs tested for are known thus
61 greatly simplifying the diagnosis.
66 To detect regressions. Simply running the test suite regularly
67 in Wine turns it into a great tool to detect regressions.
68 When a test fails, one immediately knows what was the expected
69 behavior and which APIs are involved. Thus regressions caught
70 this way should be detected earlier, because it is easy to run
71 all tests on a regular basis, and easier to fix because of the
72 reduced diagnosis work.
77 Tests written in advance of the Wine development (possibly even
78 by non Wine developers) can also simplify the work of the
79 future implementer by making it easier for him to check the
80 correctness of his code.
85 Conformance tests will also come in handy when testing Wine on
86 new (or not as widely used) architectures such as FreeBSD,
87 Solaris x86 or even non-x86 systems. Even when the port does
88 not involve any significant change in the thread management,
89 exception handling or other low-level aspects of Wine, new
90 architectures can expose subtle bugs that can be hard to
91 diagnose when debugging regular (complex) applications.
99 <sect1 id="testing-what">
100 <title>What to test for?</title>
102 The first thing to test for is the documented behavior of APIs
103 and such as CreateFile. For instance one can create a file using a
104 long pathname, check that the behavior is correct when the file
105 already exists, try to open the file using the corresponding short
106 pathname, convert the filename to Unicode and try to open it using
107 CreateFileW, and all other things which are documented and that
108 applications rely on.
111 While the testing framework is not specifically geared towards this
112 type of tests, it is also possible to test the behavior of Windows
113 messages. To do so, create a window, preferably a hidden one so that
114 it does not steal the focus when running the tests, and send messages
115 to that window or to controls in that window. Then, in the message
116 procedure, check that you receive the expected messages and with the
120 For instance you could create an edit control and use WM_SETTEXT to
121 set its contents, possibly check length restrictions, and verify the
122 results using WM_GETTEXT. Similarly one could create a listbox and
123 check the effect of LB_DELETESTRING on the list's number of items,
124 selected items list, highlighted item, etc.
127 However, undocumented behavior should not be tested for unless there
128 is an application that relies on this behavior, and in that case the
129 test should mention that application, or unless one can strongly
130 expect applications to rely on this behavior, typically APIs that
131 return the required buffer size when the buffer pointer is NULL.
136 <sect1 id="testing-wine">
137 <title>Running the tests in Wine</title>
139 The simplest way to run the tests in Wine is to type 'make test' in
140 the Wine sources top level directory. This will run all the Wine
144 The tests for a specific Wine library are located in a 'tests'
145 directory in that library's directory. Each test is contained in a
146 file (e.g. <filename>dlls/kernel/tests/thread.c</>). Each
147 file itself contains many checks concerning one or more related APIs.
150 So to run all the tests related to a given Wine library, go to the
151 corresponding 'tests' directory and type 'make test'. This will
152 compile the tests, run them, and create an '<replaceable>xxx</>.ok'
153 file for each test that passes successfully. And if you only want to
154 run the tests contained in the <filename>thread.c</> file of the
155 kernel library, you would do:
157 <prompt>$ </>cd dlls/kernel/tests
158 <prompt>$ </>make thread.ok
162 Note that if the test has already been run and is up to date (i.e. if
163 neither the kernel library nor the <filename>thread.c</> file has
164 changed since the <filename>thread.ok</> file was created), then make
165 will say so. To force the test to be re-run, delete the
166 <filename>thread.ok</> file, and run the make command again.
169 You can also run tests manually using a command similar to the
172 <prompt>$ </>../../../tools/runtest -q -M kernel32.dll -p kernel32_test.exe.so thread.c
173 <prompt>$ </>../../../tools/runtest -p kernel32_test.exe.so thread.c
174 thread.c: 86 tests executed, 5 marked as todo, 0 failures.
176 The '-P wine' options defines the platform that is currently being
177 tested. Remove the '-q' option if you want the testing framework
178 to report statistics about the number of successful and failed tests.
179 Run <command>runtest -h</> for more details.
184 <sect1 id="testing-windows">
185 <title>Building and running the tests on Windows</title>
187 <title>Using pre-compiled binaries</title>
189 Unfortunately there are no pre-compiled binaries yet. However if
190 send an email to the Wine development list you can probably get
191 someone to send them to you, and maybe motivate some kind soul to
192 put in place a mechanism for publishing such binaries on a regular
197 <title>With Visual C++</title>
203 Run msvcmaker to generate Visual C++ project files for the tests.
204 'msvcmaker' is a perl script so you may be able to run it on
207 <prompt>$ </>./tools/winapi/msvcmaker --no-wine
211 If the previous steps were done on your Linux development
212 machine, make the Wine sources accessible to the Windows machine
213 on which you are going to compile them. Typically you would do
214 this using Samba but copying them altogether would work too.
217 On the Windows machine, open the <filename>winetest.dsw</>
218 workspace. This will load each test's project. For each test there
219 are two configurations: one compiles the test with the Wine
220 headers, and the other uses the Visual C++ headers. Some tests
221 will compile fine with the former, but most will require the
225 Open the <menuchoice><guimenu>Build</> <guimenu>Batch
226 build...</></> menu and select the tests and build configurations
227 you want to build. Then click on <guibutton>Build</>.
230 To run a specific test from Visual C++, go to
231 <menuchoice><guimenu>Project</> <guimenu>Settings...</></>. There
232 select that test's project and build configuration and go to the
233 <guilabel>Debug</> tab. There type the name of the specific test
234 to run (e.g. 'thread') in the <guilabel>Program arguments</>
235 field. Validate your change by clicking on <guibutton>Ok</> and
236 start the test by clicking the red exclamation mark (or hitting
237 'F5' or any other usual method).
240 You can also run the tests from the command line. You will find
241 them in either <filename>Output\Win32_Wine_Headers</> or
242 <filename>Output\Win32_MSVC_Headers</> depending on the build
243 method. So to run the kernel 'path' tests you would do:
245 <prompt>C:\></>cd dlls\kernel\tests\Output\Win32_MSVC_Headers
246 <prompt>C:\dlls\kernel\tests\Output\Win32_MSVC_Headers></>kernel32_test thread
252 <title>With MinGW</title>
254 This needs to be documented. The best may be to ask on the Wine
255 development mailing list and update this documentation with the
256 result of your inquiry.
260 <title>Cross compiling with MinGW on Linux</title>
262 Here is how to generate Windows executables for the tests straight
263 from the comfort of Linux.
267 First you need to get the MinGW cross-compiler. On Debian all
268 you need to do is type <command>apt-get install mingw32</>.
271 If you had already run <command>configure</>, then delete
272 <filename>config.cache</> and re-run <command>configure</>.
273 You can then run <command>make crosstest</>. To sum up:
275 <prompt>$ </><userinput>rm config.cache</>
276 <prompt>$ </><userinput>./configure</>
277 <prompt>$ </><userinput>make crosstest</>
281 If you get an error when compiling <filename>winsock.h</> then
282 you probably need to apply the following patch:
283 <ulink url="http://www.winehq.com/hypermail/wine-patches/2002/12/0157.html">http://www.winehq.com/hypermail/wine-patches/2002/12/0157.html</>
290 <sect1 id="testing-test">
291 <title>Inside a test</title>
294 When writing new checks you can either modify an existing test file or
295 add a new one. If your tests are related to the tests performed by an
296 existing file, then add them to that file. Otherwise create a new .c
297 file in the tests directory and add that file to the
298 <varname>CTESTS</> variable in <filename>Makefile.in</>.
301 A new test file will look something like the following:
303 #include <wine/test.h>
304 #include <winbase.h>
306 /* Maybe auxiliary functions and definitions here */
310 /* Write your checks there or put them in functions you will call from
317 The test's entry point is the START_TEST section. This is where
318 execution will start. You can put all your tests in that section but
319 it may be better to split related checks in functions you will call
320 from the START_TEST section. The parameter to START_TEST must match
321 the name of the C file. So in the above example the C file would be
322 called <filename>paths.c</>.
325 Tests should start by including the <filename>wine/test.h</> header.
326 This header will provide you access to all the testing framework
327 functions. You can then include the windows header you need, but make
328 sure to not include any Unix or Wine specific header: tests must
332 You can use <function>trace</> to print informational messages. Note
333 that these messages will only be printed if 'runtest -v' is being used.
335 trace("testing GlobalAddAtomA");
340 Then just call functions and use <function>ok</> to make sure that
341 they behaved as expected:
343 ATOM atom = GlobalAddAtomA( "foobar" );
344 ok( GlobalFindAtomA( "foobar" ) == atom, "could not find atom foobar" );
345 ok( GlobalFindAtomA( "FOOBAR" ) == atom, "could not find atom FOOBAR" );
347 The first parameter of <function>ok</> is an expression which must
348 evaluate to true if the test was successful. The next parameter is a
349 printf-compatible format string which is displayed in case the test
350 failed, and the following optional parameters depend on the format
355 <sect1 id="testing-error-messages">
356 <title>Writing good error messages</title>
358 The message that is printed when a test fails is
359 <emphasis>extremely</> important.
362 Someone will take your test, run it on a Windows platform that
363 you don't have access to, and discover that it fails. They will then
364 post an email with the output of the test, and in particular your
365 error message. Someone, maybe you, will then have to figure out from
366 this error message why the test failed.
369 If the error message contains all the relevant information that will
370 be easy. If not, then it will require modifying the test, finding
371 someone to compile it on Windows, sending the modified version to the
372 original tester and waiting for his reply. In other words, it will
376 So how do you write a good error message? Let's start with an example
377 of a bad error message:
379 ok(GetThreadPriorityBoost(curthread,&disabled)!=0,
380 "GetThreadPriorityBoost Failed");
384 thread.c:123: Test failed: GetThreadPriorityBoost Failed
388 Did you notice how the error message provides no information about
389 why the test failed? We already know from the line number exactly
390 which test failed. In fact the error message gives strictly no
391 information that cannot already be obtained by reading the code. In
392 other words it provides no more information than an empty string!
395 Let's look at how to rewrite it:
399 rc=GetThreadPriorityBoost(curthread,&disabled);
400 ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d",
401 rc,GetLastError(),disabled);
405 thread.c:123: Test failed: rc=0 error=120 disabled=0
409 When receiving such a message, one would check the source, see that
410 it's a call to GetThreadPriorityBoost, that the test failed not
411 because the API returned the wrong value, but because it returned an
412 error code. Furthermore we see that GetLastError() returned 120 which
413 winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of
414 the problem is obvious: this Windows platform (here Windows 98) does
415 not support this API and thus the test must be modified to detect
416 such a condition and skip the test.
419 So a good error message should provide all the information which
420 cannot be obtained by reading the source, typically the function
421 return value, error codes, and any function output parameter. Even if
422 more information is needed to fully understand a problem,
423 systematically providing the above is easy and will help cut down the
424 number of iterations required to get to a resolution.
427 It may also be a good idea to dump items that may be hard to retrieve
428 from the source, like the expected value in a test if it is the
429 result of an earlier computation, or comes from a large array of test
430 values (e.g. index 112 of _pTestStrA in vartest.c). In that respect,
431 for some tests you may want to define a macro such as the following:
433 #define eq(received, expected, label, type) \
434 ok((received) == (expected), "%s: got " type " instead of " type, (label),(received),(expected))
438 eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );
444 <sect1 id="testing-platforms">
445 <title>Handling platform issues</title>
447 Some checks may be written before they pass successfully in Wine.
448 Without some mechanism, such checks would potentially generate
449 hundred of known failures for months each time the tests are being run.
450 This would make it hard to detect new failures caused by a regression.
451 or to detect that a patch fixed a long standing issue.
454 Thus the Wine testing framework has the concept of platforms and
455 groups of checks can be declared as expected to fail on some of them.
456 In the most common case, one would declare a group of tests as
457 expected to fail in Wine. To do so, use the following construct:
460 SetLastError( 0xdeadbeef );
461 ok( GlobalAddAtomA(0) == 0 && GetLastError() == 0xdeadbeef, "failed to add atom 0" );
464 On Windows the above check would be performed normally, but on Wine it
465 would be expected to fail, and not cause the failure of the whole
466 test. However. If that check were to succeed in Wine, it would
467 cause the test to fail, thus making it easy to detect when something
468 has changed that fixes a bug. Also note that todo checks are accounted
469 separately from regular checks so that the testing statistics remain
470 meaningful. Finally, note that todo sections can be nested so that if
471 a test only fails on the cygwin and reactos platforms, one would
480 <!-- FIXME: Would we really have platforms such as reactos, cygwin, freebsd & co? -->
481 But specific platforms should not be nested inside a todo_wine section
482 since that would be redundant.
485 When writing tests you will also encounter differences between Windows
486 9x and Windows NT platforms. Such differences should be treated
487 differently from the platform issues mentioned above. In particular
488 you should remember that the goal of Wine is not to be a clone of any
489 specific Windows version but to run Windows applications on Unix.
492 So, if an API returns a different error code on Windows 9x and
493 Windows NT, your check should just verify that Wine returns one or
496 ok ( GetLastError() == WIN9X_ERROR || GetLastError() == NT_ERROR, ...);
500 If an API is only present on some Windows platforms, then use
501 LoadLibrary and GetProcAddress to check if it is implemented and
502 invoke it. Remember, tests must run on all Windows platforms.
503 Similarly, conformance tests should nor try to correlate the Windows
504 version returned by GetVersion with whether given APIs are
505 implemented or not. Again, the goal of Wine is to run Windows
506 applications (which do not do such checks), and not be a clone of a
507 specific Windows version.
510 FIXME: What about checks that cause the process to crash due to a bug?
515 <!-- FIXME: Strategies for testing threads, testing network stuff,
516 file handling, eq macro... -->
520 <!-- Keep this comment at the end of the file
523 sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")
projects / wine / blob