Updated regression testing documentation.

[wine] / documentation / cvs-regression.sgml

  <chapter id="cvs-regression">
    <title>How to do regression testing using CVS</title>

    <para>
      written by Gerard Patel
    </para>
    <para>
      (Extracted from <filename>wine/documentation/bugreports</filename>)
    </para>

    <para>
      A problem that can happen sometimes is 'it used to work
      before, now it doesn't anymore...'. Here is a step by step
      procedure to try to pinpoint when the problem occured. This is
      <emphasis>NOT</emphasis> for casual users.
    </para>

    <orderedlist>
      <listitem>
        <para>
          Get the <quote>full CVS</quote> archive from winehq. This archive is
          the CVS tree but with the tags controlling the versioning
          system. It's a big file (> 40 meg) with a name like
          wine-cvsdirs-&lt;last update date> (it's more than 100mb
          when uncompressed, you can't very well do this with
          small, old computers or slow Internet connections).
        </para>
      </listitem>
      <listitem>
        <para>
          untar it into a repository directory:
          <screen>
cd /home/gerard
tar -zxf cvs-dirs-2003-01-15.tar.gz
mv wine repository
          </screen>
        </para>
      </listitem>
      <listitem>
        <para>
          extract a new destination directory. This directory must
          not be in a subdirectory of the repository else
          <command>cvs</command> will think it's part of the
          repository and deny you an extraction in the repository:
          <screen>
cd /home/gerard
mv wine wine_current (-> this protects your current wine sandbox, if any)
export CVSROOT=/home/gerard/repository
cd /home/gerard
cvs -d $CVSROOT checkout wine
          </screen>
        </para>
        <para>
          Note that it's not possible to do a checkout at a given
          date; you always do the checkout for the last date where
          the wine-cvsdirs-xxx snapshot was generated.
        </para>
        <para>
          Note also that it is possible to do all this with a direct
          CVS connection, of course. The full CVS file method is less
          painful for the winehq CVS server and probably a bit faster
          if you don't have a very good net connection.
        </para>
        <note>
          <para>
            If you use CVS directly from the winehq.com server, do not
            forget to add to your <filename>.cvsrc</filename> file:
          </para>
          <screen>
cvs -z 3
update -dPA
diff -u
          </screen>
         </note>
      </listitem>
      <listitem>
        <para>
          you will have now in the <filename>~/wine</filename>
          directory an image of the CVS tree, on the client side.
          Now update this image to the date you want:
          <screen>
cd /home/gerard/wine
cvs -d $CVSROOT update -D "2002-06-01 CST"   
          </screen>
        </para>
        <para>
          The date format is <literal>YYYY-MM-DD HH:MM:SS</literal>.
          Using the CST date format ensure that you will be able to
          extract patches in a way that will be compatible with the
          wine-cvs archive
          <ulink url="http://www.winehq.com/hypermail/wine-cvs">
          http://www.winehq.com/hypermail/wine-cvs</ulink>
        </para>
        <para>
          Many messages will inform you that more recent files have
          been deleted to set back the client cvs tree to the date
          you asked, for example:
          <screen>
cvs update: tsx11/ts_xf86dga2.c is no longer in the repository
          </screen>
        </para>
        <para>
          <command>cvs update</command> is not limited to upgrade to
          a <emphasis>newer</emphasis> version as I have believed for
          far too long :-(
        </para>
      </listitem>
      <listitem>
        <para>
          Now proceed as for a normal update:
        </para>
        <screen>
./configure
make depend && make
        </screen>
        <para>
          If any non-programmer reads this, the fastest method to get
          at the point where the problem occured is to use a binary
          search, that is, if the problem occured in 1999, start at
          mid-year, then is the problem is already here, back to 1st
          April, if not, to 1st October, and so on.
        </para>
        <para>
          If you have lot of hard disk free space (a full compile currently
          takes 400 Mb), copy the oldest known working version before
          updating it, it will save time if you need to go back. (it's
          better to <command>make distclean</command> before going back in
          time, so you have to make everything if you don't backup the older
          version)
        </para>
        <para>  
          When you have found the day where the problem happened, continue
          the search using the wine-cvs archive (sorted by date) and a
          more precise cvs update including hour, minute, second :
          <screen>
cvs -d $CVSROOT update -D "2002-06-01 15:17:25 CST"
          </screen>
          This will allow you to find easily the exact patch that did it.
        </para>
      </listitem>
      <listitem>
        <para>
          If you find the patch that is the cause of the problem, you have
          almost won; report about it to
          <ulink url="http://bugs.winehq.com/">Wine Bugzilla</ulink>
          or susbscribe to wine-devel and post it there. There is a chance
          that the author
          will jump in to suggest a fix; or there is always the possibility
          to look hard at the patch until it is coerced to reveal where is
          the bug :-) 
        </para>
      </listitem>
    </orderedlist>
  </chapter>

  <!-- Keep this comment at the end of the file
  Local variables:
  mode: sgml
  sgml-parent-document:("wine-doc.sgml" "set" "book" "part" "chapter" "")
  End:
  -->