
Test-suite automation information
---------------------------------

 All of our test cases now working under automation. 
 We have two comprehensive testing frameworks: the "regcheck"
 framework, and the "outcheck" framework. The former looks at the final
 values of registers after a test case is run, and the latter looks at the
 output that vmips prints out when a test case is run. Between these two,
 we have covered almost all the test cases *and* achieved what I believe
 is a fairly easy-to-use test framework. If you want to run the tests, the
 current way to do it is:

 1. install a copy of dejagnu (any old copy >=1.3 should do)
 2. configure and make vmips in the manner to which you have become accustomed
 3. "make check" at toplevel (no need to call "runtest" explicitly)
 4. Look at the end of the output, you should have something like 30 
    expected passes and 1 expected failure.

 To write a new test case, first decide whether it is easier to have your
 test case print out something or to look at the "haltdumpcpu" option's
 output to verify it. This will tell you whether it should be an "outcheck" 
 test (if it prints out something) or a "regcheck" case (if you look at the
 register values).  Then write up a ".par" file for the test case (see
 examples in the test_code directory). Then move the .par file and the .rom to
 the appropriate subdirectory (vmips.regcheck or vmips.outcheck). Now you can
 try to make sure that it passes (or fails, as the case may be), by running
 "runtest --tool vmips" on it.


 What's left to do in the testsuite
 ----------------------------------

 - adding more tests so that the testsuite adequately tests the program
   (See the "additional testcases needed" section, below)
 - making it easier to figure out what went wrong when a test case fails
   (currently you have to re-run everything with "runtest --debug --verbose
    --tool vmips" and grovel through hundreds of pages of spam)
 - documentation


 Troubleshooting
 ---------------

 You can get "endian_flag is not set" errors if you call "runtest" without
 making site.exp first. (To make site.exp, just say "make site.exp"). 


 Additional testcases needed
 ---------------------------

  5/Jan/2002
  I have run gcov over the testsuite and am compiling a list
  of some things the testsuite doesn't check (not a complete
  list). Certainly the instructions, and probably the exceptions, we
  need to write test cases for.  There are some options that would be
  easy to check.

  Vmipstool
  The debugger interface
  Exception stuff:
   Exception prioritizing
   Non-boot-time exception vectors
   All the TLB exceptions
   Unimplemented Coprocessor exceptions
   Reserved Instruction exceptions
   Exceptions due to PC address translation
   Address error exceptions due to unaligned accesses
  /* Instructions:
   *  add when the answer doesn't overflow
   *  blez, bgtz, slti, lbu, lhu, sb, sh, jalr, divu, subu, or, nor
   *  tlbp when the tlb entry you want isn't there
   *  mult when you don't HAVE_LONG_LONG
   * 8/Jan/2002: these are done.
   */
  Interrupts
  Options:
   Instdump, dumpcpu, dumpcp0, haltdumpcp0, instcounts, bootmsg
   Non-existing romfile
   tilde_expand where it's not your home directory
   Behavior w/ bogus args and/or unknown options
  Address translations in KSEG0 or KSEG2 or KUSEG or User mode


CVS: $Id: README,v 1.23 2002/06/07 23:50:57 brg Exp $
 
