tcltest Tk Applications

I think I might have finally hit on a formula for testing my Tk
applications.  And I would be interested in comments on this approach.

I've used tcltest for exercising procedures and objects.  But I had
never really figured out how to use it to functionally test a whole
application.  If you [source main.tcl] in a test, it will load the
code, build the GUI and start the application.  But shutting down the
application is more difficult.  There is typically a lot of state left
in the interpreter, and if you destroy the Tk application (by wm
deleting "."), then it gets really hard to start over again.

I considered a GUI-driving test program like android or tkreplay, but
I don't necessarily want to drive the GUI widgets.  I want to exercise
the application's API, which is called by the widgets' -command
options.  Instead of "black box" testing the GUI, I want to "white
box" test the application logic, ping and poking at the internal
state of the application.

Suddenly, it is obvious to me that the answer is to use slave
interpreters.  Each test creates a slave interpreter and uses it to
source the mainline code.  The slave loads Tk and other extensions,
builds the GUI, and initializes the application.  The test code can
then use [$slave eval] to command and probe the state of the
application.  Cleanup is done by simply deleting the slave
interpreter.

For example, for the TNT application, I can create a standard -setup
and -cleanup script like this:

set mainProgram ../tnt.tcl

set startTNT {
set app [interp create]
$app alias exit return "application called 'exit'"
$app eval source $mainProgram
update
}

set stopTNT {
catch {interp delete $app}
}


Now I can write tests which start the application, twiddle its
innards, and clean it up, no matter what state it is left in.

test tnt-1.1 {run application} -body {
# if the application started, it set this global variable
$app eval info exists TNT_VERSION
} -setup $startTNT -cleanup $stopTNT -result {1}

test tnt-1.2 {normal exit, as called from File->Exit menu} -body {
$app eval _exit
} -setup $startTNT -cleanup $stopTNT -result {application called
'exit'}

There are some "gotchas."  My setup script should alias [exit] to
prevent the application from aborting the tests.  It must set argv0,
argv, and argc.  And I also need to be aware that the environment
array (::env) is shared between all interpreters.  But are there other
things that are going to sneak up and bite me?  Or is this a good
approach for application testing?

Thanks,
Bob
--
Bob Techentin                   techentin.robert@NOSPAMmayo.edu
Mayo Foundation                                 (507) 538-5495
200 First St. SW                            FAX (507) 284-9171
Rochester MN, 55901  USA            http://www.mayo.edu/sppdg/

Bob Techentin wrote:
> I think I might have finally hit on a formula for testing my Tk
> applications.  And I would be interested in comments on this approach.
>
> I've used tcltest for exercising procedures and objects.  But I had
> never really figured out how to use it to functionally test a whole
> application.  If you [source main.tcl] in a test, it will load the
> code, build the GUI and start the application.  But shutting down the
> application is more difficult.  There is typically a lot of state left
> in the interpreter, and if you destroy the Tk application (by wm
> deleting "."), then it gets really hard to start over again.
>
> I considered a GUI-driving test program like android or tkreplay, but
> I don't necessarily want to drive the GUI widgets.  I want to exercise
> the application's API, which is called by the widgets' -command
> options.  Instead of "black box" testing the GUI, I want to "white
> box" test the application logic, ping and poking at the internal
> state of the application.
>
> Suddenly, it is obvious to me that the answer is to use slave
> interpreters.  Each test creates a slave interpreter and uses it to
> source the mainline code.  The slave loads Tk and other extensions,
> builds the GUI, and initializes the application.  The test code can
> then use [$slave eval] to command and probe the state of the
> application.  Cleanup is done by simply deleting the slave
> interpreter.
>
> For example, for the TNT application, I can create a standard -setup
> and -cleanup script like this:
>
>   set mainProgram ../tnt.tcl
>
>   set startTNT {
>       set app [interp create]
>       $app alias exit return "application called 'exit'"
>       $app eval source $mainProgram
>       update
>   }
>
>   set stopTNT {
>       catch {interp delete $app}
>   }
>
>
> Now I can write tests which start the application, twiddle its
> innards, and clean it up, no matter what state it is left in.
>
>   test tnt-1.1 {run application} -body {
>       # if the application started, it set this global variable
>       $app eval info exists TNT_VERSION
>   } -setup $startTNT -cleanup $stopTNT -result {1}
>
>   test tnt-1.2 {normal exit, as called from File->Exit menu} -body {
>       $app eval _exit
>   } -setup $startTNT -cleanup $stopTNT -result {application called
> 'exit'}
>
> There are some "gotchas."  My setup script should alias [exit] to
> prevent the application from aborting the tests.  It must set argv0,
> argv, and argc.  And I also need to be aware that the environment
> array (::env) is shared between all interpreters.  But are there other
> things that are going to sneak up and bite me?  Or is this a good
> approach for application testing?
>
> Thanks,
> Bob


Hi Bob,

I think the slave interpreter approach is a good one.  It stops your
test from polluting your test control environment and as you say allows
an easy gauranteed cleanup for each test.

We've been using an in house developed test exec for several years that
runs our test scripts in slave interpreters.

Glenn

Bob Techentin wrote:

> But are there other
> things that are going to sneak up and bite me?  Or is this a good
> approach for application testing?
>

I also once used the setup that you describe, using a slave interp. In
such a setup, tests that check stdout and stderr appear not to don't
work; tests that use -output and -errorOuput fail.

I tried to cure this by loading tcltest also into the slave interp, but
to no avail. Appended you find a small script that exercises the failure.


Greetings,

Erik Leunissen
==============

package require tcltest 2.2
namespace import -force tcltest::*
tcltest::configure -verbose {body pass error}

set setupScript {
interp create slave
# This did not help:
# slave eval [list package require tcltest 2.2]
}
set cleanupScript {
interp delete slave
}


test errorOutput-1.1 {writes to stderr from main interp} -body {
puts stderr "Do you read stderr (master)?"
} -errorOutput "Do you read stderr (master)?\n" \
-result ""

test errorOutput-1.2 {writes to stderr from slave interp} -body {
slave eval [list puts stderr "Do you read stderr (slave)?"]
} -errorOutput "Do you read stderr (slave)?\n" \
-result "" -setup $setupScript -cleanup $cleanupScript


--
leunissen@       nl | Merge the left part of these two lines into one,
e.          hccnet.   | respecting a character's position in a line.

I just found a fix for the problem that I addressed in the previous
posting. See the code inside "setupScript" in the test script below.

Greetings,

Erik Leunissen
==============


package require tcltest 2.2
namespace import -force tcltest::*
tcltest::configure -verbose {body pass error}

set setupScript {
interp create slave
# But this does help:
slave alias puts puts; # <------- FIX
}
set cleanupScript {
interp delete slave
}


test errorOutput-1.1 {writes to stderr from main interp} -body {
puts stderr "Do you read stderr (master)?"
} -errorOutput "Do you read stderr (master)?\n" \
-result ""

test errorOutput-1.2 {writes to stderr from slave interp} -body {
slave eval [list puts stderr "Do you read stderr (slave)?"]
} -errorOutput "Do you read stderr (slave)?\n" \
-result "" -setup $setupScript -cleanup $cleanupScript

--
leunissen@       nl | Merge the left part of these two lines into one,
e.          hccnet.   | respecting a character's position in a line.

Erik Leunissen wrote:

> I just found a fix for the problem that I addressed in the previous
> posting. See the code inside "setupScript" in the test script below.
>

But all access to files other than stdout, stderr and stdin will remain
a very big


PITA


The following example test "writeToFile" will cause an error because the
call to puts in [writeToFile] requires the channel $fd to exist. While
that is the case in a normal execution environment, it is not in a slave
interpreter unless you make it explicitly available using [interp
transfer]. Since the [puts] immediately follows the opening of the file,
there is no way to do that in a timely fashion when you're writing the
test script. You could of course insert it into the proc to take into
account that in the future it may be executed in a test environment that
uses slave interpreters, but ... good heavens, NO.

Are there any people who see a workable solution here?

Or should we conclude that the best thing is not to use slave
interpreters when the code that you want to test accesses files other
than the standard ones?



set proc writeToFile {txt} {
set fd [open ./testFile w]
puts $fd $txt
close $fd
}

set setupScript {
interp create slave
slave alias puts puts
}
set cleanupScript {
interp delete slave
}

test writeToFile {exercises a file access error} \
-setup $setupScript -body {
slave eval writeToFile tralala
} -result "" -returnCodes ok -cleanup $cleanupScript


--
leunissen@       nl | Merge the left part of these two lines into one,
e.          hccnet.   | respecting a character's position in a line.

GlennH wrote:
>
> We've been using an in house developed test exec for several years that
> runs our test scripts in slave interpreters.
>

Did your code access files other than stdout, stderr, stdin ?
If so, how did you solve issues regarding non-accessible channels in
slave interpreters (please see my other postings in this thread today)?

Thanks in advance for sharing your experience,

Erik Leunissen
--
leunissen@       nl | Merge the left part of these two lines into one,
e.          hccnet.   | respecting a character's position in a line.

Erik Leunissen wrote:
>
> While
> that is the case in a normal execution environment, it is not in a slave
> interpreter unless you make it explicitly available using [interp
> transfer].

Correction:

In general, channels are accessible in slave interps. But apparently it
is not in my case.


Erik Leunissen.
--
leunissen@       nl | Merge the left part of these two lines into one,
e.          hccnet.   | respecting a character's position in a line.

"Erik Leunissen" <look@the.footer.invalid> wrote
> 
previous 
below. 
>
> But all access to files other than stdout, stderr and stdin will
remain
> a very big ... PITA


Thanks, Erik, for the caution and workaround on standard channels and
slave interps.  That's one issue I hadn't considered, but could be
included in the strategy.

But I don't see that as a serious problem for many file I/O tests.
The example you provide (proc writeToFile) could be handled with
regular tcltests, without the use of slave interps.  My proposed use
of slave interps was to facilitate full Tk applications.  Stdin/stdout
could be a significant issue.  But most file I/O shouldn't be a
problem, because all the code will be run by the slave.  (It won't
have to straddle interpreters.)

Bob
--
Bob Techentin                   techentin.robert@NOSPAMmayo.edu
Mayo Foundation                                 (507) 538-5495
200 First St. SW                            FAX (507) 284-9171
Rochester MN, 55901  USA            http://www.mayo.edu/sppdg/

Erik Leunissen" <look@the.footer.invalid> wrote
>
> In general, channels are accessible in slave interps. But
> apparently it is not in my case.


I've taken a look at the tcltest code, and I see that tcltest defines
tcltest::Replace::puts which buffers stdout and stderr so that the
test suite can capture -output and -errorOutput.  So the slave interp
test setup script would need to do something similar:  define a new
puts command which would redirect all stdout/stderr output to the
master interp (probably via an alias), and use the default puts
command for everything else.

Bob
--
Bob Techentin                   techentin.robert@NOSPAMmayo.edu
Mayo Foundation                                 (507) 538-5495
200 First St. SW                            FAX (507) 284-9171
Rochester MN, 55901  USA            http://www.mayo.edu/sppdg/

Bob Techentin wrote:


So the slave interp
> test setup script would need to do something similar:  define a new
> puts command which would redirect all stdout/stderr output to the
> master interp (probably via an alias), and use the default puts
> command for everything else.
>

Bob,

Did you see the fix that I suggested in an earlier posting in this
thread?  (look for "<----- FIX")

That fix does what you propose here.

In the meantime, I resolved the issues with non-standard files too and
fixed a problem that had been created when defining the alias for puts.

In case you're interested: I've attached the complete setup for using
slave interps with tcltest to this message. It contains a self test also.


Greetings,

Erik Leunissen
--
leunissen@       nl | Merge the left part of these two lines into one,
e.          hccnet.   | respecting a character's position in a line.