Home > Archive > Java Help > February 2005 > Block in synchronized mathod
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Block in synchronized mathod
|
|
|
| Hello, the situation is as followes: I have a bunch of Foo threads which
all have a pointer to the same instance of SomeClass. SomeClass has a
synchronized method someFunction() so that only one Foo can acces it at
a time. so far so good.
Now: in someFunction() if someCondition the current thread must block
until further notification by another thread Foo. (A piece of code below
to make it more clear.)
What would be a good way to do that? I have tried wait() and notify() in
a lot of ways but I am beginning to feel those should not be used for
this particular problem. (besides all the IllegalMonitorStateExceptions
I keep getting with it! ;-) )
A suggestion would be deeply appreciated.
Yuri
---------
in code:
---------
class Foo extends thread
{
SomeClass someClass;
Foo(SomeClass someClass)
{
this.someClass = someClass;
}
run()
{
if(someOtherCondition)
this.someClass.someFunction(this);
else
//notify this.someClass to continue;
}
}
class SomeClass
{
private synchronized void someFunction(Foo foo)
{
if(someCondition)
{
//block until notification
}
}
}
| |
| Gordon Beaton 2005-02-21, 8:58 am |
| On Mon, 21 Feb 2005 12:08:15 +0100, Yuri wrote:
> Hello, the situation is as followes: I have a bunch of Foo threads
> which all have a pointer to the same instance of SomeClass.
> SomeClass has a synchronized method someFunction() so that only one
> Foo can acces it at a time. so far so good.
>
> Now: in someFunction() if someCondition the current thread must
> block until further notification by another thread Foo. (A piece of
> code below to make it more clear.)
>
> What would be a good way to do that? I have tried wait() and
> notify() in a lot of ways but I am beginning to feel those should
> not be used for this particular problem. (besides all the
> IllegalMonitorStateExceptions I keep getting with it! ;-) )
You should be using wait() and notify(). The situation you describe is
exactly what they're for.
If you are getting IllegalMonitorStateExceptions, it means that you
are not waiting (or notifying) on the same object you are currently
syncrhonized with (or you are not inside a syncrhonized block).
When you wait(), always use a loop with a condition that can be
tested. Both the test and the wait must be done inside the
synchronized block:
synchronized (foo) {
while (!is_data_ready()) {
foo.wait();
}
}
Similarly when you use notify(), make the condition true before
notifying, and again do both within the synchronized block:
synchronized (foo) {
make_data_ready();
foo.notify();
}
"foo" must be a reference to the same object in both waiter and
notifier.
If "foo" is not specified, "this" is implied. Again, it must refer to
the same object in both waiter and notifier.
/gordon
--
[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e
| |
| Tilman Bohn 2005-02-21, 9:00 pm |
| In message <cvcfgb$g0s$1@azure.qinip.net>,
Yuri wrote on Mon, 21 Feb 2005 12:08:15 +0100:
> Hello, the situation is as followes: I have a bunch of Foo threads which
> all have a pointer to the same instance of SomeClass. SomeClass has a
> synchronized method someFunction() so that only one Foo can acces it at
> a time. so far so good.
> Now: in someFunction() if someCondition the current thread must block
> until further notification by another thread Foo. (A piece of code below
> to make it more clear.)
This is exactly what the wait/notify mechanism is for, so you're
on the right track.
> What would be a good way to do that? I have tried wait() and notify() in
> a lot of ways but I am beginning to feel those should not be used for
> this particular problem.
Yes, they are specifically meant for this type of problem.
> (besides all the IllegalMonitorStateExceptions
> I keep getting with it! ;-) )
That's because you're not using them correctly. :-) You probably
failed to synchronize on someClass before invoking notify(). See below.
> A suggestion would be deeply appreciated.
> Yuri
>
> ---------
> in code:
> ---------
(I'll leave out catching InterruptedException for brevity.)
> class Foo extends thread
> {
> SomeClass someClass;
>
> Foo(SomeClass someClass)
> {
> this.someClass = someClass;
> }
>
> run()
> {
> if(someOtherCondition)
> this.someClass.someFunction(this);
> else
> //notify this.someClass to continue;
synchronized (someClass){
// first modify someClass's state so that someCondition no
// longer holds, and then
someClass.notify();
}
> }
> }
>
> class SomeClass
> {
> private synchronized void someFunction(Foo foo)
> {
> if(someCondition)
> {
> //block until notification
> }
The correct idiom is
while (someCondition){
wait();
}
// at this point you know that a) the guard condition no longer holds,
// b) you have been notified, and c) you have reacquired the lock.
// So now you can do what you need to do on notification, reassured
// that no other thread can be executing anything synchronized on this
// monitor.
// If at this point you modify this object's state so that some
// _other_ guard condition's value might have changed, make sure to
// notify() other threads that might be waiting for that!
} else { ... }
The loop is _absolutely_ necessary because the condition, which was
certainly true when you started waiting, and which no longer holds when
notify() is called, could again hold by the time you actually wake up.
Look at what happens to the monitor: On entering your wait() loop, you
hold the lock. On starting to wait(), you release it. At some point in
the future, another thread enters the synchronized block in Foo's run()
method, acquiring that lock. On calling notify(), it again releases it,
first waking up some thread in that monitor's wait set. However, even if
the first thread I mentioned is the only candidate, you cannot be sure
that some other thread might not have acquired that same lock before you
wake up, so wait() can still keep blocking arbitrarily long after having
been notified, during which time someone else could manipulate
someClass's state -- possibly changing the value of someCondition. So
it is imperative that you go back and check your guard again, and if
needed, start wait()ing again.
And _don't_ think you can cut corners here. Even if you have only one
thread waiting for only one condition, you could still get bitten by one
or both of spurious wakeups (google for details) and simply some other
thread maliciously or erroneously calling notify() -- which is a public
method after all.
BTW, note that the guarded wait is conventionally written as `while
(!guard) wait();', I just used the positive form because that's how it
was in your code.
--
Cheers, Tilman
`Boy, life takes a long time to live...' -- Steven Wright
| |
|
| Gordon Beaton wrote:
> When you wait(), always use a loop with a condition that can be
> tested. Both the test and the wait must be done inside the
> synchronized block:
>
> synchronized (foo) {
> while (!is_data_ready()) {
> foo.wait();
> }
> }
>
> Similarly when you use notify(), make the condition true before
> notifying, and again do both within the synchronized block:
>
> synchronized (foo) {
> make_data_ready();
> foo.notify();
> }
Thanks Gordon, I got it to work now! I have to study a bit on the use of
the synchronized block before I _fully_ understand what is going on. I
wasn't sure I needed the while loop, however Tilman in a parallel reply
sugested it is absolutely necessary so I'll try understand his message
first.
tx again,
yuri
| |
| Tilman Bohn 2005-02-21, 9:00 pm |
| Oops, this was left over from an earlier version:
In message <slrnd1jq4g.8o5.myfirstname@urizen.tilmanbohn.com>,
Tilman Bohn wrote on Mon, 21 Feb 2005 14:56:32 +0100:
[...]
> // If at this point you modify this object's state so that some
> // _other_ guard condition's value might have changed, make sure to
> // notify() other threads that might be waiting for that!
>
> } else { ... }
I had this in an earlier version of my reply, where I added a separate
guard condition on top of your someCondition because it wasn't quite
clear to me at first from the example whether it would be an appropriate
predicate. But later I was pretty sure it either is appropriate, or else
could be made appropriate in any case, and forgot to delete this.
So ignore this orphaned else clause.
--
Cheers, Tilman
`Boy, life takes a long time to live...' -- Steven Wright
| |
|
| Tilman Bohn wrote:
<snip>
> And _don't_ think you can cut corners here. Even if you have only one
> thread waiting for only one condition, you could still get bitten by one
> or both of spurious wakeups (google for details) and simply some other
> thread maliciously or erroneously calling notify() -- which is a public
> method after all.
Ok ok! I won't cut any corners this time! ;) Thank you for your
_very_clear_ explenation! This realy helps me a lot. As soon is I find
just a shadow of free time I'll try to bring it into practice. And I
promise to do my best to implement it correctly.
Thanks,
Yuri
| |
| Tilman Bohn 2005-02-21, 9:00 pm |
| In message <cvcqae$r4v$1@azure.qinip.net>,
Yuri wrote on Mon, 21 Feb 2005 15:13:18 +0100:
[...]
> the synchronized block before I _fully_ understand what is going on. I
> wasn't sure I needed the while loop, however Tilman in a parallel reply
> sugested it is absolutely necessary so I'll try understand his message
> first.
That's why I was so insistent on it: Most people, when first seeing
this idiom, think they can somehow dispense with the loop. Even in the
most trivial cases, that is a grave error.
--
Cheers, Tilman
`Boy, life takes a long time to live...' -- Steven Wright
| |
| Tony Dahlman 2005-02-22, 8:58 am |
| Tilman Bohn wrote:
> In message <cvcfgb$g0s$1@azure.qinip.net>,
> Yuri wrote on Mon, 21 Feb 2005 12:08:15 +0100:
>
>
>
> The correct idiom is
>
> while (someCondition){
> wait();
> }
> // at this point you know that a) the guard condition no longer holds,
> // b) you have been notified, and c) you have reacquired the lock.
>
> // So now you can do what you need to do on notification, reassured
> // that no other thread can be executing anything synchronized on this
> // monitor.
>
> // If at this point you modify this object's state so that some
> // _other_ guard condition's value might have changed, make sure to
> // notify() other threads that might be waiting for that!
>
> [snip]
>
> The loop is _absolutely_ necessary because the condition, which was
> certainly true when you started waiting, and which no longer holds when
> notify() is called, could again hold by the time you actually wake up.
>
> Look at what happens to the monitor: On entering your wait() loop, you
> hold the lock. On starting to wait(), you release it. At some point in
> the future, another thread enters the synchronized block in Foo's run()
> method, acquiring that lock. On calling notify(), it again releases it,
> first waking up some thread in that monitor's wait set. However, even if
> the first thread I mentioned is the only candidate, you cannot be sure
> that some other thread might not have acquired that same lock before you
> wake up, so wait() can still keep blocking arbitrarily long after having
> been notified, during which time someone else could manipulate
> someClass's state -- possibly changing the value of someCondition. So
> it is imperative that you go back and check your guard again, and if
> needed, start wait()ing again.
>
> And _don't_ think you can cut corners here. Even if you have only one
> thread waiting for only one condition, you could still get bitten by one
> or both of spurious wakeups (google for details) and simply some other
> thread maliciously or erroneously calling notify() -- which is a public
> method after all.
>
> [snip]
>
You have thought carefully and done your homework on this. Thanks, Tillman!
But could there be something spurious about "spurious wakeups?"
If so, this or news:comp.lang.java.programmer would be a good place to
find out.
Google gives 4,500+ hits, but the JavaDoc comments for java.lang.Object 1.5
are interesting:
http://java.sun.com/j2se/1.5.0/docs...ang/Object.html
[color=darkred]
-----------------------------------------------------------------
A thread can also wake up without being notified, interrupted, or timing out, a
so-called spurious wakeup. While this will rarely occur in practice, applications must
guard against it by testing for the condition that should have caused the thread to be
awakened, and continuing to wait if the condition is not satisfied. In other words, waits
should always occur in loops, like this one:
synchronized (obj) {
while (<condition does not hold> )
obj.wait(timeout);
... // Perform action appropriate to condition
}
(For more information on this topic, see Section 3.2.3 in Doug Lea's "Concurrent
Programming in Java (Second Edition)" (Addison-Wesley, 2000), or Item 50 in Joshua Bloch's
"Effective Java Programming Language Guide" (Addison-Wesley, 2001).
------------------------------------------------------
So I looked for some context and found that maybe the main issue is whether JVM
authors should allow spurious wakeups. I assume that would be for regaining
control to an IDE, perhaps in its integrated debugger. I can't comment on the
merits/demerits of that debate, but trying to force everyone writing multi-threaded
code to put all their notify()s in while loops seems somehow impure, if not
downright undemocratic.
And if the IDE wants to wake up (and kill) some threads before they are orphaned
in memory, the loop is probably irrelevant.
Maybe more important, if using these loops can compensate for sloppy code, which I
imagine it could (by failing to raise runtime exceptions), then maybe the founding
fathers and mothers of Java need to take a second (or first?) look at this.
Meanwhile Joshua Bloch and Doug Lea are going to sell a lot of books <gr> (which
is great of course) and they deserve it. Or maybe you are writing the next one!
Regards -- Tony Dahlman
| |
| Tilman Bohn 2005-02-22, 8:58 am |
| [I've taken the liberty to re-format your post to shorter lines.]
In message <421A83A1.7020701@jps.net.invalid>,
Tony Dahlman wrote on Tue, 22 Feb 2005 00:58:13 GMT:
[...]
> (For more information on this topic, see Section 3.2.3 in Doug Lea's
> "Concurrent Programming in Java (Second Edition)" (Addison-Wesley,
> 2000), or Item 50 in Joshua Bloch's "Effective Java Programming
> Language Guide" (Addison-Wesley, 2001).
BTW, while we're at it, anyone trying to write moderately complex
multi-threaded code and have it actually be correct and safe should have
an extra copy of Lea's excellent book under their pillow. Bloch's book
should be under any Java programmer's pillow anyway. ;-)
> ------------------------------------------------------
>
> So I looked for some context and found that maybe the main issue is
> whether JVM authors should allow spurious wakeups.
[...]
No, the issue is that the underlying threading libraries, which Java
builds on, exhibit this behavior. The guarded wait idiom is exactly the
same in C as it is in Java. For example with pthreads:
pthread_mutex_lock(&mut);
while (x <= y) {
pthread_cond_wait(&cond, &mut);
}
/* operate on x and y */
pthread_mutex_unlock(&mut);
(Taken verbatim from my pthread_cond(3) man page.)
The reason it is implemented that way is this: It would be possible of
course to ensure 100% correct behavior within the library, but doing so
would be so much extra effort that performance would suffer. So instead
a conscious design decision was made to allow for the small chance of
spurious wakeups and instead expect client code to only ever use guarded
wait loops. Note that this is not the only reason for using that idiom,
so the fact that the possibility of spurious wakeups forces you to use
it is not in any way a di vantage. In fact, in the OpenGroup's SUS
(Single UNIX Specification) man page for the same group of functions
(that I posted a link to yesterday) this is explicitly mentioned, if
only in the informative section:
`The effect is that more than one thread can return from its call to
pthread_cond_wait() or pthread_cond_timedwait() as a result of one
call to pthread_cond_signal(). This effect is called "spurious
wakeup". Note that the situation is self-correcting in that the
number of threads that are so awakened is finite; for example, the
next thread to call pthread_cond_wait() after the sequence of events
above blocks.
`While this problem could be resolved, the loss of efficiency for a
fringe condition that occurs only rarely is unacceptable, especially
given that one has to check the predicate associated with a
condition variable anyway. Correcting this problem would
unnecessarily reduce the degree of concurrency in this basic
building block for all higher-level synchronization operations.
`An added benefit of allowing spurious wakeups is that applications
are forced to code a predicate-testing-loop around the condition
wait. This also makes the application tolerate superfluous condition
broadcasts or signals on the same condition variable that may be
coded in some other part of the application. The resulting
applications are thus more robust. Therefore, IEEE Std 1003.1-2001
explicitly documents that spurious wakeups may occur.'
http://makeashorterlink.com/?L1072278A
(Ok, deciding to paste this makes my last paragraph above sort of
redundant. But perhaps it can't be stated often enough.)
Now I can't claim to be sure that _all_ threading libraries Java uses
on _all_ platforms show this behavior, but 1. I suspect they do, and
2. it's enough that some do anyway. (Think portability.)
--
Cheers, Tilman
`Boy, life takes a long time to live...' -- Steven Wright
| |
| Tilman Bohn 2005-02-22, 8:58 am |
| In message <slrnd1lhlm.8o5.myfirstname@urizen.tilmanbohn.com>,
Tilman Bohn wrote on Tue, 22 Feb 2005 06:44:22 +0100:
[...]
> (Taken verbatim from my pthread_cond(3) man page.)
Errm. I guess that's rather ambiguous. I didn't mean to imply I
_wrote_ it, just that it's the one on the system I'm posting from right
now. ;-)
--
Cheers, Tilman
`Boy, life takes a long time to live...' -- Steven Wright
| |
| Tony Dahlman 2005-02-23, 4:01 pm |
| Tilman Bohn wrote:
> [I've taken the liberty to re-format your post to shorter lines.]
>
> In message <421A83A1.7020701@jps.net.invalid>,
> Tony Dahlman wrote on Tue, 22 Feb 2005 00:58:13 GMT:
>
> [...]
>
>
>
> BTW, while we're at it, anyone trying to write moderately complex
> multi-threaded code and have it actually be correct and safe should have
> an extra copy of Lea's excellent book under their pillow. Bloch's book
> should be under any Java programmer's pillow anyway. ;-)
>
>
>
> [...]
>
> No, the issue is that the underlying threading libraries, which Java
> builds on, exhibit this behavior. The guarded wait idiom is exactly the
> same in C as it is in Java. For example with pthreads:
>
> pthread_mutex_lock(&mut);
> while (x <= y) {
> pthread_cond_wait(&cond, &mut);
> }
> /* operate on x and y */
> pthread_mutex_unlock(&mut);
>
> (Taken verbatim from my pthread_cond(3) man page.)
>
> The reason it is implemented that way is this: It would be possible of
> course to ensure 100% correct behavior within the library, but doing so
> would be so much extra effort that performance would suffer. So instead
> a conscious design decision was made to allow for the small chance of
> spurious wakeups and instead expect client code to only ever use guarded
> wait loops. Note that this is not the only reason for using that idiom,
> so the fact that the possibility of spurious wakeups forces you to use
> it is not in any way a divantage. In fact, in the OpenGroup's SUS
> (Single UNIX Specification) man page for the same group of functions
> (that I posted a link to yesterday) this is explicitly mentioned, if
> only in the informative section:
>
> `The effect is that more than one thread can return from its call to
> pthread_cond_wait() or pthread_cond_timedwait() as a result of one
> call to pthread_cond_signal(). This effect is called "spurious
> wakeup". Note that the situation is self-correcting in that the
> number of threads that are so awakened is finite; for example, the
> next thread to call pthread_cond_wait() after the sequence of events
> above blocks.
>
> `While this problem could be resolved, the loss of efficiency for a
> fringe condition that occurs only rarely is unacceptable, especially
> given that one has to check the predicate associated with a
> condition variable anyway. Correcting this problem would
> unnecessarily reduce the degree of concurrency in this basic
> building block for all higher-level synchronization operations.
>
> `An added benefit of allowing spurious wakeups is that applications
> are forced to code a predicate-testing-loop around the condition
> wait. This also makes the application tolerate superfluous condition
> broadcasts or signals on the same condition variable that may be
> coded in some other part of the application. The resulting
> applications are thus more robust. Therefore, IEEE Std 1003.1-2001
> explicitly documents that spurious wakeups may occur.'
>
> http://makeashorterlink.com/?L1072278A
>
> (Ok, deciding to paste this makes my last paragraph above sort of
> redundant. But perhaps it can't be stated often enough.)
>
> Now I can't claim to be sure that _all_ threading libraries Java uses
> on _all_ platforms show this behavior, but 1. I suspect they do, and
> 2. it's enough that some do anyway. (Think portability.)
>
Thanks, Tillman. Actually I saw your previous post about the underlying threading
libraries being at fault. This is a full and final explanation as far as I'm
concerned, and you're right: "...it can't be stated often enough." At least
not for old farts/purists like me.
Clearly I was on the wrong track thinking this was one of those business "issues"
with the tool and JVM producers. Curiosity is undiminished, however. Any idea
how much efficiency would be lost if the Java spec didn't permit spurious
wakeups?
But I'm happy because in all my published code all the wait()s
somehow happen to be nested in while( !condition ) loops. All because the logic
seemed to require it some 7 to 10 years ago. In one case, however, the while()
was just checking if the application was ready to complete. (There was no "done"
variable for a thread, only an "allDone" variable for the application.) Was
that right or should I rewrite it with a tighter loop? As I see it, a spurious
wakeup would do nothing....
--------------------------------------------------------
public synchronized void runTasks() {
while (!allDone) {
startTask(0);
if( isDone[0] )
startTask(1);
if( isDone[0] )
startTask(2);
if( isDone[1] )
startTask(3);
if( isDone[2] && isDone[3] )
startTask(4);
// wait() till notify()ed by setTasksDone()
try {
wait();
} catch ( InterruptedException e ) {
}
} /* endwhile */
//...
}
-------------------------------------------------------
Complete code at
http://pws.prserv.net/ad/programs/P...l#TaskScheduler
But I'm also curious about those "superflous condition broadcasts or signals on
the same condition". Where are those coming from in a Java program that (rarely)
might produce them? Is some of the underlying code of the listener-event construct,
which is such a plus for Java, doing that, even if rarely?
Anyway, whether you can help satisfy my curiosity or not, thanks for posting some
of that nostalgic C code in support of your argument (never mind who wrote it).
I'm convinced.
-- Tony Dahlman
| |
| Tilman Bohn 2005-02-23, 4:02 pm |
| [Some re-wrapping of quoted text ahead]
In message <421C3EA4.3040109@jps.net>,
Tony Dahlman wrote on Wed, 23 Feb 2005 08:28:23 GMT:
[...]
> Clearly I was on the wrong track thinking this was one of those
> business "issues" with the tool and JVM producers. Curiosity is
> undiminished, however. Any idea how much efficiency would be lost if
> the Java spec didn't permit spurious wakeups?
Sorry, no idea. I've never investigated it since the loop is the
correct and robust way to do it anyway.
[...]
> seemed to require it some 7 to 10 years ago. In one case, however,
> the while() was just checking if the application was ready to
> complete. (There was no "done" variable for a thread, only an
> "allDone" variable for the application.) Was that right or should I
> rewrite it with a tighter loop? As I see it, a spurious wakeup would
> do nothing....
Looks ok to me at first glance, but I don't have time to look at it in
detail right now, so don't depend on my word here. Come to think of it,
don't depend on my word in any case. It's not like I haven't produced my
share of deadlocks and livelocks in my time. :-)
[...]
> But I'm also curious about those "superflous condition broadcasts or
> signals on the same condition". Where are those coming from in a Java
> program that (rarely) might produce them?
This means code executed in another thread that notifies threads
waiting on that same lock, without having altered state to make the
predicate hold. That could be either by accident (coding error), by
malice (third party code), or possibly by design. In many cases bad
design I suppose, but using coarse-grained monitors when you're waiting
for different but supposedly somehow related conditions can sometimes
make sense. For instance, maybe the notifying code can't be sure exactly
which of those related predicates have changed at the time it notifies
waiters. Then it would be natural to wake up all threads waiting on that
lock under these different guards.
> Is some of the underlying
> code of the listener-event construct, which is such a plus for Java,
> doing that, even if rarely?
[...]
Nah, just some other application code. You never know who might send
you a signal, or when, even if you keep very tight wraps on your monitor
(remember you can use any old Object as a monitor).
--
Cheers, Tilman
`Boy, life takes a long time to live...' -- Steven Wright
| |
| Tony Dahlman 2005-02-25, 4:00 am |
| Tilman Bohn wrote:
> [Some re-wrapping of quoted text ahead]
>
> In message <421C3EA4.3040109@jps.net>,
> Tony Dahlman wrote on Wed, 23 Feb 2005 08:28:23 GMT:
>
> [...]
>
>
>
> Sorry, no idea. I've never investigated it since the loop is the
> correct and robust way to do it anyway.
>
> [...]
>
>
>
> Looks ok to me at first glance, but I don't have time to look at it in
> detail right now, so don't depend on my word here. Come to think of it,
> don't depend on my word in any case. It's not like I haven't produced my
> share of deadlocks and livelocks in my time. :-)
>
> [...]
>
>
>
> This means code executed in another thread that notifies threads
> waiting on that same lock, without having altered state to make the
> predicate hold. That could be either by accident (coding error), by
> malice (third party code), or possibly by design. In many cases bad
> design I suppose, but using coarse-grained monitors when you're waiting
> for different but supposedly somehow related conditions can sometimes
> make sense. For instance, maybe the notifying code can't be sure exactly
> which of those related predicates have changed at the time it notifies
> waiters. Then it would be natural to wake up all threads waiting on that
> lock under these different guards.
>
>
>
> [...]
>
> Nah, just some other application code. You never know who might send
> you a signal, or when, even if you keep very tight wraps on your monitor
> (remember you can use any old Object as a monitor).
>
Like,
Object lock = new Object();
...
synchronized(lock) {
...
}
....which is totally missing from, yet useable and maybe adviseable in, my
code.
Your comments are much appreciated. But you are too humble about your status
as an authority. Thanks for the work you've done understanding this at so
many levels: convincing and instructive.
--Tony Dahlman
|
|
|
|
|