Role of process priority in locking mechanism

From: Casper H.S. Dik (Casper.Dik@Sun.COM)
Subject: Re: Role of process priority in locking mechanism

View this article only
Newsgroups: comp.unix.programmer
Date: 2004-12-22 04:04:16 PST

"Mona" <indian.monalisa@gmail.com> writes:

>Can anyone explain me, role played by spls in kernel locks ?
splxxx() are a set of routines -which provide a priority level(xxx),
which preempts other lower priority interrupts from preempting the
code. They are used throughout the UNIX kernel, but they do not
necessarily have anything to do with the hw. You could use splxxx()
within an implementation of a syscall() , a device's read(), mmap(),
ioctl() etc.. Its just to ensure atomicity of execution of a certain
fragment of a code block.

>This depends very much on the specific kernel you are playing with.

>In traditional Unix kernel, the spl*() function are used to lock
>out interrupts and some other activities while manipulating certain
>data structures.  In those systems, spl*() functions as a primitive
>locking scheme.

a primitive lock would be something like a spinlock().

>It can be used to lock out
>If you take a modern, multi-threaded kernel, like Solaris, you'll
>find that you can no longer manipulate interrupt masks inside
drivers;

Well -whether the kernel is multi-threaded or not does not ahve much
to do with the nature of locks. Perhaps you mean, MP has a lot to with
the implementation of locks.

>rather, the system does it all for you.  With each driver mutex
>Solaris associates a "cookie" which describes the interrupt
properties
>of the driver.  The interrupt handler can then safely obtain the lock
>and lower level code can also safely obtain the lock as the kernel

then how will you implement splnet()? Its not related to any driver.

>will magically prevent interrupts from occuring while the lock is
held.

netbsd, freebsd etc.. are all examples of multi-threaded kernels that
dont work that way You are describing an implementation.

regards
-kamal

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

kamalp@acm.org (Kamal R. Pra) writes:

>splxxx() are a set of routines -which provide a priority level(xxx),
>which preempts other lower priority interrupts from preempting the
>code. They are used throughout the UNIX kernel, but they do not
>necessarily have anything to do with the hw. You could use splxxx()
>within an implementation of a syscall() , a device's read(), mmap(),
>ioctl() etc.. Its just to ensure atomicity of execution of a certain
>fragment of a code block.

Except that it doesn't in a Solaris kernel (there is more than one CPU)
and you also block progress of all unrelated threads; in that it
is a very primitive, coarse grained, lock.
(splnet(), e.g., locks at the network interrupts so you can safely
do administrative in the network stack)

>a primitive lock would be something like a spinlock().

Why is that primitive?  Spinlocks are used a lot in MP systems
for short term waits (the Solaris kernel spins when it tries to
grab a lock held by a thread currently running on a CPU); it assumes
the lock will be short term.  (It is an adaptive lock)

>Well -whether the kernel is multi-threaded or not does not ahve much
>to do with the nature of locks. Perhaps you mean, MP has a lot to with
>the implementation of locks.

>then how will you implement splnet()? Its not related to any driver.

You don't; the Solaris kernel does not have any interrupt level
manipulation routines available for use by driver writers.

>netbsd, freebsd etc.. are all examples of multi-threaded kernels that
>dont work that way You are describing an implementation.

Yes, I did not claim otherwise; threaded implementations which still
require spl*() calls are not MT-hot as those which do not.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Casper H.S. Dik wrote:
> kamalp@acm.org (Kamal R. Pra) writes:
>
> 
>
>
> Except that it doesn't in a Solaris kernel (there is more than one CPU)
> and you also block progress of all unrelated threads; in that it
> is a very primitive, coarse grained, lock.
> (splnet(), e.g., locks at the network interrupts so you can safely
> do administrative in the network stack)
>
> 
>
>
> Why is that primitive?  Spinlocks are used a lot in MP systems
> for short term waits (the Solaris kernel spins when it tries to
> grab a lock held by a thread currently running on a CPU); it assumes
> the lock will be short term.  (It is an adaptive lock)

Pardon some thread drift here.

You are abosutely correct when you say that the
Solaris kernel assumes the lock will be short term and
that the code is adaptive.  As a matter of fact,
there is a comment at the top of the adaptive
mutex code which reinforces this and says
something along the lines of: "if this
is not the case, re-think your algorithm."
(But how many developers read comments anyway? :)
As I recall, the adaptive mutexes default to
spinlocks.)

[Yes, we have access under NDA to the Solaris
source.  I don't think I am violating the NDA
by paraphrasing the comment above.]

A poorly written device driver can eat up gobs
of CPU time if the mutexes are at too great a
granularity or are held too long.

We had an experience a couple of years ago with
some third-party boards.  Our performance testing
indicated that the bulk of CPU was spent in
the "care and feeding" of the board (>90%
writes.  For the most part, all the interrupts
were doing was setting an IODONE flag.).
In order to get more throughput, we
upgraded the CPU's to double the speed, but,
we got zero increase in throughput!

Now, the vendor recommended that, in order
to increase throughput, we put in more boards.
(Thus increasing the vendor's sales and profits
at 8K USD per board?  I wonder?)
However we were out of card-slots in the frame
and this was not an option for us.

Upon further investigation, we found that their
driver code was using adaptive mutexes and that
the granularity of the locking was *per board*
as opposed to per channel (64 channels per board).

What does this have to do with the "splxxx()"
discussion?  Not much, other than that any kind
of locking mechanism can be mis-used and abused.

End Thread Drift.

[Rest Snipped ]

NPL


--
"It is impossible to make anything foolproof
because fools are so ingenious"
- A. Bloch