Interrupt latency

Hi,

I am writing my own real-time kernel for x86. Now I face something
really strange (or may be rather it's not; it has been some time since
I was in the details of x86 microarchitecture).

I measured CPU clocks elapsed between the first assembly instruction
executed at interrupt's entry point in IDT and beginning of the C code
of user-defined interrupt handler and the result was a big
surprise :-) It took about 2500 cycles despite that I have only a
handful of assembly instructions before a call to user-supplied IRQ
handler.

A little more testing showed that almost all cycles (2300+) were spent
at access to a global variable (via ds:[] addressing). Nothing that
accesses stack memory (push, pop, call, mov) makes a noticeable
difference. Does anybody have an idea why this happens? I test on
Celeron 2.8G in protected mode set up for flat model with paging
disabled.

I can post the code of the interrupt's entry point (until the C entry
point is called), but it's rather trivial and not long.

Thanks,
D

Stargazer wrote:
> Hi,
>
> I am writing my own real-time kernel for x86. Now I face something
> really strange (or may be rather it's not; it has been some time since
> I was in the details of x86 microarchitecture).
>
> I measured CPU clocks elapsed between the first assembly instruction
> executed at interrupt's entry point in IDT and beginning of the C code
> of user-defined interrupt handler and the result was a big
> surprise :-) It took about 2500 cycles despite that I have only a
> handful of assembly instructions before a call to user-supplied IRQ
> handler.
>
> A little more testing showed that almost all cycles (2300+) were spent
> at access to a global variable (via ds:[] addressing). Nothing that
> accesses stack memory (push, pop, call, mov) makes a noticeable
> difference. Does anybody have an idea why this happens? I test on
> Celeron 2.8G in protected mode set up for flat model with paging
> disabled.
>
> I can post the code of the interrupt's entry point (until the C entry
> point is called), but it's rather trivial and not long.
>
> Thanks,
> D

I cannot help you but just want to thank you for the educating (for
me) post.
Demonstrates I have been right to stay away from x86 (I have
considered
it 2 or 3 times every 5 years or so, and have not done so last 8 years
IIRC).
The latency is probably compiler generated, of course, but this does
not make things any better :-).

Dimiter

------------------------------------------------------
Dimiter Popoff               Transgalactic Instruments

http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_t...57600228621276/

On Mar 16, 4:02 pm, Stargazer  <spamt...@crayne.org> wrote:
> Hi,
>
> I am writing my own real-time kernel for x86. Now I face something
> really strange (or may be rather it's not; it has been some time since
> I was in the details of x86 microarchitecture).
>
> I measured CPU clocks elapsed between the first assembly instruction
> executed at interrupt's entry point in IDT and beginning of the C code
> of user-defined interrupt handler and the result was a big
> surprise :-) It took about 2500 cycles despite that I have only a
> handful of assembly instructions before a call to user-supplied IRQ
> handler.
>
> A little more testing showed that almost all cycles (2300+) were spent
> at access to a global variable (via ds:[] addressing). Nothing that
> accesses stack memory (push, pop, call, mov) makes a noticeable
> difference. Does anybody have an idea why this happens? I test on
> Celeron 2.8G in protected mode set up for flat model with paging
> disabled.
>
> I can post the code of the interrupt's entry point (until the C entry
> point is called), but it's rather trivial and not long.
>
> Thanks,
> D

What are the min, max and average cycle counts (you need to repeat the
measurement many times)?
What are the numbers on other PCs?

I wonder if it's SMIs. On my Dell Latitude D610 notebook an SMI (or a
short burst of thereof) may take up to ~240K cycles, which is ~150
microseconds at 1.6 GHz; on the old Compaq Armada 7800 notebook it's
only 12K cycles or ~40 microseconds at 300 MHz. hardware bugfixes and
control are moving into the CPU. :(

Alex

On Sun, 16 Mar 2008 16:02:18 -0700, Stargazer wrote:

> Hi,
>
> I am writing my own real-time kernel for x86. Now I face something
> really strange (or may be rather it's not; it has been some time since I
> was in the details of x86 microarchitecture).
>
> I measured CPU clocks elapsed between the first assembly instruction
> executed at interrupt's entry point in IDT and beginning of the C code
> of user-defined interrupt handler and the result was a big surprise :-)
> It took about 2500 cycles despite that I have only a handful of assembly
> instructions before a call to user-supplied IRQ handler.
>
> A little more testing showed that almost all cycles (2300+) were spent
> at access to a global variable (via ds:[] addressing). Nothing that
> accesses stack memory (push, pop, call, mov) makes a noticeable
> difference. Does anybody have an idea why this happens? I test on
> Celeron 2.8G in protected mode set up for flat model with paging
> disabled.
>
> I can post the code of the interrupt's entry point (until the C entry
> point is called), but it's rather trivial and not long.
>
> Thanks,
> D

Cache latency?

--
Tim Wescott
Control systems and communications consulting
http://www.wescottdesign.com

Need to learn how to apply control theory in your embedded system?
"Applied Control Theory for Embedded Systems" by Tim Wescott
Elsevier/Newnes, http://www.wescottdesign.com/actfes/actfes.html

On Mar 16, 7:02_pm, Stargazer  <spamt...@crayne.org> wrote:

> difference. Does anybody have an idea why this happens? I test on
> Celeron 2.8G in protected mode set up for flat model with paging

God knows. Cache line fill, maybe. Bus contention with a shared-memory
graphics adapter. Any one of a million things. x86 in V86 mode is a
nondeterministic architecture; design accordingly.

Stargazer wrote:
> Hi,
>
> I am writing my own real-time kernel for x86. Now I face something
> really strange (or may be rather it's not; it has been some time since
> I was in the details of x86 microarchitecture).
>
> I measured CPU clocks elapsed between the first assembly instruction
> executed at interrupt's entry point in IDT and beginning of the C code
> of user-defined interrupt handler and the result was a big
> surprise :-) It took about 2500 cycles despite that I have only a
> handful of assembly instructions before a call to user-supplied IRQ
> handler.
>
> A little more testing showed that almost all cycles (2300+) were spent
> at access to a global variable (via ds:[] addressing). Nothing that
> accesses stack memory (push, pop, call, mov) makes a noticeable
> difference. Does anybody have an idea why this happens? I test on
> Celeron 2.8G in protected mode set up for flat model with paging
> disabled.

Something is not right.

The biggest part of latency is created by the SDRAM Tpre + Tras + Trcd +
Tcl + Tburst. The ds:[] access also causes the shadow descriptor miss.
So in the worst case it translates to the two SDRAM bursts to flush the
dirty cache lines, and another two bursts to reload. I would expect the
delay at the order of hundreds of the CPU cycles.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

Stargazer wrote:
> Hi,
>
> I am writing my own real-time kernel for x86. Now I face something
> really strange (or may be rather it's not; it has been some time since
> I was in the details of x86 microarchitecture).
>
> I measured CPU clocks elapsed between the first assembly instruction
> executed at interrupt's entry point in IDT and beginning of the C code
> of user-defined interrupt handler and the result was a big
> surprise :-) It took about 2500 cycles despite that I have only a
> handful of assembly instructions before a call to user-supplied IRQ
> handler.
>
> A little more testing showed that almost all cycles (2300+) were spent
> at access to a global variable (via ds:[] addressing). Nothing that
> accesses stack memory (push, pop, call, mov) makes a noticeable
> difference. Does anybody have an idea why this happens? I test on
> Celeron 2.8G in protected mode set up for flat model with paging
> disabled.

That's unlikely. 2300 cycles is about 1us, no memory is that slow. That
looks more like ISA bus access time. Do you have any I/O instructions or
MMIO accesses there, by any chance? How do you measure cycles, did you
remember to use synchronizing instructions with RDTSC?

> I can post the code of the interrupt's entry point (until the C entry
> point is called), but it's rather trivial and not long.

That would be helpful.

> Cache latency?

That's the first thing that jumps into mind, but... over 2000 cycles?
OTOH, may it be that L2 cache on Celeron (or on-board L3?) is
organized so badly that it read-modify-write to memory *always* causes
a cache miss? Interesting, I ought to test what happens if there is an
off-chip cache and if I can disable it.


D

Stargazer wrote:

> 	; Acknowledge interupt with PIC(s).
> 	; After calling callbacks, to enable nesting of interrupt handlers
> according to priorities
> 	; TODO: check rotation priorities of 8259 and other options
> 	;
> 	mov	al, CMD_EOI
> 	out	PIC_MASTER, al
> 	cmp	ebx, 8
> 	jb	fin
> 	out	PIC_SLAVE, al

These 2 OUT's are what takes 2000+ cycles. The reason you see it elsewhere i
s
most likely a bug in your measurement code.

Stargazer wrote:
> I measured CPU clocks elapsed between the first assembly instruction
> executed at interrupt's entry point in IDT and beginning of the C code
> of user-defined interrupt handler and the result was a big
> surprise :-) It took about 2500 cycles despite that I have only a
> handful of assembly instructions before a call to user-supplied IRQ
> handler.

Is the Celeron 2.8G related to the Pentium 4?  I'm not clued up on
processors of the last few years.

Interrupts, int and iret took about 1900 to 2000 cycles on the P4
processors for some reason.  I know it was significantly higher than the
earlier processors that took a few hundred cycles.   I'm not sure
how many cycles each instruction takes on its own though.

I think someone, maybe Piotr ran some benchmarks of int/iret and
sysenter/sysexit on the P4 a few years ago and posted them in a.o.d.


--
Marv