Fortran - Re: MPI/Fortran95 incompatibility? Was

Author

Re: MPI/Fortran95 incompatibility? Was - Re: Pass by reference in

Rich Townsend

2006-01-10, 4:11 am

Paul Van Delst wrote:
> James Giles wrote:
>
>
>
> O.k., that's pretty much what I thought. The reason I ask is because
> someone asked my opinion about a problem calling an MPI routine
> (MPI_ISSEND) from Fortran95 code where the actual argument is a
> subscripted array. This forced a copy of the array subsection into a
> buffer. To quote the original email:
>
> <quote from email, not J.Giles>
>
>
>
> Anyone have any insights? It seems to me MPI might need a bit of an
> update if it relies upon call-by-reference.
>

All I can offer is not very much: the Fortran bindings to MPI are terribly
broken, and appear to have been constructed by people with very little
understanding of Fortran. You may want to construct your own set of wrappers to
the C routines, that explicitly deal with the issues you outline.

cheers,

Rich

Greg Lindahl

2006-01-10, 4:11 am

In article <dpmj2e$l8g$1@scrotar.nss.udel.edu>,
Rich Townsend <rhdt@barVOIDtol.udel.edu> wrote:

>All I can offer is not very much: the Fortran bindings to MPI are terribly
>broken, and appear to have been constructed by people with very little
>understanding of Fortran.

Rich, do you normally refer to people in this insulting fashion? There
are plenty of people with Fortran experience on the committee, and
there is no easy, right answer. And these issues are discussed in the
MPI standard document. What more could you ask for? Bon bons on your
pillow every night?

http://www-unix.mcs.anl.gov/mpi/mpi...de19.htm#Node19

-- greg

Paul Van Delst

2006-01-10, 4:11 am

Dan Nagle

2006-01-10, 4:11 am

Hello,

The modern Fortran binding to MPI is, IMNSHO, very broken.
FWIW, so is the C++ binding. The _only_ binding
that seems right to me is the C binding.

Just off-hand, the book _MPI -The Complete Reference_ (V2) (whose
authors include MPI committee members) lists 6 problems
with the Fortran binding.

Further, why does a routine taking any type of array
need to have the size and the type argument (MPI_REAL etc)
passed as well?

Generic resolution seems to be unused, nor assumed-size arrays.

An /Advice to users/ discusses fudges to get around type checking
problems.

Etc.

No bon bons on the pillow, please, just reasonably support
the language. I think it's a reasonable surmise to believe
that the designers of the MPI Fortran binding didn't understand
Fortran very well.

Greg Lindahl wrote:
> In article <dpmj2e$l8g$1@scrotar.nss.udel.edu>,
> Rich Townsend <rhdt@barVOIDtol.udel.edu> wrote:
>
>
>
>
> Rich, do you normally refer to people in this insulting fashion? There
> are plenty of people with Fortran experience on the committee, and
> there is no easy, right answer. And these issues are discussed in the
> MPI standard document. What more could you ask for? Bon bons on your
> pillow every night?
>
> http://www-unix.mcs.anl.gov/mpi/mpi...de19.htm#Node19
>
> -- greg
>

--
Cheers!

Dan Nagle
Purple Sage Computing Solutions, Inc.

Rich Townsend

2006-01-10, 4:11 am

Greg Lindahl wrote:
> In article <dpmj2e$l8g$1@scrotar.nss.udel.edu>,
> Rich Townsend <rhdt@barVOIDtol.udel.edu> wrote:
>
>
>
>
> Rich, do you normally refer to people in this insulting fashion? There
> are plenty of people with Fortran experience on the committee, and
> there is no easy, right answer. And these issues are discussed in the
> MPI standard document. What more could you ask for? Bon bons on your
> pillow every night?
>

No, I ask for an implementation that is actually portable, rather than one
riddled with inconsistencies with the Fortran standard, as the standard document
itself points out. Is that really too much to ask?

A specific example is how derived types are dealt with. This will always be a
difficult issue, because Fortran is so strongly typed. But instead of presenting
a restricted set of routines that portably accomplish what can be done within
the limits of the language, the MPI comittee cooked up a really terrible kludge
that smacks of an attempt at direct, naive C -> Fortran conversion.

Ultimately, my stance is this: if it can't be done in Fortran, then don't try to
do it in Fortran. Use a different language.

cheers,

Rich

Greg Lindahl

2006-01-10, 4:11 am

In article <dpn09n$p8n$1@scrotar.nss.udel.edu>,
Rich Townsend <rhdt@barVOIDtol.udel.edu> wrote:

>No, I ask for an implementation that is actually portable, rather than one
>riddled with inconsistencies with the Fortran standard, as the standard document
>itself points out. Is that really too much to ask?

Er, I missed how insulting the Committee members is going to get what
you want.

Cheers,

-- greg

Rich Townsend

2006-01-10, 4:11 am

Greg Lindahl wrote:
> In article <dpn09n$p8n$1@scrotar.nss.udel.edu>,
> Rich Townsend <rhdt@barVOIDtol.udel.edu> wrote:
>
>
>
>
> Er, I missed how insulting the Committee members is going to get what
> you want.
>

I wasn't asking them to fix the problems. In fact, based on the fact they
couldn't get it right the first time, I won't be holding my breath in that respect.

In fact, I was merely explaining to Paul that the MPI90 bindings have severe
problems, and suggesting a plausible explanation for these problems. I'm sorry
that -- kindly acting on behalf of the MPI committee -- you've taken such
umbrage at my remarks.

cheers,

Rich

PS Care to address any of my criticisms? Or still acting in the committee's best
interests?

Greg Lindahl

2006-01-10, 4:11 am

In article <dpmt7g$lkf$1@news.nems.noaa.gov>,
Paul Van Delst <Paul.vanDelst@noaa.gov> wrote:

>
>This url is very helpful - thanks. A point is mentioned that seems particularly relevant:
>
>"Many MPI routines assume that actual arguments are passed by address and that arguments
>are not copied on entrance to or exit from the subroutine."

You're very welcome, Paul, I'm glad you found my posting to be
constructive and helpful.

-- greg

Greg Lindahl

2006-01-10, 4:11 am

In article <dpna15$s27$1@scrotar.nss.udel.edu>,
Rich Townsend <rhdt@barVOIDtol.udel.edu> wrote:

>In fact, I was merely explaining to Paul that the MPI90 bindings have severe
>problems,

You did more than that.

>PS Care to address any of my criticisms? Or still acting in the committee's best
>interests?

Can you stop attacking the messenger? Or assuming that people who
disagree with one thing you said must disagree with everything you
said? Or that anyone complaining about your lack of tact is "acting in
the committee's best interests"?

-- greg

glen herrmannsfeldt

2006-01-10, 4:11 am

Ron Shepard wrote:

> The problem seems to be related to the way strided arrays are passed
> to subprograms whose arguments are not assumed shape.

(snip)

> It seems on the surface that all that is necessary would be for
> mpi_xxx() to declare the array with assumed shape and with the
> target attribute.

> real, target :: dummy(:)

> Then no copy would be necessary going in or comping out, and the
> saved address (and stride info) would correspond to the actual
> argument. This would of course require explicit interfaces in the
> calling program, and some knowledge of the fortran compiler calling
> conventions, but at least it seems like it would solve the problem.

I could get out my MPI book, but I presume the MPI routines need
a contiguous array. The Fortran mpi_xxx() routine would then need
to make an explicit copy before calling the actual MPI routine.
Then one would need to figure out when and how to free that copy.

-- glen

Paul Van Delst

2006-01-10, 4:11 am

Ron Shepard wrote:
> The problem seems to be related to the way strided arrays are passed
> to subprograms whose arguments are not assumed shape.
>
> call mpi_xxx(... actual(n,:), ...)
>
> with
>
> subroutine mpi_xxx(... dummy, ...)
> real :: dummy(*) ! or dummy(n), etc.
>
> Many (most, all?) compilers will make a temporary copy of the actual
> array that has stride 1, then pass the address of that copy to
> mpi_xxx(). Inside mpi_xxx(), that address is saved for the
> subsequent asynchronous processing, the routine returns to the
> calling program, and the calling program immediately copies the
> (unset) values back to actual(n,:) and releases the temporary memory
> used to store the stride 1 copy. Later, the temporary memory is
> changed asynchronously as that no-longer-valid address is used,
> causing all kinds of havoc.

Yes - that is exactly the problem that we're seeing.
>
> It seems on the surface that all that is necessary would be for
> mpi_xxx() to declare the array with assumed shape and with the
> target attribute.
>
> real, target :: dummy(:)
>
> Then no copy would be necessary going in or comping out, and the
> saved address (and stride info) would correspond to the actual
> argument. This would of course require explicit interfaces in the
> calling program, and some knowledge of the fortran compiler calling
> conventions, but at least it seems like it would solve the problem.

Not necessarily. It's not clear to me whether or not that some compilers will /still/ make
copies regardless of whether the dummy arg is declared assumed shape. The IBM xlf compiler
comes to mind -- but I still haven't managed to get a definitive answer on this from an
"official" source.

> Since this is such a simple and obvious solution to the problem, I
> assume that there is something that I'm missing, or this would have
> been done 15 years ago. What is it that I'm missing?

That was the same question I was asking people here at work wrt the MPI standard. From
what I can gather about MPI, the Fortran bindings were implicitly built for Fortran77 due
to the assumption that the argument passing mechanism is always pass-by-reference. MPI
does work with Fortran90/95, but with all the caveats mentioned in this thread.

Does anyone know if the Fortran API for MPI is a layer on top of "base" C (I presume, or
whatever language) code -- ala netCDF, where the f90 API is an additional layer on top of
the f77 API??

cheers,

paulv

--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC

Paul Van Delst

2006-01-10, 4:11 am

Ron Shepard wrote:
> In article <dptvsf$69f$1@news.nems.noaa.gov>,
> Paul Van Delst <Paul.vanDelst@noaa.gov> wrote:
>
>
>
>
> This can be tested on most compilers using a LOC() or %LOC()
> function. Print out the location of the actual and dummy array with
> various declarations and see which ones result in local copies and
> which ones pass the original strided array.
>
> MPI is not simply a subroutine library, it is meant to work
> intimately with a particular compiler (or at least a set of
> compilers with the same calling conventions).

Yeah, I sort of figured that was probably the case. My netCDF analogy is not a good one in
this case. But then, it seems to me that simply doing things like
real, target :: dummy(:)
in mpi_xxx() as you suggested in a previous post may be a necessary part of a "fix", but
maybe not sufficient. (?)

cheers,

paulv

--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC

Greg Lindahl

2006-01-10, 4:11 am

In article <dptvsf$69f$1@news.nems.noaa.gov>,
Paul Van Delst <Paul.vanDelst@noaa.gov> wrote:

> The IBM xlf compiler comes to mind -- but I still haven't managed to
> get a definitive answer on this from an "official" source.

No compiler vendor is going to constrain what future compilers might
do.

By the way, this fix doesn't work for multi-dimensional arrays.

>Does anyone know if the Fortran API for MPI is a layer on top of "base" C (I presume, or
>whatever language) code -- ala netCDF, where the f90 API is an additional layer on top of
>the f77 API??

That's an implementation detail. MPICH happens to do the Fortran77 interface as a
set of C routines, which is how we were able to get our InfiniPath MPI library to
link with a bunch of different Fortrans, using just a single MPI library.

-- greg
(employed by, not speaking for, PathScale)

Jan Vorbrüggen

2006-01-10, 4:11 am

> IIRC, Nag has an option that causes copying of noncontiguous arguments
> even with assumed shape dummies. The idea is that in some cases there
> can be sufficient performance benefit to justify the copy.

As a side note, the programmer is of course always able to perform this
optimization (if it indeed is one) herself by explicitly writing out the
code to do the gather/scatter operation to a temporary. This seems a more
appropriate way of handling the issue to me, because it makes the issue
and the trade-offs involved visible in the code.

Jan

Jan Vorbrüggen

2006-01-10, 4:11 am

> But then, it seems to me that simply doing things like
> real, target :: dummy(:)
> in mpi_xxx() as you suggested in a previous post may be a necessary part
> of a "fix", but maybe not sufficient. (?)

As I understand it from this discussion, the underlying problem really is
that MPI's lower layers expect to be passed a contiguous buffer of data.
Your solution will solve the problem at the MPI interface - but then the
MPI routine will need to check whether what it has been passed is contiguous
or not, and make an appropriate contiguous temporary copy if required, and
not forget to release that temporary after it's done with it.

Is there a standard-conforming way to detect whether the actual to an
assumed-shape dummy is contiguous or not?

Jan

glen herrmannsfeldt

2006-01-10, 4:11 am

Jan Vorbrüggen wrote:

(snip)

> As I understand it from this discussion, the underlying problem really is
> that MPI's lower layers expect to be passed a contiguous buffer of data.
> Your solution will solve the problem at the MPI interface - but then the
> MPI routine will need to check whether what it has been passed is
> contiguous
> or not, and make an appropriate contiguous temporary copy if required, and
> not forget to release that temporary after it's done with it.

> Is there a standard-conforming way to detect whether the actual to an
> assumed-shape dummy is contiguous or not?

Write it in C and read the descriptor for the array. It shouldn't be
hard to do, though it might be Fortran compiler specific.

I think it is then possible for the interface to MPI_WAIT or similar
routines to recognize when each can be released.

-- glen

Paul Van Delst

2006-01-10, 9:58 pm

Richard E Maine wrote:
> Paul Van Delst <Paul.vanDelst@noaa.gov> wrote:
>
>
>
>
> IIRC, Nag has an option that causes copying of noncontiguous arguments
> even with assumed shape dummies. The idea is that in some cases there
> can be sufficient performance benefit to justify the copy. However, that
> argument is not on by default. I'd be surprised if XLF did this by
> default, but I don't actually know.

Well, I finally pulled my finger out and asked an IBM person. The reply I got was that xlf
generates array copies for both array subsets and array triplets. Even when the array
subsets are contiguous.

I did try the test program that James van Buskirk posted yesterday and xlf printed:
'The compiler passes the test'
so the issue isn't that cut and dried (if I correctly understand what's supposed to
happen). Still, it seems the simple case of passing contiguous array slices/subsets, or
regularly strided array triplets, as actual arguments will generate a copy "by default" in
xlf. I used the quotes since the optimisation opportunity of handling these arguments in a
more, uh, sophisticated manner was acknowledged followed by the statement that it hasn't
been done yet. So there would appear to be no switch to turn this behaviour off.

cheers,

paulv

Note that there are situations where
> the standard all but forbids a copy, though not in those words - it just
> would make correct implementation very tricky. I would be truly shocked
> if any compiler did a copy and managed to make it work correctly for
> those cases. I'd have to look up all the fine points and exceptions, but
> the target attribute is a major part.
>
>
>
>
> The Fortran 77 standard did not specify pass-by-reference either. This
> has *NEVER* been a part of the Fortran standard, and there have always
> been cases where some compilers used different passing mechanisms. The
> exceptions were just less common in f77. Erroneous statements about this
> are widespread. In fact, in some sense, many compilers (including f77
> ones) do things other than strict pass by reference, particularly when
> optimizers are turned on. Storing a variable in a register during a loop
> is a form of copying - one can write (nonstandard) sample codes where
> this is detectable.
>

--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC

Paul Van Delst

2006-01-10, 9:58 pm

Jugoslav Dujic wrote:
> Paul Van Delst wrote:
> | Richard E Maine wrote:
> || Paul Van Delst <Paul.vanDelst@noaa.gov> wrote:
> ||
> ||
> ||| Not necessarily. It's not clear to me whether or not that some compilers
> ||| will /still/ make copies regardless of whether the dummy arg is declared
> ||| assumed shape. The IBM xlf compiler comes to mind -- but I still haven't
> ||| managed to get a definitive answer on this from an "official" source.
> ||
> ||
> || IIRC, Nag has an option that causes copying of noncontiguous arguments
> || even with assumed shape dummies. The idea is that in some cases there
> || can be sufficient performance benefit to justify the copy. However, that
> || argument is not on by default. I'd be surprised if XLF did this by
> || default, but I don't actually know.
> |
> | Well, I finally pulled my finger out and asked an IBM person. The reply I got
> | was that xlf generates array copies for both array subsets and array
> | triplets. Even when the array subsets are contiguous.
> |
> | I did try the test program that James van Buskirk posted yesterday and xlf
> | printed: 'The compiler passes the test'
> | so the issue isn't that cut and dried (if I correctly understand what's
> | supposed to happen).
>
> Note that James's test program is not closely related to that issue --
> namely, with assumed-shape dummy and a non-contiguous array section
> (excluding vector subscripted ones, which I anyway count as an "expression"
> in my book) as real argument, a copy *never* has to be made. A compiler
> might do that for performance reasons (although it more often degrades
> performance than not)...
>
> Still, it seems the simple case of passing contiguous
> | array slices/subsets, or regularly strided array triplets, as actual
> | arguments will generate a copy "by default" in xlf. I used the quotes since
> | the optimisation opportunity of handling these arguments in a more, uh,
> | sophisticated manner was acknowledged followed by the statement that it
> | hasn't been done yet. So there would appear to be no switch to turn this
> | behaviour off.
>
> ...rather, I think that XLF always copies array subsets when they
> match an *assumed-size* dummy.

I'm not sure I understand your use of the word "rather" (i..e. agreeing or disagreeing)
and the assumed-size reference. So let me re-iterate.

Correctly or not, I always assumed (no pun) that an assumed /size/ dummy arg generated a
copy when an array subset/triplet was passed as an actual arg. I have no problem with that
since, if I understand things correctly, there is no "extra" information passed that would
allow one to use the SIZE() and/or SHAPE() intrinsics on the assumed /size/ dummy arg and,
thus, there would also be no "extra" information available that would allow the compiler
to generate code to access the data in question via a reference+amount of data+stride.
E.g. the following code,
subroutine assumedsize( a )
real :: a(*)
print *, shape(a)
end subroutine assumedsize
won't compile (at least, not on any compiler I've ever used)

But, my question to the IBM people was when an array subset/triplet actual arg was passed
to a subprogram with an assumed /shape/ dummy arg. In this case, there is that "extra"
info available, e.g. the code
subroutine assumedshape( a )
real :: a(:)
print *, shape(a)
end subroutine assumedshape
is fine. The response I got tells me that even in those cases, xlf generates a copy.

cheers,

paulv

That's plausible, but often degrades
> performance (and causes misbehavior with MPI, which is where we started
> from). I wouldn't expect a compiler switch though -- it's more like a
> not-implemented feature. Early versions of CVF suffered from that too.
>

--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC

Paul Van Delst

2006-01-10, 9:58 pm

Jugoslav Dujic wrote:
> Paul Van Delst wrote:
> | Jugoslav Dujic wrote:
> ||
> || Note that James's test program is not closely related to that issue --
> || namely, with assumed-shape dummy and a non-contiguous array section
> || (excluding vector subscripted ones, which I anyway count as an "expression"
> || in my book) as real argument, a copy *never* has to be made. A compiler
> || might do that for performance reasons (although it more often degrades
> || performance than not)...
> ||
> || Still, it seems the simple case of passing contiguous
> ||| array slices/subsets, or regularly strided array triplets, as actual
> ||| arguments will generate a copy "by default" in xlf. I used the quotes since
> ||| the optimisation opportunity of handling these arguments in a more, uh,
> ||| sophisticated manner was acknowledged followed by the statement that it
> ||| hasn't been done yet. So there would appear to be no switch to turn this
> ||| behaviour off.
> ||
> || ...rather, I think that XLF always copies array subsets when they
> || match an *assumed-size* dummy.
> |
> | I'm not sure I understand your use of the word "rather" (i..e. agreeing or
> | disagreeing) and the assumed-size reference. So let me re-iterate.
>
> I'm not sure I do too -- it was a mis-reference to another sentence :o)
>
> | Correctly or not, I always assumed (no pun) that an assumed /size/ dummy arg
> | generated a copy when an array subset/triplet was passed as an actual arg. I
> | have no problem with that since, if I understand things correctly, there is
> | no "extra" information passed that would allow one to use the SIZE() and/or
> | SHAPE() intrinsics on the assumed /size/ dummy arg and, thus, there would
> | also be no "extra" information available that would allow the compiler to
> | generate code to access the data in question via a reference+amount of
> | data+stride. E.g. the following code, subroutine assumedsize( a )
> | real :: a(*)
> | print *, shape(a)
> | end subroutine assumedsize
> | won't compile (at least, not on any compiler I've ever used)
> |
> | But, my question to the IBM people was when an array subset/triplet actual
> | arg was passed to a subprogram with an assumed /shape/ dummy arg. In this
> | case, there is that "extra" info available, e.g. the code
> | subroutine assumedshape( a )
> | real :: a(:)
> | print *, shape(a)
> | end subroutine assumedshape
> | is fine. The response I got tells me that even in those cases, xlf generates
> | a copy.
>
> Sorry, but I didn't figure that out from the context. I thought the question
> was related with assumed-size dummies. Now that you specify it's assumed-shape,
> the behavior as stated does not make much sense to me. Yes, it is (seems?)
> possible,
> but it IMO would be a lousy quality of implementation; for a . Are you *sure*
> you
> understood each other right?

If a Fortran-knowledgeable person such as yourself couldn't understand what I meant, maybe
not. (Sometimes I have to read what I wrote a couple times to figure out what I meant.
Sigh. :o)

It's quite easy to make a size/shape mixup. As
> you said, it passes James's test OK, so it makes me wonder....

Well, I was quite explicit in differentiating the two cases. I'm assuming the IBM folks
know the difference.

> To summarize:
> 1)If XLF always passes a copy for a stride or triplet even for assumed-shape
> target, that's quite unusual (and would be lousy). That would mean that
> they didn't implement "dope vectors" at all, but just somehow retrofitted
> assumed-shape into assumed-size semantics. That would also mean that they
> had to special-case James's test somehow.
> 2)If XLF always passes a copy for a stride or triplet for assumed-size argument,
> that's acceptable, if somewhat lazy way of doing it.
>
> But why don't you try case(1) for yourself -- you just have to figure out
> how to get address of RealArgArray(1) and DummyArgArray(1) with a LOC()
> extension or call to C routine?

I just checked the xlf manual and LOC() is a supported extension, so I think some mucking
about to test all this stuff and put the matter to rest is warranted.

From what I recall from a long-ago hallway discussion with the IBM in-building rep was
that the reason the copies are still being done is that they didn't want to special case
certain types of arguments and until they figured out a general solution, they would hold
off implementing a smarter way of doing it. But, now, I'm not sure that we were
necessarily talking about the same thing. The fact that you state they would have to
special case James's test case makes me think we weren't -- since I gathered special
casing situations was not their MO (but I can't verify that. And don't really want to anyway.)

cheers,

paulv

--
Paul van Delst
CIMSS @ NOAA/NCEP/EMC

Sponsored Links