Limits on globbing?

I have a directory with around 20k files in it (input files to a simulation,
if it matters).  I've written a perl script that will build a run script for
input files, but it doesn't work when I pass a certain limit...  (I think
it's characters, not number of files.)

I'm using the:

@files = <*.in>;

... notation with Perl 5.0.  If I have too many files, @files ends up with
nothing.  Is there a way to read in all 20k names?

Thanks!

- Bryan

If you have a question and get no response, PLEASE wait more than 90
minutes before sending your question in again. Give it at least a day,
if not a couple of days. We're all volunteers here and expectations of
immediate responses are not reasonable.

Thanks.

On Thu, 21 Oct 2004, Bryan Harris wrote:

> I have a directory with around 20k files in it (input files to a
> simulation, if it matters).  I've written a perl script that will
> build a run script for input files, but it doesn't work when I pass a
> certain limit...  (I think it's characters, not number of files.)
>
> I'm using the:
>
> @files = <*.in>;
>
> ... notation with Perl 5.0.  If I have too many files, @files ends up
> with nothing.  Is there a way to read in all 20k names?

Is that a typo, or are you really running Perl 5.0 (zero) ? That version
of Perl must be something like 15 years old by now. Please send the list
the output of a `perl -v` or `perl -V` to clarify this.

The other approach I can think of is to skip the plain glob and instead
loop over the directory, `push`ing each file into the @files array but
also including debugging statements:

while ... {
$iterator++
push( @files, $file ) or
die "She can' take no more than $iterator, cap'n! $file $!";
}

That might at least pin down what your Perl's limits are...



--
Chris Devers

On Fri, Oct 22, 2004 at 07:49:11AM -0400, Chris Devers wrote:

> Is that a typo, or are you really running Perl 5.0 (zero) ? That version
> of Perl must be something like 15 years old by now.

10 years old just this w, as it happens.

Happy Birthday Perl 5.

--
Paul Johnson - paul@pjcj.net
http://www.pjcj.net

>>>>> "Bryan" == Bryan Harris <bryan@harrisfam.net> writes:

Bryan> I'm using the:

Bryan> @files = <*.in>;

Bryan> ... notation with Perl 5.0.  If I have too many files, @files ends up
 with
Bryan> nothing.  Is there a way to read in all 20k names?

Upgrade to Perl 5.6 or later.  The globbing no longer calls /bin/csh,
and thus avoids the shell's command-line-length limitations.

Or, learn to use readdir(), which is clunkier.

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training
!

On 22 Oct 2004 06:41:10 -0700, Randal L. Schwartz <merlyn@stonehenge.com> wrote: 
>
> Bryan> I'm using the:
>
> Bryan> @files = <*.in>;
>
> Bryan> ... notation with Perl 5.0.  If I have too many files, @files ends 
up with
> Bryan> nothing.  Is there a way to read in all 20k names?
>
> Upgrade to Perl 5.6 or later.  The globbing no longer calls /bin/csh,
> and thus avoids the shell's command-line-length limitations.
>
> Or, learn to use readdir(), which is clunkier.

Ouch!

I *always* used readdir() in preference to the bizarre <*> notation... But t
hen
I'm a C programmer at heart..

opendir( DIR,"$base_dir" ) ;
%files = map { "$base_dir/$_" => [ stat( "$base_dir/$_" ) ] }
grep { /\.in$/ } readdir( DIR );
closedir ( DIR );

voila - a HASH of filenames where each filename has its inode info attached
for optimising later when you sort by file size, or access date..

I prefer the ability to 'grep' the directory listing rather than rely
on globbing
which isnt always precise enough.. This is especially useful when you are
grepping on say, whether it is a file or directory, whether the modified tim
e is
earlier than today etc..

As a consultant, I have had to convert what Bryan does with <*> to
what I consider
'proper' readdir style syntax to overcome filename expansion limits on
directories
with lots of files.. (ie, when 20000 files in a directory would be
considered a slow
day).   Note that these are transient files in an OLTP environment and on so
me
days they arrive so fast I've found '<*>' sometimes doesn't return....

>
> --
> Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 009
5
> <merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
> See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl traini
ng!
>
>
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> <http://learn.perl.org/> <http://learn.perl.org/first-response>
>
>

> If you have a question and get no response, PLEASE wait more than 90
> minutes before sending your question in again. Give it at least a day,
> if not a couple of days. We're all volunteers here and expectations of
> immediate responses are not reasonable.


Please forgive me!  I've gotten more help here than I deserve, thanks to
some really terrific guys.  I only resent the email because I thought I'd
sent it from the wrong account, and I hadn't seen it show up from the
listserv yet.  I certainly wasn't being impatient, despite how it may have
seemed.


 
>
> Is that a typo, or are you really running Perl 5.0 (zero) ? That version
> of Perl must be something like 15 years old by now. Please send the list
> the output of a `perl -v` or `perl -V` to clarify this.

Despite my nearly 3 years with perl, I'm definitely still a beginner!

I had assumed that "revision 5.0 version 4" meant perl 5.0, not perl 5.4.

Randal's solution I'm sure will be the preferred -- having our admin upgrade
perl is much easier than rewriting my code.  =)

Thanks to all!

- Bryan

On Fri, 22 Oct 2004, Bryan Harris wrote:

> Please forgive me!

No sweat :-)

> I had assumed that "revision 5.0 version 4" meant perl 5.0, not perl
> 5.4.

No, Perl versioning was weird for a while there. Perl 5.004x was the
notation for what later got shortened to just 5.4.x.

But even Perl 5.4 is old by now, and 5.6 is getting up there. The
current version is 5.8.4, so if you're going to have your admin upgrade
Perl anyway, you might as well have it brought up to 5.8.x.

> Randal's solution I'm sure will be the preferred -- having our admin
> upgrade perl is much easier than rewriting my code.  =)

Well, it may still be worth trying the other way, if only to get
yourself comfortable with the two approaches, and to benchmark which
style runs faster for what you're trying to do. You're still dealing
with a whole lot of files, and it would be good to know which mode
handles such a big set more efficiently.

(That said, Randal knows this a hell of a lot better than I do -- I'd
also defer to his suggestion :-)



--
Chris Devers

> If thats so, here's something to keep you entertained.


Yet another script that doesn't work as received... To make it work I had to
remove line 1, and then it still threw a bunch of errors at runtime.

Still, it's pretty .  =)

- B