SysFileTree on Linux

in 37752 20080331 224010 Michael Lueck <mlueck@lueckdatasystems.com> wrote:
>Steve Swift wrote: 
>
>The OP's question was how to make "hidden" files (Those starting with a per
iod) to show up when doing a SysFileTree for '*'.
>
>I confirmed they do not show up, I must specify '.*' to have those files sh
ow up.
>
>The OP's suggested that it was necessary to call SysFileTree for each direc
tory in the tree, however I find that it will pick up hidden files in subdir
s. Thus my question of which version of ooRexx he
>was using. I am using ooRexx 3.2.

I am using ooRexx 3.2

I have done extensive testing on this and I stand by my claim that SysFileTr
ee
does not recurse through the subdirs when searching for hidden files and dir
s
unless there is some way of specifying it that I don't know about.

Can we have some word from one of the developers please?

To reiterate :
this line
rc = SysFileTree('~/*','LIST.','BSL')
lists all the non-hidden files and dirs in my home directory and all lower l
evel directories

this line
rc = SysFileTree('~/.*','LIST.','BSL')
lists all the hidden files and dirs in my home directory - and no further!

On Mar 31, 11:51 pm, Bob Martin <bob.mar...@excite.com> wrote:
> in 37752 20080331 224010 Michael Lueck <mlu...@lueckdatasystems.com> wrote
:
> 
> 
>
...
> I have done extensive testing on this and I stand by my claim that SysFile
Tree
> does not recurse through the subdirs when searching for hidden files and d
irs
> unless there is some way of specifying it that I don't know about.
>
> Can we have some word from one of the developers please?

It has been quite a while since I looked at the unix SysFileTree
code.  Unfortunately, I don't have the time to delve into it right
now.

If you tested it well, then the behavior you see is the ooRexx 3.2.0
behavior.  Whether it should or shouldn't be the behavior is another
question.

Without testing it myself, I would expect SysFileTree to *only*
descend into subdirectories if the subdirectory matched your
pattern.

In other words if you used the pattern: .* it should only descend into
subdirectories whose names start with a dot and it should only find
files that start with a dot.  If it doesn't, then that might be a bug.

A quick test shows that, on Windows, it does work as I would expect.

However, since on unix-like systems the convention is to not display
files that start with a dot, declaring what the expected behavior
should be might not be so straight forward.

--
Mark Miesfeld

Greetings Mark-

> Without testing it myself, I would expect SysFileTree to *only*
> descend into subdirectories if the subdirectory matched your
> pattern.

I suspect I see 3.2 behaving differently. The following code found returned 
these files:

2008-04-01 13:46:32           5  -rw-r--r--  /home/mdlueck/parallels-vm/win2
000.test2k/.win2000.test2k.pvs.lock
2008-03-28 15:04:08           3  -rw-r--r--  /home/mdlueck/bin/.test

#!/usr/bin/rexx

rc=SysFileTree('~/.*', f., 'BSL')

if f.0>0 then do i=1 to f.0
say f.i
end

exit

The files start with a period, the directory in which they were found does n
ot.

--
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

in 37761 20080402 004302 Michael Lueck <mlueck@lueckdatasystems.com> wrote:
>Greetings Mark-
> 
>
>I suspect I see 3.2 behaving differently. The following code found returned
 these files:
>
>2008-04-01 13:46:32           5  -rw-r--r--  /home/mdlueck/parallels-vm/win
2000.test2k/.win2000.test2k.pvs.lock
>2008-03-28 15:04:08           3  -rw-r--r--  /home/mdlueck/bin/.test
>
>#!/usr/bin/rexx
>
>rc=SysFileTree('~/.*', f., 'BSL')
>
>if f.0>0 then do i=1 to f.0
>say f.i
>end
>
>exit
>
>The files start with a period, the directory in which they were found does 
not.
>
>--
>Michael Lueck
>Lueck Data Systems
>http://www.lueckdatasystems.com/


Surely you should see far more output from your program than that?
This is what I see when running your program :

2007-11-16 14:07:59         566  -rw-r--r--  /home/bob/.profile
2008-02-26 08:11:59        2880  -rwxrwxrwx  /home/bob/.bashrc
2007-11-16 14:07:59         220  -rw-r--r--  /home/bob/.bash_logout
2008-03-26 07:26:18         654  -rw-r--r--  /home/bob/.Xauthority
2008-03-11 20:02:38        6642  -rw-r--r--  /home/bob/.bash_history~
2008-04-02 07:34:29          26  -rw-------  /home/bob/.dmrc
2008-03-28 12:11:54          80  -rw-r--r--  /home/bob/.mtinkrc
2008-03-24 12:25:35         210  -rw-r--r--  /home/bob/.gtk-bookmarks
2008-04-02 07:34:31        1085  -rw-------  /home/bob/.ICEauthority
2007-11-16 14:45:23          85  -rw-r--r--  /home/bob/.gtkrc-1.2-gnome2
2007-11-16 14:45:32          16  -rw-r--r--  /home/bob/.esd_auth
2008-04-02 07:41:23           0  -rw-r--r--  /home/bob/.gksu.lock
2007-11-16 14:46:30           0  -rw-r--r--  /home/bob/.sudo_as_admin_succes
sful
2008-04-01 20:49:14        6397  -rw-r--r--  /home/bob/.bash_history
2008-04-01 20:49:14       80470  -rw-r--r--  /home/bob/.recently-used.xbel
2008-03-18 16:00:37         103  -rw-r--r--  /home/bob/.twmE8Xw2R
2007-11-17 13:59:51           0  -rw-r--r--  /home/bob/..OOREXX3.2.0.0_bob
2008-02-04 14:05:56        2821  -rw-rw-rw-  /home/bob/.bashrc~
2008-04-02 07:52:49        7271  -rw-r--r--  /home/bob/.xsession-errors
2008-03-18 21:31:12         168  -rw-r--r--  /home/bob/.twmX4IX2b
2008-04-01 07:24:23        1323  -rw-r--r--  /home/bob/.recently-used
2008-03-24 13:10:55          31  -rw-r--r--  /home/bob/.mcoprc
2008-01-24 19:44:15         332  -rw-r--r--  /home/bob/.nvidia-settings-rc
2008-02-16 13:34:08         103  -rw-r--r--  /home/bob/.twmrFFDgG
2007-12-18 16:29:35          49  -rw-r--r--  /home/bob/.serverauth.9932
2008-01-06 14:53:02          15  -rw-r--r--  /home/bob/.fulistrc
2008-02-16 14:30:50         103  -rw-r--r--  /home/bob/.twmByaMbk
2008-02-16 14:43:19         103  -rw-r--r--  /home/bob/.twmXztVdT
2008-02-16 22:01:39         172  -rw-r--r--  /home/bob/.twmmTqqq1
2007-11-20 09:51:52        4096  drwxr--r--  /home/bob/.w3m
2008-04-01 20:49:13        4096  drwxr--r--  /home/bob/.gnome2
2008-03-11 12:56:24        4096  drwxr--r--  /home/bob/.purple
2007-11-21 15:22:22        4096  drwxr-xr-x  /home/bob/.pan2
2007-12-13 20:10:30        4096  drwxr--r--  /home/bob/.tsclient
2007-11-27 10:11:26        4096  drwxr-xr-x  /home/bob/.java
2008-01-06 13:51:53        4096  drwxr-xr-x  /home/bob/.config
2008-04-02 07:34:32        4096  drwxr--r--  /home/bob/.gconf
2008-04-02 07:51:04        4096  drwxr--r--  /home/bob/.gconfd
2007-11-16 14:45:22        4096  drwx------  /home/bob/.gnome2_private
2008-04-01 20:49:13        8192  drwxr-xr-x  /home/bob/.nautilus
2007-11-16 14:45:25        4096  drwxr-xr-x  /home/bob/.local
2008-01-06 13:44:00        4096  drwxr-xr-x  /home/bob/.cache
2007-12-05 14:51:52        4096  drwxr--r--  /home/bob/.update-notifier
2007-12-15 13:17:45        4096  drwxr-xr-x  /home/bob/.gnome
2008-03-03 15:20:14        4096  drwxr-xr-x  /home/bob/.gstreamer-0.10
2007-11-27 10:09:01        4096  drwxr-xr-x  /home/bob/.fontconfig
2007-11-16 14:45:38        4096  drwxr--r--  /home/bob/.metacity
2008-03-29 14:54:57        4096  drwxr--r--  /home/bob/.Trash
2007-11-16 14:46:14        4096  drwxr-xr-x  /home/bob/.update-manager-core
2007-12-18 19:20:35        4096  drwxr--r--  /home/bob/.mozilla
2007-11-29 15:17:51        4096  drwxr-xr-x  /home/bob/.mc
2007-11-17 12:46:57        4096  drwxr-xr-x  /home/bob/.themes
2008-01-06 13:51:53        4096  drwxr-xr-x  /home/bob/.icons
2008-01-06 13:46:33        4096  drwxr--r--  /home/bob/.dbus-keyrings
2007-12-11 09:26:27        4096  drwxr--r--  /home/bob/.kde
2007-11-27 10:07:03        4096  drwxr--r--  /home/bob/.macromedia
2008-03-22 12:58:26        4096  drwxr-xr-x  /home/bob/.qt
2008-04-01 10:44:15        4096  drwxr--r--  /home/bob/.openoffice.org2
2007-12-15 13:17:43        4096  drwxr--r--  /home/bob/.loki
2008-02-04 13:12:17        4096  drwxr-xr-x  /home/bob/.wapi
2008-03-18 16:00:33        4096  drwxr-xr-x  /home/bob/.vnc
2007-12-11 09:26:28        4096  drwxr-xr-x  /home/bob/.mcop
2008-02-04 10:54:54        4096  drwxr-xr-x  /home/bob/.gramps
2007-12-18 19:20:48        4096  drwxr--r--  /home/bob/.adobe
2008-03-29 09:43:18        4096  drwx------  /home/bob/.thumbnails
2007-12-27 15:59:26        4096  drwxr-xr-x  /home/bob/.xemacs
2008-01-06 13:34:52        4096  drwxr--r--  /home/bob/.worker
2008-03-30 09:19:00        4096  drwxr-xr-x  /home/bob/.jedit
2008-02-16 18:56:33        4096  drwxr--r--  /home/bob/.synaptic
2008-01-27 14:27:31        4096  drwxr-xr-x  /home/bob/.jogl_ext
2008-02-18 15:25:55        4096  drwxr-xr-x  /home/bob/.xine

Several of the dot directories contain files and directories whose names beg
in with a dot
but which are not found.

Incidentally,  many of those dot directories contain files whose name do *no
t* begin
with a dot but which are not found by
rc=SysFileTree('~/*', f., 'BSL')

Between the two forms only 50% of the files in my home directory are found.

Bob Martin wrote:

> Surely you should see far more output from your program than that?

Yes I do see more files. I was merely briefly pointing out the exceptions to
 what had been written.

--
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

Mark Miesfeld wrote:
> On Mar 31, 11:51 pm, Bob Martin <bob.mar...@excite.com> wrote: 
> ... 
>
> It has been quite a while since I looked at the unix SysFileTree
> code.  Unfortunately, I don't have the time to delve into it right
> now.
>
> If you tested it well, then the behavior you see is the ooRexx 3.2.0
> behavior.  Whether it should or shouldn't be the behavior is another
> question.
>
> Without testing it myself, I would expect SysFileTree to *only*
> descend into subdirectories if the subdirectory matched your
> pattern.

Interesting...I just took a little stroll through the unix sysfiletree
source, and it appears the source of this behavior is the fnmatch()
function that's used to perform the filename matches.  This function
conforms to the XCU specification, which can be found here:

http://www.opengroup.org/onlinepubs...2_13_
01

Of particular importance is section 2.13.3 Patterns Used for Filename
Expansion, where it states:

> #
>
> If a filename begins with a period ( '.' ), the period shall be explicitly
 matched by using a period as the first character of the pattern or immediat
ely following a slash character. The leading period shall not be matched by:
>
>     *
>
>       The asterisk or question-mark special characters
>     *
>
>       A bracket expression containing a non-matching list, such as "[!a]", a range 
expression, such as "[%-0]", or a character class expression, such as "[[:punct:]]"
>
> It is unspecified whether an explicit period in a bracket expression matching list, such
 as "[.abc]", can match a leading period in a filename.

So, according to unix file matching rules, a pattern of "*" should not
locate any files beginning with ".".

Rick



>
> In other words if you used the pattern: .* it should only descend into
> subdirectories whose names start with a dot and it should only find
> files that start with a dot.  If it doesn't, then that might be a bug.
>
> A quick test shows that, on Windows, it does work as I would expect.
>
> However, since on unix-like systems the convention is to not display
> files that start with a dot, declaring what the expected behavior
> should be might not be so straight forward.
>
> --
> Mark Miesfeld

Rick McGuire wrote:

> So, according to unix file matching rules, a pattern of "*" should not
> locate any files beginning with ".".

Thanks for looking up the spec. Then it seems it is working properly.

To get the DOS equiv of '*' I guess use '*' plus a search for '.*' and add t
he two stems together.

At least that is the way it is working for me.

I can not reproduce the OP's finding that option S does not work for a query
 of '.*' as it works as expected for me.

--
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/

in 37765 20080402 125633 Rick McGuire <object.rexx@gmail.com> wrote:
>Mark Miesfeld wrote: 
>
>Interesting...I just took a little stroll through the unix sysfiletree
>source, and it appears the source of this behavior is the fnmatch()
>function that's used to perform the filename matches.  This function
>conforms to the XCU specification, which can be found here:
>
>http://www.opengroup.org/onlinepubs...2_13
_01
>
>Of particular importance is section 2.13.3 Patterns Used for Filename
>Expansion, where it states:
> 
>
>So, according to unix file matching rules, a pattern of "*" should not
>locate any files beginning with ".".
>
>Rick

Thanks for checking that, Rick, but SysFileTree's behaviour is not consisten
t with
that of ls and vdir commands.

vdir -a -R ~/*
finds directories with names not starting with a dot and files in those dire
ctories whose
names do not start with a dot.  It does not find files with names not beginn
ing with a dot
if they are in directories whose name does start with a dot.
This is the same as SysFileTree

BUT

vdir -a -R ~/.*
finds all files in the starting directory and sub-directories, whether their
 names start with
a dot or not.
SysFileTree does not do this.

This means that over 90% of my files are unreachable using SysFileTree.

I can't explain the discrepancy between my results and Michael's.

in 37766 20080402 140143 Michael Lueck <mlueck@lueckdatasystems.com> wrote:
>Rick McGuire wrote:
> 
>
>Thanks for looking up the spec. Then it seems it is working properly.
>
>To get the DOS equiv of '*' I guess use '*' plus a search for '.*' and add 
the two stems together.
>
>At least that is the way it is working for me.
>
>I can not reproduce the OP's finding that option S does not work for a query of '.*
' as it works as expected for me.

Michael,

please run this program and tell me what you get:

#!/usr/bin/rexx

rc = SysFileTree('~/*','DIR.','BSL')
say 'no dot' rc DIR.0
rc = SysFileTree('~/.*','DIR.','BSL')
say 'dot' rc DIR.0

I get
no dot 0 713
dot 0 71

yet according to
vdir -a -R ~/.*
there are thousands of files in my home dir and sub-dirs.

Bob Martin wrote:
> please run this program and tell me what you get:
> I get
> no dot 0 713
> dot 0 71

I get:
mdlueck@aleks:~/bin$ ./hiddenfiles-test.rex
no dot 0 743
dot 0 69

> yet according to
>  vdir -a -R ~/.*
> there are thousands of files in my home dir and sub-dirs.

And vdir output'ed this for example:

/home/mdlueck/.wine/drive_e/apps/notes/vdkhome/style:
total 36
drwxr-xr-x 2 mdlueck mdlueck  150 2006-04-05 10:58 .
drwxr-xr-x 5 mdlueck mdlueck  142 2006-04-05 10:58 ..
-rwxr-xr-x 1 mdlueck mdlueck 1998 2006-04-05 10:58 style.ddd
-rwxr-xr-x 1 mdlueck mdlueck 1802 2006-04-05 10:58 style.did
-rwxr-xr-x 1 mdlueck mdlueck    4 2006-04-05 10:58 style.dts
-rwxr-xr-x 1 mdlueck mdlueck 1066 2006-04-05 10:58 style.pdd
-rwxr-xr-x 1 mdlueck mdlueck 1352 2006-04-05 10:58 style.plc
-rwxr-xr-x 1 mdlueck mdlueck    4 2006-04-05 10:58 style.prf
-rwxr-xr-x 1 mdlueck mdlueck  773 2006-04-05 10:58 style.sid
-rwxr-xr-x 1 mdlueck mdlueck   96 2006-04-05 10:58 style.vgw
-rwxr-xr-x 1 mdlueck mdlueck 1193 2006-04-05 10:58 style.wld
mdlueck@aleks:~/bin$

The only '.something' thing I see in this case is 5 directory levels up happ
ens to be /.wine/

I surly do not think those files qualify as '.*', therefor vdir's findings a
re too broad.

I hope this helps.

--
Michael Lueck
Lueck Data Systems
http://www.lueckdatasystems.com/