Reading in file containing Low Values doesn't work correctly

I'm trying to read in a line sequential file in my COBOL program on my
NT machine. The file is a DB2 delimited export file, which is basically
a plain text file.

Two of the columns that are exported contain Low Values. I can verify
that they are low values in the exported data file by looking at the
hex values in an editor.

The data looks something like this:

123456~LL~L~ABCD

The tildes (~) are my delimiters.  I'm representing the Low Lalues with
an (L). So this is 4 columns being exported, the first column
containing 123456, the next containing 2 low values characters, the 3rd
containing 1 low value character, and the final column containing ABCD.

The problem is that when I open then read the file into my COBOL
program, what it reads in looks like this:

123456~L~~ABCD

Why didn't it read all the low values? It didn't even replace them with
a space... It just ignored them as if they didn't exist. Although it
did manage to read one of the low value characters, but not the other
2. Any idea what the problem is? I'm just reading the line into a PIC X
(8000) field. I animate the process and I'm able to see what it reads
in immediately after the read statement. I'm also absolutely sure that
I'm reading in the correct file, because any changes in other data do
show up in the animator.

Timofmars@gmail.com wrote:
> I'm trying to read in a line sequential file in my COBOL program on my
> NT machine. The file is a DB2 delimited export file, which is
> basically a plain text file.
>
> Two of the columns that are exported contain Low Values. I can verify
> that they are low values in the exported data file by looking at the
> hex values in an editor.
>
> The data looks something like this:
>
> 123456~LL~L~ABCD
>
> The tildes (~) are my delimiters.  I'm representing the Low Lalues
> with an (L). So this is 4 columns being exported, the first column
> containing 123456, the next containing 2 low values characters, the
> 3rd containing 1 low value character, and the final column containing
> ABCD.
>
> The problem is that when I open then read the file into my COBOL
> program, what it reads in looks like this:
>
> 123456~L~~ABCD
>
> Why didn't it read all the low values? It didn't even replace them
> with a space... It just ignored them as if they didn't exist.
> Although it did manage to read one of the low value characters, but
> not the other
> 2. Any idea what the problem is? I'm just reading the line into a PIC
> X (8000) field. I animate the process and I'm able to see what it
> reads in immediately after the read statement. I'm also absolutely
> sure that I'm reading in the correct file, because any changes in
> other data do show up in the animator.

Low-values doesn't really make sense in a LINE SEQUENTIAL file. I wouldn't
be surprised if the results were unpredictable.

I created a process to export the data from my company's DB2 tables and
read it into our backup program which creates a custom backup file. The
process using the exported data to create a backup is much faster than
our old backup process.

Unfortunately, because these fields with low values are not being read
in correctly, we are getting some bad data (spaces in fields where low
values should be).

Is there perhaps a way to have all low values exported be represented
by a different character?

Now I am trying to look into using exported data in IXF (Integrated
exchange format) instead of DEL (Delimited ASCII format). The IXF
format looks to be record sequential, which is fine. I think the only
problem is that it starts by exporting the table attributes to the
export file, followed by the data. I only need the data and I'm not
sure how I could get my program to process only the data.

On Jan 25, 3:44 am, "HeyBub" <heybubNOS...@gmail.com>

> Low-values doesn't really make sense in a LINE SEQUENTIAL file. I wouldn't
> be surprised if the results were unpredictable.

The results are entirely predictable, but the prediction needs to be
based on what the appropriate manual states will occur.

In the case of MicroFocus the default behaviour is that when writing a
line sequential record it will be processed and all control characters
will be prefixed by a null (which is the default 'low-value').  On
reading, the null is used as a signal the the next character is a
control character that needs to be preserved. This allows a file to be
written and read that will contain control characters (eg tab, line
feed).

In the case in point the nulls are being seen as guard characters and
are dropped, except where two nulls in a row the second is retained.

To change this behaviour set the 'N' switch to off (-N).

Reading the manual makes the behaviour predictable.

Richard wrote:
> On Jan 25, 3:44 am, "HeyBub" <heybubNOS...@gmail.com>
> 
>
> The results are entirely predictable, but the prediction needs to be
> based on what the appropriate manual states will occur.
>
> In the case of MicroFocus the default behaviour is that when writing a
> line sequential record it will be processed and all control characters
> will be prefixed by a null (which is the default 'low-value').  On
> reading, the null is used as a signal the the next character is a
> control character that needs to be preserved. This allows a file to be
> written and read that will contain control characters (eg tab, line
> feed).
>
> In the case in point the nulls are being seen as guard characters and
> are dropped, except where two nulls in a row the second is retained.
>
> To change this behaviour set the 'N' switch to off (-N).
>
> Reading the manual makes the behaviour predictable.

Here's another prediction:

"Line-sequential files are files that reside in the hierarchical file system
(HFS) and that contain only printable characters and certain control
characters as data."

Note the "... only printable characters..." qualifier. Low-values is not
printable, nor is it a normal control character like TAB or Linefeed.

http://publib.boulder.ibm.com/infoc...oc/tplsq01a.htm

Why a file system (like the one you mention) would go to the trouble of
adding LV bytes only to remove them later is a mystery and evidence of
carrot eating. If the file system can detect a control character for the
purpose of preceeding it with a LV byte, why does it need the LV byte to
identify the very same character when it reads the file?

Yep, definite carrot eating.

On Jan 25, 9:33 am, "HeyBub" <heybubNOS...@gmail.com> wrote:
> Richard wrote: 
> 
> 
> 
> 
> 
> 

> Here's another prediction:
>
> "Line-sequential files are files that reside in the hierarchical file syst
em
> (HFS) and that contain only printable characters and certain control
> characters as data."
>
> Note the "... only printable characters..." qualifier. Low-values is not
> printable, nor is it a normal control character like TAB or Linefeed.
>

In fact there is a list of allowable control characters. The prediction
is that in those files in that system a null will never occur and tabs,
for example, will be passed to the program in the record area (as
specified in the manual).

> http://publib.boulder.ibm.com/infoc...ex.jsp?topic...


> Why a file system (like the one you mention) would go to the trouble of
> adding LV bytes only to remove them later is a mystery and evidence of
> carrot eating. If the file system can detect a control character for the
> purpose of preceeding it with a LV byte, why does it need the LV byte to
> identify the very same character when it reads the file?
>
> Yep, definite carrot eating.

Because, upon reading a line sequential file, certain control
characters, such as tab, line feed and carriage return will be
processed by the file system and will not be passed into the program.
tabs will be expanded and cr/lf will signal the end of record.

If it is required to have these characters passed from one program to
another then the low-values prefix acts as an escape character.  The
low value will be dropped and the character following will be passed to
the program as data.

If that behaviour is unrequired then the 'N' switch can be turned off
by a command-line entry or an environment variable (COBSW=-N) or by the
program using called routines to change the setting (as specified in
the manual).

That is to say that 'low-values in line sequential files' may not make
sense on your system, but your system is not the whole world and they
may make sense on other systems.

As it happens I have no use for the nulls in that way and have always
had the N switch off, but that is the reason they are used.

DB2 is famous for placing LV in fields that are initialized but not
loaded.  I have suggested to DBAs on mf that spaces be used for
initialization.

On mf I have requested that all report programs using columns from DB2
row have code

IF column LESS THAN SPACES
MOVE SPACES TO ofield
ELSE
MOVE COLUMN TO ofield
END-IF

If the file with LV is transmitted to some servers, the record/line is
truncated at the LV.  Pita on reports being distributed.

On Jan 23, 2:26 pm, Timofm...@gmail.com wrote:
> I'm trying to read in a line sequential file in my COBOL program on my
> NT machine. The file is a DB2 delimited export file, which is basically
> a plain text file.
>
> Two of the columns that are exported contain Low Values. I can verify
> that they are low values in the exported data file by looking at the
> hex values in an editor.
>
> The data looks something like this:
>
> 123456~LL~L~ABCD
>
> The tildes (~) are my delimiters.  I'm representing the Low Lalues with
> an (L). So this is 4 columns being exported, the first column
> containing 123456, the next containing 2 low values characters, the 3rd
> containing 1 low value character, and the final column containing ABCD.
>
> The problem is that when I open then read the file into my COBOL
> program, what it reads in looks like this:
>
> 123456~L~~ABCD
>
> Why didn't it read all the low values? It didn't even replace them with
> a space... It just ignored them as if they didn't exist. Although it
> did manage to read one of the low value characters, but not the other
> 2. Any idea what the problem is? I'm just reading the line into a PIC X
> (8000) field. I animate the process and I'm able to see what it reads
> in immediately after the read statement. I'm also absolutely sure that
> I'm reading in the correct file, because any changes in other data do
> show up in the animator.

Ok, I found what I needed to do in the Micro Focus documentation. I
need to set a COBSW environment variable in either the command line,
script, or environment card.

set COBSW=(-N)

That did it and it fixed the problem.

Thanks a lot.

On Jan 23, 4:29 pm, "Rick Smith" <ricksm...@mfi.net> wrote:
> Well, I am not familiar with Net Express, having stayed
> with the OLD Micro Focus COBOL 3.2; but I used it
> from the command line as, ...
>
> xm run (-N) readnull
>
> So it could possibly be placed in a script that you run.
>
> For applications, I set the runtime switches in a configuration
> file (.cfg) that the COBOL runtime uses at start up.
>
> Perhaps, someone familiar with Net Express could provide
> more explicit information.