Performance problem with a VSAM files

Hi.

Since last w we're experiencing  performance problem when writing to a
VSAM file.
Context:
We have a big master compressed VSAM file (about 30 million records) that we
read to decompress it and writing it into 50 sequentiel files depending on
wich record we read.  For one of them we decided to change it into a VSAM
file for convenience. This is the biggest one with more than 1 million
records.

Problem:
For some reason, when finishing testing we were amazed to see that it took
10 times more CPU time to run that our other program and that much I/O. The
only thing that I could thing of was to put back our older program and run
an IDCAMS to convert it into a VSAM file. Is there a problem with Cobol when
writing big VSAM file or is it just some kind of limitation when using VSAM
files into Cobol program.

Is there another way to do it instead of using  IDCAMS.

Thank you for your input and good night..

J R Duval

You don't say what else (if anything) has changed, but there WAS a *signific
ant*
change to CISIZE "stuff" with z/OS V1R3.

See:
http://www-03.ibm.com/support/techd...ndex/FLASH10206

and have your "performance" or "symin" people see if this is relevant.

--
Bill Klein
wmklein <at> ix.netcom.com
"JRDuval" <jrduval@videotron.ca> wrote in message
news:2yhbe.9244$oy2.391259@weber.videotron.net...
> Hi.
>
> Since last w we're experiencing  performance problem when writing to a
> VSAM file.
> Context:
> We have a big master compressed VSAM file (about 30 million records) that 
we
> read to decompress it and writing it into 50 sequentiel files depending on
> wich record we read.  For one of them we decided to change it into a VSAM
> file for convenience. This is the biggest one with more than 1 million
> records.
>
> Problem:
> For some reason, when finishing testing we were amazed to see that it took
> 10 times more CPU time to run that our other program and that much I/O. Th
e
> only thing that I could thing of was to put back our older program and run
> an IDCAMS to convert it into a VSAM file. Is there a problem with Cobol wh
en
> writing big VSAM file or is it just some kind of limitation when using VSA
M
> files into Cobol program.
>
> Is there another way to do it instead of using  IDCAMS.
>
> Thank you for your input and good night..
>
> J R Duval
>
>

JRDuval wrote:

>Hi.
>
>Since last w we're experiencing  performance problem when writing to a
>VSAM file.
>Context:
>We have a big master compressed VSAM file (about 30 million records) that w
e
>read to decompress it and writing it into 50 sequentiel files depending on
>wich record we read.  For one of them we decided to change it into a VSAM
>file for convenience. This is the biggest one with more than 1 million
>records.
>
>Problem:
>For some reason, when finishing testing we were amazed to see that it took
>10 times more CPU time to run that our other program and that much I/O. The
>only thing that I could thing of was to put back our older program and run
>an IDCAMS to convert it into a VSAM file. Is there a problem with Cobol whe
n
>writing big VSAM file or is it just some kind of limitation when using VSAM
>files into Cobol program.
>
>Is there another way to do it instead of using  IDCAMS.
>
>Thank you for your input and good night..
>
>J R Duval
>
>
>
>
I tested using VSAM for an application that we were going to write years
ago.  It was compared against an IMS data base, a DB2 data base, and
good old sequential data sets (VB format).

Nothing came close to the QSAM data set for performance.

I asked a contractor who had worked at another place that used VSAM
extensively, and she said that it was her impression that the VSAM I/O
was generally done using an Assembler program.  I seemed to find that
adding a record to a control interval (I hope I'm remembering the
terminology correctly) was inefficient in COBOL, but could be controlled
in Assembler.  I expected to be able to add ten records to the data set
for one EXCP (physical I/O), but instead, I was getting two EXCP's for
each record, and one more when a "block" was actually written to the
VSAM data set.

So, if you need that big data set to be VSAM, you are probably on track
to create it as a QSAM data set, then use a utility to load the data
into a VSAM data set.

If it is just for some "convenience", I would forget using VSAM, as it
sounds as if you already have a working application.

"JRDuval" (jrduval@videotron.ca) writes:
>
> Since last w we're experiencing  performance problem when writing to a
> VSAM file.
> Context:
> We have a big master compressed VSAM file (about 30 million records) that 
we
> read to decompress it and writing it into 50 sequentiel files depending on
> wich record we read.  For one of them we decided to change it into a VSAM
> file for convenience. This is the biggest one with more than 1 million
> records.

ESDS, KSDS or RRDS? Were you using ACCESS IS SEQUENTIAL and OPEN OUTPUT?
>
> Problem:
> For some reason, when finishing testing we were amazed to see that it took
> 10 times more CPU time to run that our other program and that much I/O. Th
e
> only thing that I could thing of was to put back our older program and run
> an IDCAMS to convert it into a VSAM file. Is there a problem with Cobol wh
en
> writing big VSAM file or is it just some kind of limitation when using VSA
M
> files into Cobol program.
>
> Is there another way to do it instead of using  IDCAMS.

Syncsort can write sorted output records to a VSAM dataset instead of a QSAM
dataset. I'd be surprised if DFSORT couldn't do the same.

Syncsort can also be used to split files. Again, I'd be surprised if DFSORT
did not also support this. You don't actually have to sort in order to split
.

Colin Campbell (cmcampb@adelphia.net) writes: 
> I tested using VSAM for an application that we were going to write years
> ago.  It was compared against an IMS data base, a DB2 data base, and
> good old sequential data sets (VB format).
>
> Nothing came close to the QSAM data set for performance.

IMS and DB2 use VSAM and OSAM, BSAM or QSAM, so there is no reason to
expect you could do any better than something that doesn't involve the
overhead of a DBMS. Only use a DB if you need to. I recall one FOCUS
application I converted to IMS where they were using a full function
hierarchical FOCUS DB for something that seemed like it could be done
by sorting a transaction file and merging it with an accum file. When
I tracked down the developer he agreed, but felt that doing random access
to a FOCUS DB would be an interesting way to do this. Apparently he had
no idea of the I/O and CPU cost of doing random update.

That said, you should have been able to get close to QSAM SAM-E performance
with IMS or DB2, at least for sequential processing.

Native VSAM sequential access should be similar to QSAM if you specify
the appropriate bufferring, that is a minimum of 2 tracks + 1 data CI
and enough index CIs to hold all a CI from all levels of your index.

DB2 has a sequential pre-fetch of 10 CIs at a time which speeds things up.

IMS doesn't have anything comparable for online VSAM access, but HSSR
can be used to speed up offline sequential access to VSAM or OSAM.

IMS uses QSAM sequential acces for online processing of OSAM DB
datasets. Online OSAM Sequential Bufferring can usually give you average
IWait times of a fraction of a millisecond for most overlapped sequential
reads, reading 10 blocks with each read. Default is 4 sets of 10 blocks
in page fixed storage.

> I asked a contractor who had worked at another place that used VSAM
> extensively, and she said that it was her impression that the VSAM I/O
> was generally done using an Assembler program.  I seemed to find that
> adding a record to a control interval (I hope I'm remembering the
> terminology correctly) was inefficient in COBOL, but could be controlled
> in Assembler.  I expected to be able to add ten records to the data set
> for one EXCP (physical I/O), but instead, I was getting two EXCP's for
> each record, and one more when a "block" was actually written to the
> VSAM data set.
>
You didn't try hard enough then. Were you doing skip sequential processing,
random processing, or a full sequential load or read?

> So, if you need that big data set to be VSAM, you are probably on track
> to create it as a QSAM data set, then use a utility to load the data
> into a VSAM data set.
>
> If it is just for some "convenience", I would forget using VSAM, as it
> sounds as if you already have a working application.

I've often found that sorting and converting to skip sequential processing
can tame IMS, DB2 and VSAM performance problems.

We had an IMS job that used to run past the end of the nightly batch
window. Requesting IMS Sequential Buffering for all IMS DBs got it to
report the total Random, Synchronous Sequential and Overlapped Sequential
reads for each IMS OSAM DB.

One was being processed sequentially and IMS activated OSAM SB for it.

A logically related IMS OSAM DB was being procesed randomly, in the order
that the logical relationships were stored in the DB being processed
sequentially. An average IWait delay of 5 milliseconds multiplied by
millions adds up to hours of IWait delay.

I had the maintenance program write a program to extact all of the possible
segments that might be retrived, sort it to physical sequence for the other
DB using Syncsort with a VSAM KSCS as the SORTOUT. This logical relationship
used symbolic pointers, that is IMS segment key values, rather than IMS
RBA pointers, so I used thems as the final part of the VSAM key after the
RBA of the root in the DB being processed sequentially.

Doing that for a 5 million segment subset in a 2 Gbyte OSAM DB took 20 minut
es.

Skip sequential processing it as VSAM while processing the sequential DB
took almost zero IWait time. A job which regularly ran from 10 pm to past 7 
am
consistently ended before midnight after the change.

A job which had a similar random logical relationship headache that took
all wend also speeded up from this approach. It started out as a once a
year job, which was a nuisance, but not worth changing. When it was changed
to monthly we applied the same technique and got it running overnight
during the w instead of from Friday Evening to Sunday afternoon.

Yes, DFSORT can write sorted output records to a VSAM data set.

Yes, DFSORT can beused to split files in a variety of ways as discussed
on the DFSORT website:

http://www.ibm.com/servers/storage/...tml#
t01

Frank Yaeger - DFSORT Team  (IBM) - yaeger@us.ibm.com
Specialties: ICETOOL, IFTHEN, OVERLAY, Symbols, Migration
=> DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort/

(yaeger@us.ibm.com) writes:
> Yes, DFSORT can write sorted output records to a VSAM data set.
>
> Yes, DFSORT can beused to split files in a variety of ways as discussed
> on the DFSORT website:
>
> http://www.ibm.com/servers/storage/...tm
l#t01
>
> Frank Yaeger - DFSORT Team  (IBM) - yaeger@us.ibm.com
> Specialties: ICETOOL, IFTHEN, OVERLAY, Symbols, Migration
> => DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort/

That is what I expected, but the site where I worked until the end of March
never had DFSORT during my 28 years working there.

This was real pain when Syncsort was part of the picture for performance
problems with IMS utilities such as Prefix Resolution and IMS Change
accumulation.

In article <2yhbe.9244$oy2.391259@weber.videotron.net>,
"JRDuval" <jrduval@videotron.ca> wrote:

> Hi.
>
> Since last w we're experiencing  performance problem when writing to a
> VSAM file.
> Context:
> We have a big master compressed VSAM file (about 30 million records) that 
we
> read to decompress it and writing it into 50 sequentiel files depending on
> wich record we read.  For one of them we decided to change it into a VSAM
> file for convenience. This is the biggest one with more than 1 million
> records.
>
> Problem:
> For some reason, when finishing testing we were amazed to see that it took
> 10 times more CPU time to run that our other program and that much I/O. Th
e
> only thing that I could thing of was to put back our older program and run
> an IDCAMS to convert it into a VSAM file. Is there a problem with Cobol wh
en
> writing big VSAM file or is it just some kind of limitation when using VSA
M
> files into Cobol program.
>
> Is there another way to do it instead of using  IDCAMS.
>
> Thank you for your input and good night..
>
> J R Duval

Have you tried the 'AMP' JCL parameter (AMP goes on the DD statement
pointing to the VSAM file - AMP is used for buffering - - akin to the
DCB BUFNO parameter for sequential files) - using the right AMP
parameters can speed up operations 10 fold! No program changes are
required. Check out a VSAM performance guide for more info - I know
there's at least one IBM manual that talks about it.

--LG

Thank you for the link.
I know for a fact that our CISIZE are not explicitely coded when we create
our VSAM files.
I'm going to take a look at this. That might not hurt.

Thank you for your help.

J R Duval

"William M. Klein" <wmklein@nospam.netcom.com> a écrit dans le message de
news:0Gjbe.5530364$Zm5.855162@news.easynews.com...
> You don't say what else (if anything) has changed, but there WAS a
*significant*
> change to CISIZE "stuff" with z/OS V1R3.
>
> See:
>   http://www-03.ibm.com/support/techd...ndex/FLASH10206
>
> and have your "performance" or "symin" people see if this is relevant.
>
> --
> Bill Klein
>  wmklein <at> ix.netcom.com
> "JRDuval" <jrduval@videotron.ca> wrote in message
> news:2yhbe.9244$oy2.391259@weber.videotron.net... 
a 
that we 
on 
VSAM 
took 
The 
run 
when 
VSAM 
>
>

"Lawrence Greenwald" <lgreenwa@cts.com> a écrit dans le message de
news:lgreenwa-38B79E.17265226042005@chiapp18.algx.net...
> In article <2yhbe.9244$oy2.391259@weber.videotron.net>,
>  "JRDuval" <jrduval@videotron.ca> wrote:
> 
a 
that we 
on 
VSAM 
took 
The 
run 
when 
VSAM 
>
> Have you tried the 'AMP' JCL parameter (AMP goes on the DD statement
> pointing to the VSAM file - AMP is used for buffering - - akin to the
> DCB BUFNO parameter for sequential files) - using the right AMP
> parameters can speed up operations 10 fold! No program changes are
> required. Check out a VSAM performance guide for more info - I know
> there's at least one IBM manual that talks about it.
>
> --LG

Yes I've tried  many combination.
So far the more we could improve the speed  is about 2 times better than it
is now.
Maybe settings are not right. I'll check out the manual if I could get one
at my site.

Thank you for your help.

J R Duval