Should COBOL be lockedUP for good?...

Hello EveryOne,

I am developing a softWare application for a multiUser operation
in a netWork environment, the software should facilitate dataEntry
functions, and provide capabilities for an information system
as well. Also, the program has to handle concurrent Read-Write
intensive requests, withOut any bottlenecks or performance blips.

However, I realize that shared dataFiles are indispensable in such
an environment and with such an application, and consideration
must be put forward to correctly create and design sharedFiles,
vis-a-vis fileHandling mechanism.

So, I got the source code cut out with some moot questions that
I would like to share with you, hopefully you can shed some lights
on the layOut, limits and pitfalls, when developing multiTasking
and multiUser softWare programs.

1). Should shared dataFiles use records locking schemes?

2). Should shared dataFiles use data compression mechanism?

3). Should shared dataFiles (largeSize) be split into several
smaller files?

4). Should shared dataFiles be handled by an I/O dynamically
called module per file, that service all processes running
concurrently?

5). Should shared dataFiles be handled by an I/O independent
module per file, that runs separately and communicates with
the application thru shared memory, and service all processes
running concurrently?


Regards, Kellie.

> 1). Should shared dataFiles use records locking schemes?

Absolutely.  Though different techniques can be used for different
applications and different business needs.

For example in an oin-line order entry system it needs to be decided
how the 'race' for stock should be handled.  If there is only a small
number of products then you cannot afford to lock the stock records
over a data-entry interaction because there will be too many clashes.
If there may be stock shortages over many different products, however,
then the stock record can be locked while the available stock is held
awaiting quantity to be entered.

You should also consider using a locking hierarchy. For example it may
be that to work on debtor's transactions every program that does so
must first lock the debtors' master record.  If all correctly follow
the rule then a requirement for multiple locks (kept locks) can be
avoided.

Also ensure that you do specific UNLOCKs when finished.  Typically you
may read through a section of records, such as all order lines for an
order.  It is necessary to read until the order number changes, there
is a risk that the program will leave a lock on the first record of the
next order.

> 2). Should shared dataFiles use data compression mechanism?

Compression probably will use less disk space (it need not) and this
means fewer disk transfers to access a set of data which may make
processing faster.

> 3). Should shared dataFiles (largeSize) be split into several
>    smaller files?

Depends on the value of 'large'.  If I thought that in a few years the
data file may get to be a Gigabyte then I may consider using some
sectioning of data.

> 4). Should shared dataFiles be handled by an I/O dynamically
>    called module per file, that service all processes running
>     concurrently?

That depends on the architecture of the system.  Are you, for example,
going to get just one 'dynamically called module' or will you actually
get an iteration of the module for each run-unit (with or without
shared code) ?

Generally you won't get a _CALLed_ routine to 'service all processes'.


Perhaps you are asking if you should have a separate 'file server'
run-unit that interacts with the programs using IPC in some way, as in
Q5.

That will depend on how you need to balance your network traffic. For
example in a networked system with the programs running on client
machines if a search is to be done by examining each record then the
_whole_ file must be sent across the network one record at a time
(plus, probably, thousands of index blocks).

If a server program runs on the file server and this does searches then
only the 'found' records are sent to the client program.

However, if the architecture of the system is centralised where all the
programs run on an 'application server' accessed via terminal systems
(eg X or putty to a Linux application or RCA clients to a TSE server)
then all disk traffic is local within the box and only screen images go
accross the network.

> 5). Should shared dataFiles be handled by an I/O independent
>     module per file, that runs separately and communicates with
>     the application thru shared memory, and service all processes
>     running concurrently?

'Shared memory' implies that all the run-units are in one machine. In a
client-server situation you need to use IPC that is _not_ shared
memory.

Hello Kelly

While I agree with Richard's good advice, I would suggest that you
consider using a relational database with SQL instead of using files
for such an application.  The sharing and  locking processes are
arguably more comprehensive and less prone to programmer error such as
forgetting to lock, share, unlock when required.  They also tend to
come with a fairly good set of management, backup and recovery
features.  They also tend to scale up well, especially when the data is
eventually accessed from a variety of programs written in a variety of
languages on a variety of platforms.

I am not saying that a file system is not appropriate in your
circumstances, but that you should at least consider the database
alternative and weigh the pros and cons.  For example, you may not know
how to use a database, or  you have to work with pre-existing files, or
a file system might be faster, etc..

regards, Robert

Hi Richard,

> Absolutely.  Though different techniques can be used for different
> applications and different business needs.

I would like to use a manual lock with kept locks for muliple record
locking,
however, what would happen if some records are locked, and the
workStation
experience a powerOutage of some sort? I know the operating system will
release
the records automatically, but I donn't know after how long, though?

> Compression probably will use less disk space (it need not) and this
> means fewer disk transfers to access a set of data which may make
> processing faster.

doesn't compression de-compression of the file records causes some
performance
problems or read/write operation delay?

> That depends on the architecture of the system.  Are you, for
example,
> going to get just one 'dynamically called module' or will you
actually
> get an iteration of the module for each run-unit (with or without
> shared code) ?

I am trying to choose between a dynamically called or an independent
I/O module.
Dynamically speaking though, it will be an iteration of the same module
for each
runUnit within the same application. I want to run the interFace
program on a client
machine, while the application runs on the file server, so the shared
dataFiles can
be used efficiently on the netWork. Also, am looking for a way to
reduce the
network traffic to exactly one roundTrip per I/O operation, if I can do
that with my
cobol code. Do you consider that approach wise?

> 'Shared memory' implies that all the run-units are in one machine. In
a
> client-server situation you need to use IPC that is _not_ shared
> memory.

All the runUnits are in one machine for each end-user --- provided
though, the main
application will run on the file server, where the main file searching
will take place
accordingly. Doesn't that approach reduces the roundTrip delay in the
network
connections and in the dataRate transfer as well??

Regards, Kellie.

Hello Robert,

You can lead a horse to the water, however, if you can make him float,
you got
something pretty good. :---))

you are raising an exellent point though --- however, I think that a
skillful
programmer with clairvoyant thinking, and a well thought program design
approach,
can produce or at least emulate a large database management system
functionality. Not to mention also, COBOL does an exellent job of
BackUp,
Restore and archive any type of files.

Personally, I think the only weakLink in my indexed file design, is the
actual
indexed files per se, indexed files tend to bog down when they get very
large in
size, however, COBOL systems can have a workAround that issue pretty
easy
as Richard pointedOut above.

Regards, Kellie.

Kellie,
I think you are mixing up one implementation (probably Micro Focus) with
"COBOL" when you say,

"COBOL does an excellent job of  BackUp,  Restore and archive any type of
files."

COBOL, itself,  has NO "backup and restore" facility and its "restart"
capabilities are limited (at best).  Specific vendors have been FORCED to cr
eate
"solutions" to this serious lack. (Just as Standard COBOL - up until the '02
Standard had *no* file-sharing or record-locking - and even in the '02 Stand
ard,
this is "processor dependent" - aka "OPTIONAL").

If one is designing an application "from scratch" - I would certainly go wit
h a
"standard" RDB (Relational Database) system over a COBOL-only solution
any-day-of-the-w.  These solutions (besides built-in "recovery" systems) 
also
provide built-in (or relatively common) user-interface tools that may or may
 NOT
be available for COBOL files.


--
Bill Klein
wmklein <at> ix.netcom.com
"Kellie Fitton" <KELLIEFITTON@YAHOO.COM> wrote in message
news:1116279557.772145.199780@g49g2000cwa.googlegroups.com...
> Hello Robert,
>
> You can lead a horse to the water, however, if you can make him float,
> you got
> something pretty good. :---))
>
> you are raising an exellent point though --- however, I think that a
> skillful
> programmer with clairvoyant thinking, and a well thought program design
> approach,
> can produce or at least emulate a large database management system
> functionality. Not to mention also, COBOL does an exellent job of
> BackUp,
> Restore and archive any type of files.
>
> Personally, I think the only weakLink in my indexed file design, is the
> actual
> indexed files per se, indexed files tend to bog down when they get very
> large in
> size, however, COBOL systems can have a workAround that issue pretty
> easy
> as Richard pointedOut above.
>
> Regards, Kellie.
>

Hi Bill,

the functionality of BackUp, Restore and Archive does NOT need to be a
built-in
function within the COBOL language --- a well designed cobol module
with a good
algorithm functions, can do the job very well as needed. My compiler
(NetExpress)
provide a file-sharing and an exellent locking mechanism as a built-In
features.

However, regarding the "Restart" mode, perhaps you can give me some
more
information, or highlight the question?

Regards, Kellie.

> however, I think that a skillful programmer with clairvoyant
thinking, and a well thought
> program design approach, can produce or at least emulate a large
database management
> system functionality.

That _may_ be true (but probably isn't), and I think that MicroFocus
may have got quite close with its 'file share' product some years ago.
But I very much doubt that it would be worth while.

I suspect that you are not aware of what the full range of DBMS
functionality actually is.

> Not to mention also, COBOL does an exellent job of BackUp,
> Restore and archive any type of files.

It does ? How so ? What _COBOL_ features actually do this ?

> Personally, I think the only weakLink in my indexed file design, is
the
> actual indexed files per se,

Of course it may be that you only see one "only weaklink" because you
haven't found the other problems yet.

For example you blithely assume that locks 'will be released when
clients stop', you assume that network traffic is the bottleneck, or
that index files 'bog down'.  You need to work with real cases to see
why they arrived at the solutions that they have and what problems
these solutions don't cater for.

If you do have a real case then comments need to be made based on what
it actually is, so you need to outline the requirement.

Kellie, you had asserted that:
 

Now we find that "a well designed cobol module with a good algorithm
functions, can do the job very well as needed".

In other words you can write your own which may, or may not, be
'excellent' depending on how good the programmer is. But it is not
_COBOL_ that does this, and it may not be 'any type of file'. How well
does COBOL deal with .xls ? or .doc ? so what does 'any type of file'
actually mean ?

> My compiler (NetExpress) provide a file-sharing and an exellent
locking mechanism as a
> built-In features.

You may start to call it 'excellent' after you have demonstrated how it
behaves when, say, the client machine is powered down after having
locked some records in the server.

You may also need to demonstrate that COMMITT and ROLLBACK work as may
be expected (ie are not treated as comments) before calling it any more
than merely adequate.

> However, regarding the "Restart" mode, perhaps you can give me some
more
> information, or highlight the question?

A 'Restart' is where a program has been updating a file and the program
crashes, or otherwise fails to complete, due to, say, power fail or
data error.  The 'restart' would continue processing from the failure
point or from a checkpoint a short time beforehand, thus saving having
to do a restore and rerun everything up to that point.

With DBMS systems it is relatively easy to emulate restarts using
transactions and commit. File based systems make this more complex.

You may also want to contemplate what happens when the server goes down
(bluescreens or powers down). Certain disk blocks may. or may not, have
been actually written to the disk. They may have been in the client's
network cache, or the server's disk cache or the drive's buffer.  As
there is no guarantee that disk blocks are written in any specific
order an index may have been written but the record not, or vice versa.

Kellie Fitton wrote:
>
> 1). Should shared dataFiles use records locking schemes?

Sure.

>
> 2). Should shared dataFiles use data compression mechanism?

Ummm.... Maybe. In addition to the obvious advantages/divantages,
compressed data is impossible to debug and cannot be "read" by an external,
generic program.

>
> 3). Should shared dataFiles (largeSize) be split into several
>    smaller files?

Again, maybe. What's "large?" In the best case your trading (how much)
programming and error potential for (how much) execution efficiency? Three
months of additional programming and debugging effort vs saving thirty
seconds per day probably is not worth it.

>
> 4). Should shared dataFiles be handled by an I/O dynamically
>    called module per file, that service all processes running
>    concurrently?

Probably not.

>
> 5). Should shared dataFiles be handled by an I/O independent
>    module per file, that runs separately and communicates with
>    the application thru shared memory, and service all processes
>    running concurrently?

I don't know.The issue has never come up.

>
>
> Regards, Kellie.