Home > Archive > Unix Programming > June 2007 > how to append to a logfile that has a max size
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
how to append to a logfile that has a max size
|
|
|
| hello everyone,
i am trying to log some information from an application to a log
file.
i want to be able to 'append' to that file information at frequent
interval.
however, once the size of the file (or the lines written in the file)
exceeds
a certain limit i want the oldest entry to be dumped, all the others
to be
'shifted' towards the beginning of the file and a new entry to be
appended.
someone told me that instead of doing that from within my application
(its an algorithm really and so looping into a disk operation would
cause it to slow down the loop endlessly), i could use mkfifo since
this is what im really doing there.
i looked into mkfifo manual, both in manual chapter 1 and chapter 3
(the c++ function).. but i can't seem to find any
setting that would allow me to just set max size of the file so that
then i would only worry about appending entries and the routine would
worry about dumping the oldest (first IN) entry and moving all the
middle entries...
can i indeed use mkfifo? if not is there another alternative?
thank you very much for your help
nass
| |
| phil-news-nospam@ipal.net 2007-06-14, 7:06 pm |
| On Thu, 14 Jun 2007 09:57:57 -0000 nass <athanasios.silis@gmail.com> wrote:
| hello everyone,
| i am trying to log some information from an application to a log
| file.
| i want to be able to 'append' to that file information at frequent
| interval.
| however, once the size of the file (or the lines written in the file)
| exceeds
| a certain limit i want the oldest entry to be dumped, all the others
| to be
| 'shifted' towards the beginning of the file and a new entry to be
| appended.
|
| someone told me that instead of doing that from within my application
| (its an algorithm really and so looping into a disk operation would
| cause it to slow down the loop endlessly), i could use mkfifo since
| this is what im really doing there.
No it isn't.
| i looked into mkfifo manual, both in manual chapter 1 and chapter 3
| (the c++ function).. but i can't seem to find any
| setting that would allow me to just set max size of the file so that
| then i would only worry about appending entries and the routine would
| worry about dumping the oldest (first IN) entry and moving all the
| middle entries...
A "FIFO" is not a file in the normal sense. It is a pipe between processes
that has a reference point in the filesystem namespace. Think of it as a
communications channel.
You would have to implement a program to run at the other end of the FIFO
to read data that was written by the program which produces the lines you
want to log.
| can i indeed use mkfifo? if not is there another alternative?
You can if there is a daemon process or some such thing at the other end
to read the data and do all the work. You'll need to implement it.
In some cases a FIFO with a daemon at the other end is the best way to go.
But in some other cases, there are alternatives that might be better for
you.
Consider a circular file. This is a file of usually a fixed size (known
to the program) in which the entries are written with a timestamp on each.
When the end of the file is reached, the writing resumes back at the start.
A line of output may be split with the first part of it at the end of the
file and the rest at the beginning. Make sure the timestamps have some
distinctive data as part of them, such as a special character to begin
each line. Then a special program can read this file by looking for the
special character, and from the timestamps determine where the oldest and
newest lines are.
Or consider a directory of separate files. After so much data is written
to a file, close it and reopen a new one. Dates and times make a convenient
way to name these files, especially if done in ISO format (e.g. year first,
month next, date/hour/etc later) so the names come out in the right order
when sorted. Then either in the program that created them, or in another
program that runs periodically, clean out the oldest files, either by
permanently deleting them if that is appropriate, or moving them out to
an archive if longer storage is needed. Hint: if doing an archive, do
the archive of everything, not just what is to be removed next.
With the circular file you need a special program to read the data in the
right order.
With the directory of files, you can do "cat *" from within that directory
and see all the log data in correct order.
--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-06-14-0704@ipal.net |
|------------------------------------/-------------------------------------|
| |
|
| Thank you for your input.
indeed what im asking is only a segment of a bigger picture that u
laid out..
in truth is algorithm is a monitoring unit. the user is given the
option of logging in circular or linear mode. circular is by choice of
the client the mode that u described with the directories. new files
are created every x hours/days according to user preference. but in
linear mode there is only 1 file and data are appended into it. now,
since the unit will be working without frequent user intervention
(every 3-4 months) the disk may become full (disk is a 128MB flash
memory hosting a debian OS, the application and the generated log
files) and so for the time being, in linear mode, logging is halted.
i wanted to implement some sort of mechanism that shifts the data so
that even in linear mode logging will never be halted. -as reference,
in circular mode i just delete the oldest files in the folder. note
that even for logging i use the asynchronous writing routines to avoid
slowing down the algorithm loop execution.
anyway, i could perhaps dispatch a thread too to do what the
combination FIFO + daemon would do, right? i was just hoping that will
all this logs that exist, there is some library for that:)
nass
On Jun 14, 3:20 pm, phil-news-nos...@ipal.net wrote:
> On Thu, 14 Jun 2007 09:57:57 -0000 nass <athanasios.si...@gmail.com> wrote:
>
> | hello everyone,
> | i am trying to log some information from an application to a log
> | file.
> | i want to be able to 'append' to that file information at frequent
> | interval.
> | however, once the size of the file (or the lines written in the file)
> | exceeds
> | a certain limit i want the oldest entry to be dumped, all the others
> | to be
> | 'shifted' towards the beginning of the file and a new entry to be
> | appended.
> |
> | someone told me that instead of doing that from within my application
> | (its an algorithm really and so looping into a disk operation would
> | cause it to slow down the loop endlessly), i could use mkfifo since
> | this is what im really doing there.
>
> No it isn't.
>
> | i looked into mkfifo manual, both in manual chapter 1 and chapter 3
> | (the c++ function).. but i can't seem to find any
> | setting that would allow me to just set max size of the file so that
> | then i would only worry about appending entries and the routine would
> | worry about dumping the oldest (first IN) entry and moving all the
> | middle entries...
>
> A "FIFO" is not a file in the normal sense. It is a pipe between processes
> that has a reference point in the filesystem namespace. Think of it as a
> communications channel.
>
> You would have to implement a program to run at the other end of the FIFO
> to read data that was written by the program which produces the lines you
> want to log.
>
> | can i indeed use mkfifo? if not is there another alternative?
>
> You can if there is a daemon process or some such thing at the other end
> to read the data and do all the work. You'll need to implement it.
>
> In some cases a FIFO with a daemon at the other end is the best way to go.
> But in some other cases, there are alternatives that might be better for
> you.
>
> Consider a circular file. This is a file of usually a fixed size (known
> to the program) in which the entries are written with a timestamp on each.
> When the end of the file is reached, the writing resumes back at the start.
> A line of output may be split with the first part of it at the end of the
> file and the rest at the beginning. Make sure the timestamps have some
> distinctive data as part of them, such as a special character to begin
> each line. Then a special program can read this file by looking for the
> special character, and from the timestamps determine where the oldest and
> newest lines are.
>
> Or consider a directory of separate files. After so much data is written
> to a file, close it and reopen a new one. Dates and times make a convenient
> way to name these files, especially if done in ISO format (e.g. year first,
> month next, date/hour/etc later) so the names come out in the right order
> when sorted. Then either in the program that created them, or in another
> program that runs periodically, clean out the oldest files, either by
> permanently deleting them if that is appropriate, or moving them out to
> an archive if longer storage is needed. Hint: if doing an archive, do
> the archive of everything, not just what is to be removed next.
>
> With the circular file you need a special program to read the data in the
> right order.
>
> With the directory of files, you can do "cat *" from within that directory
> and see all the log data in correct order.
>
> --
> |---------------------------------------/----------------------------------|
> | Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
> | first name lower case at ipal.net / spamtrap-2007-06-14-0...@ipal.net |
> |------------------------------------/-------------------------------------|
| |
|
|
|
| On Jun 14, 4:47 pm, Duncan Muirhead <d...@csl.co.uk> wrote:
> On Thu, 14 Jun 2007 12:57:20 +0000, nass wrote:
>
>
> <snip>
> Could you get away with writing two files all the time? For
> example (if the max file size is say 1MB) start writing to file A; when
> that gets to 0.5MB start writing *also* to file B. When file A gets to 1MB
> close it, reopen it (for write, ie throw the old contents away) and start
> writing at 0 offset again, and so on. That way you never use more than
> 1.5MB of space, but always have at least the last 0.5MB of log. But you
> are doing twice as much file io..
1.5x the max size of the file size can't be acceptable in this
situation,
because i'm trying to use the whole disk space for that file. it cant
be that
i'll use 1/3 of this space for the 2nd file...
as for using mkfifo for piping data to another deamon... i searched
around
but didn't find any information regarding the performance of this.
meaning,
that since mkfifo is 'applied' onto a file on disk, does it mean that
the
communicating applications do so through this file on disk? or all
comms
is carried out at memory and therefore is 'fast' ?
| |
|
| On Jun 14, 4:47 pm, Duncan Muirhead <d...@csl.co.uk> wrote:
> On Thu, 14 Jun 2007 12:57:20 +0000, nass wrote:
>
>
> <snip>
> Could you get away with writing two files all the time? For
> example (if the max file size is say 1MB) start writing to file A; when
> that gets to 0.5MB start writing *also* to file B. When file A gets to 1MB
> close it, reopen it (for write, ie throw the old contents away) and start
> writing at 0 offset again, and so on. That way you never use more than
> 1.5MB of space, but always have at least the last 0.5MB of log. But you
> are doing twice as much file io..
1.5x the max size of the file size can't be acceptable in this
situation,
because i'm trying to use the whole disk space for that file. it cant
be that
i'll use 1/3 of this space for the 2nd file...
as for using mkfifo for piping data to another deamon... i searched
around
but didn't find any information regarding the performance of this.
meaning,
that since mkfifo is 'applied' onto a file on disk, does it mean that
the
communicating applications do so through this file on disk? or all
comms
is carried out at memory and therefore is 'fast' ?
|
|
|
|
|
dam-hussein-execution.org/de-de-execution-l-