manipulating csv file fields through perl

Hi,

I have a csv file. I wanted to do some calculations on some of its fields,
like multiplying the 7th field with the 13th field and overwriting the 13th
field with the answer of my calculation.

Regarding this, can I do the calculations on the input file and overwrite it
after calculating, or I will have to open the output file and write into it?
I am asking this because I will have several files in the directory. So, I
will have to read the directory using readdir,  and process each file. It is
better if I open the file in read-write mode, process it and overwrite the
file. Just wanted to know if it is safe?

Please guide on how to get started with this?

Also can I do something like below:-


while ($line=readline($IN_FILE))
{
my @cdr=split (/,/, $line) ;
$cdr[13] = $cdr[6]*5 ; ###Can I do something like this
}

Thanks,
Mihir

2007/8/24, Mihir Kamdar <kamdarmihir06@gmail.com>:

>         $cdr[13] = $cdr[6]*5 ; ###Can I do something like this

sure,why can't?

On 8/24/07, Jeff Pang <rwwebs@gmail.com> wrote:
>
> 2007/8/24, Mihir Kamdar <kamdarmihir06@gmail.com>:
> 
>
> sure,why can't?

Hi,

Please look at my code below and comment. I am trying to manipulate 13th
field of my record. But I am not getting the desired result in the output.
The output file is the same as the input file.

#!/usr/bin/perl

use warnings ;

my $file_path =
'/home/user71/RangerDatasource/Customization/TelekomMalaysia/Scripts/Tests/c
progs/files/ratetest';
my $write_path =
'/home/user71/RangerDatasource/Customization/TelekomMalaysia/Scripts/Tests/c
progs/files/rateop'
;
my %times ;
my $continue  = 1;

$SIG{INT} = $SIG{TERM} = sub { $continue = 0 };

while ($continue) {
opendir my $dh, $file_path or die $!;
while (my $file = readdir $dh) {
my $fname = "$file_path/$file" ;
next unless -f $fname;
unless (exists $times{$file}){
my $line;
open (my $IN_FILE,"<","$file_path/$file") or die
$!." file not found" ;

while ($line=readline($IN_FILE))
{
my @cdr=split (/,/, $line) ;
$cdr[13] = $cdr[6]*5 ;
$hash{"@cdr[2,3,6,7]"}=$line;
}
close $IN_FILE ;

open (my $OUT_FILE,">","$write_path/$file.out") or
die $!;
while (my($key, $value) = each %hash)
{
print $OUT_FILE $value;
}
close $OUT_FILE;
}
}
closedir $dh ;
}



Thanks,
Mihir

Your idea should work pretty well assuming that you are 100% sure that the
thing in field 7 really is a number, you might get strange results if field
7 is empty or somehting else then a number.

As for how to handle the files that is really up to you and the environment
you work in... if your files are masive things of several hundreds of Mb
each you might not want to place them completely in memory, if they are
changed by other processes you might want to use some form of a locking
scheme while handeling them. Etc, etc...



On 8/24/07, Mihir Kamdar <kamdarmihir06@gmail.com> wrote:
>
> Hi,
>
> I have a csv file. I wanted to do some calculations on some of its fields,
> like multiplying the 7th field with the 13th field and overwriting the
> 13th
> field with the answer of my calculation.
>
> Regarding this, can I do the calculations on the input file and overwrite
> it
> after calculating, or I will have to open the output file and write into
> it?
> I am asking this because I will have several files in the directory. So, I
> will have to read the directory using readdir,  and process each file. It
> is
> better if I open the file in read-write mode, process it and overwrite the
> file. Just wanted to know if it is safe?
>
> Please guide on how to get started with this?
>
> Also can I do something like below:-
>
>
> while ($line=readline($IN_FILE))
>    {
>        my @cdr=split (/,/, $line) ;
>        $cdr[13] = $cdr[6]*5 ; ###Can I do something like this
>    }
>
> Thanks,
> Mihir
>

2007/8/24, Mihir Kamdar <kamdarmihir06@gmail.com>:
>                                 $cdr[13] = $cdr[6]*5 ;
>                                 $hash{"@cdr[2,3,6,7]"}=$line;

Hello,

I checked the codes,but I'm not sure what's the meanings of those 2 lines ab
ove.
I can't see the logic relation with the 2 lines.
Yes $cdr[13] = $cdr[6]*5 has changed the array element,but didn't
change the line content.Are you sure this?
Also as I've said, @hash{@arr} is different with $hash{"@arr"}.see below,

$ perl -Mstrict -Mwarnings -MData::Dumper -e 'my %hash ;@hash{qw/a b
c/} = 111;print Dumper \%hash'
$VAR1 = {
'c' => undef,
'a' => 111,
'b' => undef
};

$ perl -Mstrict -Mwarnings -MData::Dumper -e 'my %hash ;$hash{qw/a b
c/} = 111;print Dumper \%hash'
$VAR1 = {
'abc' => 111
};

Of course it is....

Your %hash gets filled with a structure that looks like this:
{ KEY               => VALUE }
{ "@cdr[2,3,6,7]"=> $line    }

Then you take all the values and write them out to a file. Since you never
changed the $line variable you should get the same result in your out file
as you had in the in file. It is good to see perl still works.

Try the following just before the line: $hash{"@cdr[2,3,6,7]"}=$line;

$line = join(/,/,@cdr);

This way you fill the $line variable with the contents of the just modified
array this should get you the desired result (though there are more stylish
ways of writting it it will get you the result you are looking for)

Rob.


On 8/24/07, Mihir Kamdar <kamdarmihir06@gmail.com> wrote:
>
> On 8/24/07, Jeff Pang <rwwebs@gmail.com> wrote: 
>
> Hi,
>
> Please look at my code below and comment. I am trying to manipulate 13th
> field of my record. But I am not getting the desired result in the output.
> The output file is the same as the input file.
>
> #!/usr/bin/perl
>
> use warnings ;
>
> my $file_path =
>
> '/home/user71/RangerDatasource/Customization/TelekomMalaysia/Scripts/Tests
/cprogs/files/ratetest';
> my $write_path =
>
> '/home/user71/RangerDatasource/Customization/TelekomMalaysia/Scripts/Tests
/cprogs/files/rateop'
> ;
> my %times ;
> my $continue  = 1;
>
> $SIG{INT} = $SIG{TERM} = sub { $continue = 0 };
>
> while ($continue) {
> opendir my $dh, $file_path or die $!;
>        while (my $file = readdir $dh) {
>                my $fname = "$file_path/$file" ;
>                next unless -f $fname;
>                unless (exists $times{$file}){
>                        my $line;
>                        open (my $IN_FILE,"<","$file_path/$file") or die
> $!." file not found" ;
>
>                        while ($line=readline($IN_FILE))
>                        {
>                                my @cdr=split (/,/, $line) ;
>                                $cdr[13] = $cdr[6]*5 ;
>                                $hash{"@cdr[2,3,6,7]"}=$line;
>                        }
>                        close $IN_FILE ;
>
>                        open (my $OUT_FILE,">","$write_path/$file.out") or
> die $!;
>                        while (my($key, $value) = each %hash)
>                        {
>                                print $OUT_FILE $value;
>                        }
>                        close $OUT_FILE;
>                }
>        }
> closedir $dh ;
> }
>
>
>
> Thanks,
> Mihir
>

2007/8/24, Mihir Kamdar <kamdarmihir06@gmail.com>:

> But a few improvizations. $cdr[13] that I am trying to manipulate is a
> "amount" field and should be a floating point value...how can I print it a
s
> floating point in perl?
>

Perl isn't a strong type language,it doesn't mind which data type it used.
You can use printf() for printing a floating vlaue,like,
$ perl -e 'printf("%.2f",3)'
3.00

> Also I am reading from an input file and writing to an output file. Is it
> possible to read and write to the same file without any side-effects??
>

Yes,just open the file for reading and handling,store the results to
an array as you did,close  the file.then re-open the file for
writing,and write the content in the array to the file.It's no
problem.


> @Jeff:- As you said that @hash{@arr} is different with $hash{"@arr"}, I am
> sorry but I am not sure what you are trying to point out through this.

@hash{@arr} means this hash has more than one keys (given the @arr has
3 elements,then this hash has 3 keys,each key is one of this array's
elements).
$hash{"@arr"} means the hash has only one key,it's the same as:
$key = join '',@arr;
$hash{$key} = ...;

Also @hash{@arr} is a hash slice,but $hash{"@arr"} isn't.

Hi,

Thanks...i got my mistake...it worked after the below as Rob suggested:-
my @cdr=split (/,/, $line) ;
$cdr[13] = $cdr[6]*5.0 ;
$line = join(",",@cdr);
$hash{"@cdr[2,3,6,7]"}=$line;

But a few improvizations. $cdr[13] that I am trying to manipulate is a
"amount" field and should be a floating point value...how can I print it as
floating point in perl?

Also I am reading from an input file and writing to an output file. Is it
possible to read and write to the same file without any side-effects??

@Jeff:- As you said that @hash{@arr} is different with $hash{"@arr"}, I am
sorry but I am not sure what you are trying to point out through this. Can
you please explain at a very basic level so that I can understand how it
works and how it affects my code?


Thanks,
Mihir

Perl isn't a strong type language,it doesn't mind which data type it used.
> You can use printf() for printing a floating vlaue,like,
> $ perl -e 'printf("%.2f",3)'
> 3.00


that's right, but with respect to this particular code of mine, the $cdr[13]
is the "amount" field which should be a float.  I am manipulating it like
this:-

while ($line=readline($IN_FILE))
{
my @cdr=split (/,/, $line) ;
$cdr[13] = $cdr[6]*5.0 ;
$line = join(",",@cdr);
$hash{"@cdr[2,3,6,7]"}=$line;
}

and then writing the output like this:-

open (my $OUT_FILE,">","$write_path/$file.out") or die $!;
while (my($key, $value) = each %hash)
{
print $OUT_FILE $value;
}

Here if I want the $cdr[13] to be written as a float(ex. 15.00), then how do
I write it?

Thanks,
Mihir

2007/8/24, Mihir Kamdar <kamdarmihir06@gmail.com>:
>
> Here if I want the $cdr[13] to be written as a float(ex. 15.00), then how 
do
> I write it?
>

$cdr[13] = sprintf "%.2f", $cdr[6]*5.0 ;