PERL Miscellaneous - How to change certain field separated by comma

Author

How to change certain field separated by comma

rashidaq@gmail.com

2005-09-25, 6:58 pm

I have file which is separated by commas.
This is like csv file, actually a database which
has 10 to 15 fields separated by commas.

I can get a print of field number 6 awk on a file which has fields
separated by commas
awk -F, '{print $6}' file
this prints number 6 field only.

Now how do I substitute a certain field with another word/string on
certain line range.

I have tried the vi editor
I have manage to append line #6 to #1876 to the end of lines
here is what I do
:2,1876s/$/stringtobeAppended/g and it works but

now how do I replace in the middle like field number 6 separated by
commas
on certain line range.

if done in PERL easily then I have to learn that too.

thanks in advance
Rashid

Enrique Perez-Terron

2005-09-25, 6:58 pm

On Sun, 25 Sep 2005 23:39:38 +0200, <rashidaq@gmail.com> wrote:

> I have file which is separated by commas.
> This is like csv file, actually a database which
> has 10 to 15 fields separated by commas.
>
> I can get a print of field number 6 awk on a file which has fields
> separated by commas
> awk -F, '{print $6}' file
> this prints number 6 field only.
>
> Now how do I substitute a certain field with another word/string on
> certain line range.
>
> I have tried the vi editor
> I have manage to append line #6 to #1876 to the end of lines
> here is what I do
> :2,1876s/$/stringtobeAppended/g and it works but
>
> now how do I replace in the middle like field number 6 separated by
> commas
> on certain line range.
>
> if done in PERL easily then I have to learn that too.
>
> thanks in advance
> Rashid
>

awk -F, 'BEGIN {OFS=","}
/FirstsPattern/,/LastPattern/{$6="TheReplacement"; print}'

-Enrique

Ed Morton

2005-09-25, 6:58 pm

rashidaq@gmail.com wrote:

> I have file which is separated by commas.
> This is like csv file, actually a database which
> has 10 to 15 fields separated by commas.
>
> I can get a print of field number 6 awk on a file which has fields
> separated by commas
> awk -F, '{print $6}' file
> this prints number 6 field only.
>
> Now how do I substitute a certain field with another word/string on
> certain line range.
>
> I have tried the vi editor
> I have manage to append line #6 to #1876 to the end of lines
> here is what I do
> :2,1876s/$/stringtobeAppended/g and it works but
>
> now how do I replace in the middle like field number 6 separated by
> commas
> on certain line range.

awk -F, 'NR>1&&NR<1877{$6=newText}1'

Regards,

Ed.

Jürgen Exner

2005-09-25, 6:58 pm

rashidaq@gmail.com wrote:
> I have file which is separated by commas.
> This is like csv file, [...]

Then why not just use the Text::CSV module?

jue

Kenny McCormack

2005-09-25, 9:56 pm

In article <lpGZe.1268$qC4.1151@trnddc02>,
Jürgen Exner <jurgenex@hotmail.com> wrote:
>rashidaq@gmail.com wrote:
>
>Then why not just use the Text::CSV module?

Ah, the dangers of cross-posting.

It should be a rule (i.e., convention) that when respoding to
a cross-posted item, you must identiy which group you are posting from.

(Posting from comp.unix.shell, hence having no idea what a "Text::CSV"
is... [*])

[*] Not true, of course. I recognize that as a Perl thing, so I am
assuming that you are posting from one of the Perl groups, and didn't check
the "Newsgroups" line before posting.

Joe Smith

2005-09-26, 3:56 am

Enrique Perez-Terron wrote:

>
> awk -F, 'BEGIN {OFS=","}
> /FirstsPattern/,/LastPattern/{$6="TheReplacement"; print}'

And now for a perl solution:

linux% cat Temp
BEGIN {FS=OFS=","} /FirstsPattern/,/LastPattern/{$6="TheReplacement";
print}
linux% a2p Temp
#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
if $running_under_some_shell;
# this emulates #! processing on NIH machines.
# (remove #! line above if indigestible)

eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
# process any FOO=bar switches

$[ = 1; # set array base to 1
$FS = ' '; # set field separator
$, = ' '; # set output field separator
$\ = "\n"; # set output record separator

$FS = $, = ',';

while (<> ) {
chomp; # strip record separator
@Fld = split($FS, $_, 9999);
if (/FirstsPattern/ .. /LastPattern/) {
$Fld[6] = 'TheReplacement';
print join($,,@Fld);
}
}
linux%
-Joe

Damian James

2005-09-26, 3:56 am

On 25 Sep 2005 14:39:38 -0700, rashidaq@gmail.com said:
> I have file which is separated by commas.
> This is like csv file, actually a database which
> has 10 to 15 fields separated by commas.

Is it really CSV, or as you say just database fields separated by
commas? There are no commas in the fields? If there are, how are
they escaped? Does this format use quote characters?

Actual CSV is annoying to parse, and if you were writing a script to
manipulate these files in Perl, you'd probably want to use the
Text::CSV module as advised elswhere in this thread.

>...
>:2,1876s/$/stringtobeAppended/g and it works but
> now how do I replace in the middle like field number 6 separated by
> commas on certain line range.

You've had a couple of awk answers, which probably make the most
sense for a one liner.

> if done in PERL easily then I have to learn that too.

Perl one liner for completeness:

perl -F, -lane '$F[6] = "new" if 2..1876; print join ",", @F' filename

I'm sure others in clpm can golf that down somewhat ;)

--Damian

Logan Shaw

2005-09-26, 3:56 am

Joe Smith wrote:
> Enrique Perez-Terron wrote:
>
>
>
> And now for a perl solution:
>
> linux% cat Temp
> BEGIN {FS=OFS=","} /FirstsPattern/,/LastPattern/{$6="TheReplacement";
> print}
> linux% a2p Temp
> #!/usr/bin/perl
> eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
> if $running_under_some_shell;
> # this emulates #! processing on NIH machines.
> # (remove #! line above if indigestible)
>
> eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
> # process any FOO=bar switches
>
> $[ = 1; # set array base to 1
> $FS = ' '; # set field separator
> $, = ' '; # set output field separator
> $\ = "\n"; # set output record separator
>
> $FS = $, = ',';
>
> while (<> ) {
> chomp; # strip record separator
> @Fld = split($FS, $_, 9999);
> if (/FirstsPattern/ .. /LastPattern/) {
> $Fld[6] = 'TheReplacement';
> print join($,,@Fld);
> }
> }

And now for a slightly more concise Perl solution:

#! /usr/bin/perl -w

while (<> )
{
my @fields = split (/,/);
$fields[6-1] = 'TheReplacement'
if 6 .. 1876;
print join (",", @fields);
}

The code is probably a little clearer if "if 6 .. 1876" is replaced
with its equivalent, "if $. == 6 .. $. == 1876". But then I was
going for brevity.

- Logan

William James

2005-09-26, 3:56 am

rashidaq@gmail.com wrote:
> I have file which is separated by commas.
> This is like csv file, actually a database which
> has 10 to 15 fields separated by commas.
>
> I can get a print of field number 6 awk on a file which has fields
> separated by commas
> awk -F, '{print $6}' file
> this prints number 6 field only.
>
> Now how do I substitute a certain field with another word/string on
> certain line range.
>
> I have tried the vi editor
> I have manage to append line #6 to #1876 to the end of lines
> here is what I do
> :2,1876s/$/stringtobeAppended/g and it works but
>
> now how do I replace in the middle like field number 6 separated by
> commas
> on certain line range.
>
> if done in PERL easily then I have to learn that too.
>
> thanks in advance
> Rashid

awk 'BEGIN{FS=OFS=","}NR>1,NR<1877{$6="new"}1' file

Darren Dunham

2005-09-26, 6:58 pm

In comp.unix.solaris Logan Shaw <lshaw-usenet@austin.rr.com> wrote:
[color=darkred]
> And now for a slightly more concise Perl solution:

> #! /usr/bin/perl -w
> while (<> )
> {
> my @fields = split (/,/);
> $fields[6-1] = 'TheReplacement'
> if 6 .. 1876;
> print join (",", @fields);
> }

> The code is probably a little clearer if "if 6 .. 1876" is replaced
> with its equivalent, "if $. == 6 .. $. == 1876". But then I was
> going for brevity.

Or even...

perl -F, -ane '$F[5] =~ s/bob/sue/ if 6 .. 1876 ;print join(",",@F);'

--
Darren Dunham ddunham@taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

William James

2005-09-26, 6:58 pm

Damian James wrote:
> On 25 Sep 2005 14:39:38 -0700, rashidaq@gmail.com said:
>
> Is it really CSV, or as you say just database fields separated by
> commas? There are no commas in the fields? If there are, how are
> they escaped? Does this format use quote characters?
>
> Actual CSV is annoying to parse, and if you were writing a script to
> manipulate these files in Perl, you'd probably want to use the
> Text::CSV module as advised elswhere in this thread.
>
>
> You've had a couple of awk answers, which probably make the most
> sense for a one liner.
>
>
> Perl one liner for completeness:
>
> perl -F, -lane '$F[6] = "new" if 2..1876; print join ",", @F' filename
>
> I'm sure others in clpm can golf that down somewhat ;)
>
> --Damian

ruby -F, -pane'$F[5]="new" if 2..1876;$_=$F*","'

l v

2005-09-26, 6:58 pm

William James wrote:

>
>
> ruby -F, -pane'$F[5]="new" if 2..1876;$_=$F*","'
>

What version of Perl are you using? I can not find any Perl
documentation regarding "ruby" on my systems.

C:\>perldoc ruby
No documentation found for "ruby".

C:\>perldoc -f ruby
No documentation for perl function `ruby' found

Both are true for ActiveState Perl 5.6.1 build 631 and AIX 5.8.0

Len

----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----

rashidaq@gmail.com

2005-09-26, 6:58 pm

Its actually record(db) file separated with commas
I used the awk script given by you guys

awk -F, 'NR>3&&NR<=7{$2="new_string"}1' file > newfile

I was wondering rather "> newfile" can I use sed to
edit the file.

the PERL stuff went over my head, but I bought the
perl book and I will open that book today.

thank you everybody for your help

Rashid

William James wrote:
> rashidaq@gmail.com wrote:
>
> awk 'BEGIN{FS=OFS=","}NR>1,NR<1877{$6="new"}1' file

Logan Shaw

2005-09-26, 6:58 pm

Darren Dunham wrote:
> In comp.unix.solaris Logan Shaw <lshaw-usenet@austin.rr.com> wrote:
>
>
>
>
>
>
>
>
>
> Or even...
>
> perl -F, -ane '$F[5] =~ s/bob/sue/ if 6 .. 1876 ;print join(",",@F);'

If you want to get cryptic, you can save a few characters:

perl -F'/(,)/' -ane '$F[10] =~ s/bob/sue/ if 6 .. 1876; print @F'

That might even run faster because it avoids the join().

- Logan

Ed Morton

2005-09-26, 6:58 pm

rashidaq@gmail.com wrote:

> Its actually record(db) file separated with commas
> I used the awk script given by you guys
>
> awk -F, 'NR>3&&NR<=7{$2="new_string"}1' file > newfile

Why NR<=7 instead of NR<8?

> I was wondering rather "> newfile" can I use sed to
> edit the file.

You mean edit the file in-line without using a tmp file? GNU sed can do
that with it's "-i" option, and "ed" can do it by default, but in either
case you still have that pesky field identification issue. You can get
round that by using an RE like this one which replaces the second field
("old") with the word "new":

$ echo "pre old post" | sed
's/ $[^[:space:]]*[[:space:]]*$[^[:space:]
]*/\1new/'
pre new post

If you need it in some other field, you can specify a repetition count
for the number of fields to skip, e.g.:

$ echo "f1 f2 f3 f4 f5 f6" | sed
's/ $\([^[:space:]]*[[:space:]]*$\{2\}\)[^
[:space:]]*/\1new/'
f1 f2 new f4 f5 f6
$ echo "f1 f2 f3 f4 f5 f6" | sed
's/ $\([^[:space:]]*[[:space:]]*$\{4\}\)[^
[:space:]]*/\1new/'
f1 f2 f3 f4 new f6

You'll need to throw in an extra pattern if you want to accomodate white
space at the start of a line.

Regards,

Ed.

Darren Dunham

2005-09-26, 6:58 pm

In comp.unix.solaris Logan Shaw <lshaw-usenet@austin.rr.com> wrote:
[color=darkred]
> If you want to get cryptic, you can save a few characters:

> perl -F'/(,)/' -ane '$F[10] =~ s/bob/sue/ if 6 .. 1876; print @F'

> That might even run faster because it avoids the join().

Ah, that made me think that doing $,="," might help. It would probably
aid clarity, but it doesn't seem to be actually shorter for any
construction of mine. (Also, I'd have either do it in a BEGIN block or
run it every time through the loop. The latter would work fine, but it
would annoy me.) :-)

--
Darren Dunham ddunham@taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

Damian James

2005-09-26, 9:56 pm

On Mon, 26 Sep 2005 23:57:53 GMT, Darren Dunham said:
> In comp.unix.solaris Logan Shaw <lshaw-usenet@austin.rr.com> wrote:
>
>
>
>
> Ah, that made me think that doing $,="," might help. It would probably
> aid clarity, but it doesn't seem to be actually shorter for any
> construction of mine. (Also, I'd have either do it in a BEGIN block or
> run it every time through the loop. The latter would work fine, but it
> would annoy me.) :-)

*scratches head*

(I note that I introduced an off-by-one error cross thread by
equating awk's $6 with $F[6] instead of $F[5]).

Anyhow, with $, set:

perl -F, -lane '$,=",";$F[5] = "new" if 2..1876; print @F'

So yeah, it takes more characters, especially if you wanted a BEGIN block.
Similarly with $".

NB, the -l permits changing $F[-1] without needing to think about
line endings, not that that was required, but it seemed like the OP
wished to change arbitrary elements.

Likewise, I understood the OP intended to replace the whole element,
not a substring of it (the various awk solutions made this assumption).

So how about :

perl -F, -ape 's/$F[5]/new/ if 2..1876'

That breaks if $F[5] is a substring of an element to its left, of course.

--Damian

William James

2005-09-26, 9:56 pm

rashidaq@gmail.com wrote:
> I have file which is separated by commas.
> This is like csv file, actually a database which
> has 10 to 15 fields separated by commas.
>
> I can get a print of field number 6 awk on a file which has fields
> separated by commas
> awk -F, '{print $6}' file
> this prints number 6 field only.
>
> Now how do I substitute a certain field with another word/string on
> certain line range.
>
> I have tried the vi editor
> I have manage to append line #6 to #1876 to the end of lines
> here is what I do
> :2,1876s/$/stringtobeAppended/g and it works but
>
> now how do I replace in the middle like field number 6 separated by
> commas
> on certain line range.
>
> if done in PERL easily then I have to learn that too.
>
> thanks in advance
> Rashid

ruby -F, -pane'$F[5]="new" if 2..7;$_=$F*","'

rashidaq@gmail.com

2005-09-27, 3:57 am

thanks for the sed script.

I have also tried the RUBY stuff in my cygwin and it worked.
This RUBY must be new, never heard of it.

How popular is it now as compared to Java, Perl or Python

I have taken a class in Java, its good but ..

But for my work working as telecom engr working on solaris system
most of the time, I think Ruby fits.
I dont know much about it.
Just downloaded ruby editor from FORGE site today.

thanks
Rashid

William James wrote:
> rashidaq@gmail.com wrote:
>
> ruby -F, -pane'$F[5]="new" if 2..7;$_=$F*","'

Enrique Perez-Terron

2005-09-27, 3:57 am

On Tue, 27 Sep 2005 07:13:44 +0200, <rashidaq@gmail.com> wrote:

> thanks for the sed script.
>
> I have also tried the RUBY stuff in my cygwin and it worked.
> This RUBY must be new, never heard of it.

Ruby's inventor named it Ruby thinking that was a suitable name
for a language that wanted to become the successor of Pe(a)rl.

> How popular is it now as compared to Java, Perl or Python

I have not statistics, but I believe it still lives pretty much
in the shadow of these. When Perl appeared, it was like the
answer to everybody's frustrations. It had everything that
the day's scripting languges had been insisting on not having
for many years. When Ruby appeared, Perl was there, and it worked.

For a long time, Perl had CPAN, an enormous internet archive of
modules that would install themselves (or that Perl would
install) *correctly* with a single command that was (is) the same
on all architectures.

Only quite recently, I believe, has Ruby got that kind of
infrastructure.

I don't know why python got so much wind in its sails. I never
had the occasion to learn Ruby or Python well enough to judge
their relative merits. To my still rather superficial judgement
they have much in common, and seem to have been inpired by
much of the same theories. Both are very clean, uncluttered,
and rich in similar ways. Yet Python does not seem to have the
same advantage when it comes to one-liners.

> I have taken a class in Java, its good but ..

:)

> But for my work working as telecom engr working on solaris system
> most of the time, I think Ruby fits.
> I dont know much about it.
> Just downloaded ruby editor from FORGE site today.

-Enrique

Mark Clements

2005-09-27, 3:57 am

[follow-ups trimmed]

rashidaq@gmail.com wrote:
> thanks for the sed script.
>
> I have also tried the RUBY stuff in my cygwin and it worked.
> This RUBY must be new, never heard of it.

OK - you have never heard of it. This puzzles me because later you say...

>
> How popular is it now as compared to Java, Perl or Python
>
> I have taken a class in Java, its good but ..
>
> But for my work working as telecom engr working on solaris system
> most of the time, I think Ruby fits.

You've never heard of it but you think it fits? Am not arguing for or
against the utility of Ruby for your problem domain, but am interested
in your logic.

> I dont know much about it.

:) But it's a good fit, right?

puzzled,

Mark

William James

2005-09-27, 3:57 am

rashidaq@gmail.com wrote:
> thanks for the sed script.
>
> I have also tried the RUBY stuff in my cygwin and it worked.
> This RUBY must be new, never heard of it.
>
> How popular is it now as compared to Java, Perl or Python
>
> I have taken a class in Java, its good but ..
>
> But for my work working as telecom engr working on solaris system
> most of the time, I think Ruby fits.
> I dont know much about it.
> Just downloaded ruby editor from FORGE site today.

Go to comp.lang.ruby and see the action.

By the way, my Ruby solution was 1 character too long; it could
have been:

ruby -F, -pae'$F[5]="new" if 2..7;$_=$F*","' file

rashidaq@gmail.com

2005-09-27, 7:00 pm

I have tried ruby in our work place solaris OS
and there are no ruby in any machine.

Thats strange, does this ruby not come with solaris Netra or sparc 5.8
systems. ?

I have it on my Windows XP cygwin tool at home.

So SUN does not like ruby or is not ready for it, thats strange.

Rashid

Anno Siegel

2005-09-28, 8:06 am

<rashidaq@gmail.com> wrote in comp.lang.perl.misc:
> I have tried ruby in our work place solaris OS
> and there are no ruby in any machine.

A solaris box is no wristwatch.

Anno
--
If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the
"Reply" at the bottom of the article headers.

rashidaq@gmail.com

2005-09-28, 6:58 pm

That where I wanted to apply ruby on solaris 5.8
and it does not have it.
I guess no ruby till next solaris release 9 in my office

Rashid

Anno Siegel wrote:
> <rashidaq@gmail.com> wrote in comp.lang.perl.misc:
>
> A solaris box is no wristwatch.
>
> Anno
> --
> If you want to post a followup via groups.google.com, don't use
> the broken "Reply" link at the bottom of the article. Click on
> "show options" at the top of the article, then click on the
> "Reply" at the bottom of the article headers.

Mark Clements

2005-09-28, 6:58 pm

rashidaq@gmail.com wrote:
> That where I wanted to apply ruby on solaris 5.8
> and it does not have it.
> I guess no ruby till next solaris release 9 in my office
>

www.sunfreeware.com
www.blastwave.org

Are your friends.

Mark

Enrique Perez-Terron

2005-09-28, 6:58 pm

On Tue, 27 Sep 2005 18:38:50 +0200, <rashidaq@gmail.com> wrote:

> I have tried ruby in our work place solaris OS
> and there are no ruby in any machine.
>
> Thats strange, does this ruby not come with solaris Netra or sparc 5.8
> systems. ?
>
> I have it on my Windows XP cygwin tool at home.
>
> So SUN does not like ruby or is not ready for it, thats strange.

Not that strange. People usually donwload and install it themselves.

-Enrique

Sponsored Links