AWK - Bash script timing out

Author

Bash script timing out

cozzmo1@hotmail.com

2007-12-19, 6:58 pm

Perhaps someone could help with this.
This script times out.

#!/bin/bash
cat ~/$1 | while read line
do
if [[ "$line" != /*/*/**Closed* && "$line" != "" && "$line" != *Auto?
Closed* ]]
then
{
if [[ "$line" == [1-6][0-9][0-9][0-9][0-9]\,*[a-z,A-Z]* ]]
then
{
read ticket date time <<EOF
$(echo "$line" | nawk 'BEGIN { FS = "," } ; { print $1, $6, $7 }')
EOF
echo "$ticket" "$date" "$time" "Opened"
}
elif [[ "$line" == \*\*\**Not?Approved* ]]
then
{
echo "$line" | nawk '{ print "$ticket" " " $4 " " $5 " " "Not-
Approved" }'
}
elif [[ "$line" == \*\*\**Picked?Up* || "$line" == \*\*\**Complete*
|| "$line" == \*\*\**Approved* ]]
then
{
echo "$ticket" "$line" | nawk '{ print $1 " " $5 " " $6 " " $8 }'
}
fi
}
fi
done

---------
Output: (note yak1 is a very huge file, and I am just trying to get
the facts out.)

$ temp2 yak1
<snip>
32626 2007-09-26 10:57:00 Opened
32626 09-26-2007 12:23 Picked
32626 09-26-2007 12:47 Approved
32626 09-27-2007 01:51 Complete
32627 2007-09-26 11:16:00 Opened
32627 09-26-2007 11:30 Picked
32627 09-26-2007 11:30 Approved
32627 09-26-2007 11:41 Complete
32628 2007-09-26 11:20:00 Opened
32628 09-26-2007 11:29 Picked
32628 09-26-2007 11:29 Approved
32628 09-26-2007 11:46 Complete
<output ends here>
but the file hasn't even been half parsed, if I pipe this to a file,
then it ends sooner (32635).

This seems to have started after I added these items in order to print
from the variables instead of just rendering straight to print.
read ticket date time <<EOF
EOF
echo "$ticket" "$date" "$time" "Opened"

I am sure it has nothing to do with the input file.
but don't know why it hangs.
maybe there is a buffer that fills up because of the variables??

Thanks,
crzzy1

Kees Nuyt

2007-12-19, 6:58 pm

On Wed, 19 Dec 2007 10:47:04 -0800 (PST),
cozzmo1@hotmail.com wrote:

>Perhaps someone could help with this.
>This script times out.
>
>
>#!/bin/bash
>cat ~/$1 | while read line
>do
>if [[ "$line" != /*/*/**Closed* && "$line" != "" && "$line" != *Auto?
>Closed* ]]
> then
>{
>if [[ "$line" == [1-6][0-9][0-9][0-9][0-9]\,*[a-z,A-Z]* ]]
>then
>{
>read ticket date time <<EOF
>$(echo "$line" | nawk 'BEGIN { FS = "," } ; { print $1, $6, $7 }')
>EOF
>echo "$ticket" "$date" "$time" "Opened"
>}
>elif [[ "$line" == \*\*\**Not?Approved* ]]
>then
>{
>echo "$line" | nawk '{ print "$ticket" " " $4 " " $5 " " "Not-
>Approved" }'
>}
>elif [[ "$line" == \*\*\**Picked?Up* || "$line" == \*\*\**Complete*
>|| "$line" == \*\*\**Approved* ]]
>then
>{
>echo "$ticket" "$line" | nawk '{ print $1 " " $5 " " $6 " " $8 }'
>}
>fi
>}
>fi
>done
>
>---------
>Output: (note yak1 is a very huge file, and I am just trying to get
>the facts out.)
>
>$ temp2 yak1
><snip>
>32626 2007-09-26 10:57:00 Opened
>32626 09-26-2007 12:23 Picked
>32626 09-26-2007 12:47 Approved
>32626 09-27-2007 01:51 Complete
>32627 2007-09-26 11:16:00 Opened
>32627 09-26-2007 11:30 Picked
>32627 09-26-2007 11:30 Approved
>32627 09-26-2007 11:41 Complete
>32628 2007-09-26 11:20:00 Opened
>32628 09-26-2007 11:29 Picked
>32628 09-26-2007 11:29 Approved
>32628 09-26-2007 11:46 Complete
><output ends here>
>but the file hasn't even been half parsed, if I pipe this to a file,
>then it ends sooner (32635).
>
>This seems to have started after I added these items in order to print
>from the variables instead of just rendering straight to print.
>read ticket date time <<EOF
>EOF
>echo "$ticket" "$date" "$time" "Opened"
>
>I am sure it has nothing to do with the input file.
>but don't know why it hangs.
>maybe there is a buffer that fills up because of the variables??
>
>Thanks,
>crzzy1

I would advise te rewrite it in 100% awk.
The jumping between bash and awk costs time. Forking,
opening file handles and such isn't for free.
--
( Kees
)
c[_] Eat, drink and be merry, for tomorrow they may make it illegal. (Stanislaw Gierlicki) (#38)

Ed Morton

2007-12-19, 6:58 pm

On 12/19/2007 12:47 PM, cozzmo1@hotmail.com wrote:
> Perhaps someone could help with this.
> This script times out.
>
>
> #!/bin/bash
> cat ~/$1 | while read line
> do
> if [[ "$line" != /*/*/**Closed* && "$line" != "" && "$line" != *Auto?
> Closed* ]]
> then
> {
> if [[ "$line" == [1-6][0-9][0-9][0-9][0-9]\,*[a-z,A-Z]* ]]
> then
> {
> read ticket date time <<EOF
> $(echo "$line" | nawk 'BEGIN { FS = "," } ; { print $1, $6, $7 }')
> EOF
> echo "$ticket" "$date" "$time" "Opened"
> }
> elif [[ "$line" == \*\*\**Not?Approved* ]]
> then
> {
> echo "$line" | nawk '{ print "$ticket" " " $4 " " $5 " " "Not-
> Approved" }'
> }
> elif [[ "$line" == \*\*\**Picked?Up* || "$line" == \*\*\**Complete*
> || "$line" == \*\*\**Approved* ]]
> then
> {
> echo "$ticket" "$line" | nawk '{ print $1 " " $5 " " $6 " " $8 }'
> }
> fi
> }
> fi
> done
>
> ---------
> Output: (note yak1 is a very huge file, and I am just trying to get
> the facts out.)
>
> $ temp2 yak1
> <snip>
> 32626 2007-09-26 10:57:00 Opened
> 32626 09-26-2007 12:23 Picked
> 32626 09-26-2007 12:47 Approved
> 32626 09-27-2007 01:51 Complete
> 32627 2007-09-26 11:16:00 Opened
> 32627 09-26-2007 11:30 Picked
> 32627 09-26-2007 11:30 Approved
> 32627 09-26-2007 11:41 Complete
> 32628 2007-09-26 11:20:00 Opened
> 32628 09-26-2007 11:29 Picked
> 32628 09-26-2007 11:29 Approved
> 32628 09-26-2007 11:46 Complete
> <output ends here>
> but the file hasn't even been half parsed, if I pipe this to a file,
> then it ends sooner (32635).
>
> This seems to have started after I added these items in order to print
> from the variables instead of just rendering straight to print.
> read ticket date time <<EOF
> EOF
> echo "$ticket" "$date" "$time" "Opened"
>
> I am sure it has nothing to do with the input file.
> but don't know why it hangs.
> maybe there is a buffer that fills up because of the variables??
>
> Thanks,
> crzzy1
>

I suspect it has to do with you trying to read from your here document at the
same time as you're reading from you UUOC pipe, but whatever the problem is it's
a shell one, not an awk one so you'll get a better answer at a shell NG such as
comp.unix.shell.

Having said that, I'm sure you'd get a much better result if you just rewrite
the script in awk. It looks like what you're trying to do (directly translating
your script) is:

awk '
!/^\*\*\*.*Closed/ && !/^$/ && !/Auto.Closed/ {
if (/^[1-6][0-9][0-9][0-9][0-9],.*[a-z,A-Z]*$) {
split($0,f,",")
ticket = f[1]
print ticket,f[6],f[7],"Opened"
} else if (/^\*\*\*.*Not.Approved*/) {
print ticket,$4,$5,"Not-Approved"
} else if (/^\*\*\*.*Picked.Up/||/\*\*\*.*Complete/||/\*\*\*.*Approved/) {
print ticket,$5,$6,$8
}
}
' "$1"

If so, that can be made a bit more readable (IMHO) as:

awk '
/(^\*\*\*.*|Auto.)Closed|^$/ {
next
}
/^[1-6][0-9][0-9][0-9][0-9],.*[a-z,A-Z]*$) {
split($0,f,","); print f[1],f[6],f[7],"Opened"
next
}
/^\*\*\*.*Not.Approved*/ {
print f[1],$4,$5,"Not-Approved"
next
}
/^\*\*\*.*(Picked.Up|Complete|Approved)/ {
print f[1],$5,$6,$8
next
}
' "$1"

You may need to tweak the conditions to be exactly what you're looking for. If
you'd like more help, post a small set of sample input to match your expected
output.

Regards,

Ed.

cozzmo1@hotmail.com

2007-12-21, 6:59 pm

On Dec 19, 6:44 pm, Ed Morton <mor...@lsupcaemnt.com> wrote:
> On 12/19/2007 12:47 PM, cozz...@hotmail.com wrote:
>
>
>
>
>
>
>
>
>
>
> I suspect it has to do with you trying to read from your here document at the
> same time as you're reading from you UUOC pipe, but whatever the problem is it's
> a shell one, not an awk one so you'll get a better answer at a shell NG such as
> comp.unix.shell.
>
> Having said that, I'm sure you'd get a much better result if you just rewrite
> the script in awk. It looks like what you're trying to do (directly translating
> your script) is:
>
> awk '
> !/^\*\*\*.*Closed/ && !/^$/ && !/Auto.Closed/ {
> if (/^[1-6][0-9][0-9][0-9][0-9],.*[a-z,A-Z]*$) {
> split($0,f,",")
> ticket = f[1]
> print ticket,f[6],f[7],"Opened"
> } else if (/^\*\*\*.*Not.Approved*/) {
> print ticket,$4,$5,"Not-Approved"
> } else if (/^\*\*\*.*Picked.Up/||/\*\*\*.*Complete/||/\*\*\*.*Approved/) {
> print ticket,$5,$6,$8
> }}
>
> ' "$1"
>
> If so, that can be made a bit more readable (IMHO) as:
>
> awk '
> /(^\*\*\*.*|Auto.)Closed|^$/ {
> next}
>
> /^[1-6][0-9][0-9][0-9][0-9],.*[a-z,A-Z]*$) {
> split($0,f,","); print f[1],f[6],f[7],"Opened"
> next}
>
> /^\*\*\*.*Not.Approved*/ {
> print f[1],$4,$5,"Not-Approved"
> next}
>
> /^\*\*\*.*(Picked.Up|Complete|Approved)/ {
> print f[1],$5,$6,$8
> next}
>
> ' "$1"
>
> You may need to tweak the conditions to be exactly what you're looking for. If
> you'd like more help, post a small set of sample input to match your expected
> output.
>
> Regards,
>
> Ed.

-----------
Thanks for doing that Ed,

I tried, but can't seem to get it to work, I like the first one as I
seem to be able to follow the logic with it pretty well.
when I try the first one (saved as "ttt"), I get

$ ttt yak1
awk: syntax error near line 3
awk: illegal statement near line 3
awk: syntax error near line 7
awk: bailing out near line 7

Second one gives.
ttt yak1
awk: syntax error near line 6
awk: bailing out near line 6
awk: newline in regular expression near line 6

Crzzy1

Ed Morton

2007-12-21, 6:59 pm

On 12/21/2007 10:54 AM, cozzmo1@hotmail.com wrote:
> On Dec 19, 6:44 pm, Ed Morton <mor...@lsupcaemnt.com> wrote:
>
>
>
> -----------
> Thanks for doing that Ed,
>
> I tried, but can't seem to get it to work, I like the first one as I
> seem to be able to follow the logic with it pretty well.
> when I try the first one (saved as "ttt"), I get
>
> $ ttt yak1
> awk: syntax error near line 3
> awk: illegal statement near line 3
> awk: syntax error near line 7
> awk: bailing out near line 7
>
> Second one gives.
> ttt yak1
> awk: syntax error near line 6
> awk: bailing out near line 6
> awk: newline in regular expression near line 6
>
> Crzzy1

You're using old, broken awk (/usr/bin/awk on Solaris). Use a modern awk such as
GNU awk (gawk), new awk (nawk) or /usr/xpg4/bin/awk on Solaris.

Ed.

cozzmo1@hotmail.com

2008-01-03, 6:58 pm

On Dec 19 2007, 6:44 pm, Ed Morton <mor...@lsupcaemnt.com> wrote:
> On 12/19/2007 12:47 PM, cozz...@hotmail.com wrote:
>
>
>
>
>
>
>
>
>
>
> I suspect it has to do with you trying to read from your here document at the
> same time as you're reading from you UUOC pipe, but whatever the problem is it's
> a shell one, not an awk one so you'll get a better answer at a shell NG such as
> comp.unix.shell.
>
> Having said that, I'm sure you'd get a much better result if you just rewrite
> the script in awk. It looks like what you're trying to do (directly translating
> your script) is:
>
> awk '
> !/^\*\*\*.*Closed/ && !/^$/ && !/Auto.Closed/ {
> if (/^[1-6][0-9][0-9][0-9][0-9],.*[a-z,A-Z]*$) {
> split($0,f,",")
> ticket = f[1]
> print ticket,f[6],f[7],"Opened"
> } else if (/^\*\*\*.*Not.Approved*/) {
> print ticket,$4,$5,"Not-Approved"
> } else if (/^\*\*\*.*Picked.Up/||/\*\*\*.*Complete/||/\*\*\*.*Approved/) {
> print ticket,$5,$6,$8
> }}
>
> ' "$1"
>
> If so, that can be made a bit more readable (IMHO) as:
>
> awk '
> /(^\*\*\*.*|Auto.)Closed|^$/ {
> next}
>
> /^[1-6][0-9][0-9][0-9][0-9],.*[a-z,A-Z]*$) {
> split($0,f,","); print f[1],f[6],f[7],"Opened"
> next}
>
> /^\*\*\*.*Not.Approved*/ {
> print f[1],$4,$5,"Not-Approved"
> next}
>
> /^\*\*\*.*(Picked.Up|Complete|Approved)/ {
> print f[1],$5,$6,$8
> next}
>
> ' "$1"
>
> You may need to tweak the conditions to be exactly what you're looking for. If
> you'd like more help, post a small set of sample input to match your expected
> output.
>
> Regards,
>
> Ed.

Ed,

thanks for the assistance.

Through virtue of your infinite wisdom and re-educating myself to
understand what your code was doing, I managed to get your first
rendition to work.
I love that "split" -array tool!!
Really, I am amazed that you can come up with this stuff on the fly
like that.

-------
$ more ttt
echo "start"
/usr/xpg4/bin/awk '
!/^\*\*\*.*Auto.Closed.*/ && !/^\*\*\*.*Closed/ && !/^$/ {
if (/^[1-6][0-9][0-9][0-9][0-9].*[a-z,A-Z]*$/) {
split($0,f,",")
ticket = f[1]
print ticket,f[6],f[7],"Opened"
} else if (/^\*\*\*.*Not.Approved*/) {
split($0,f," ")
print ticket,f[6],f[7],"Not-Approved"
} else if (/^\*\*\*.*Picked.Up.*/||/\*\*\*.*Complete.*/||/\*\*
\*.*Approved.*/) {
split($0,f," ")
print ticket,f[4],f[5],f[7]
}
}

' "$1"

-------

gives the output of

host $ ttt yak1 | more
start
32645 2007-09-26 16:14:00 Opened
32645 09-26-2007 16:38 Picked
32645 09-26-2007 16:38 Approved
32645 09-27-2007 07:11 Complete
32646 2007-09-26 16:42:00 Opened
32646 09-26-2007 17:30 Picked
32646 09-26-2007 17:38 Approved
32646 09-27-2007 16:57 Complete
32647 2007-09-26 17:21:00 Opened
32647 09-26-2007 17:27 Picked
32647 09-27-2007 13:53 Approved
32647 09-27-2007 14:27 Complete
etc <snip>

Since we are dealing with Military time here I don't think a simple
algorithm will do, it would be

to find how long the tickets are
opened, since the date may change, and the "Opened" fields have a
different date/time format, that adds a lot of complexity.

Presently I paste the output into Excel and it does a fair job of
computing the time, but I would like to be able to have the script do
it.

desired example.
32645 2007-09-26 16:14:00 Opened 0:00
32645 09-26-2007 16:38 Picked 0:24
32645 09-26-2007 16:38 Approved 0:24
32645 09-27-2007 07:11 Complete 14:33

Thanks,
Cozzmo

-------

Ed Morton

2008-01-03, 6:58 pm

On 1/3/2008 10:23 AM, cozzmo1@hotmail.com wrote:
<snip>
> $ more ttt
> echo "start"

delete the above echo and add the below BEGIN section

> /usr/xpg4/bin/awk '

BEGIN{ print "start" }

> !/^\*\*\*.*Auto.Closed.*/ && !/^\*\*\*.*Closed/ && !/^$/ {
> if (/^[1-6][0-9][0-9][0-9][0-9].*[a-z,A-Z]*$/) {
> split($0,f,",")
> ticket = f[1]
> print ticket,f[6],f[7],"Opened"
> } else if (/^\*\*\*.*Not.Approved*/) {
> split($0,f," ")
> print ticket,f[6],f[7],"Not-Approved"
> } else if (/^\*\*\*.*Picked.Up.*/||/\*\*\*.*Complete.*/||/\*\*
> \*.*Approved.*/) {
> split($0,f," ")
> print ticket,f[4],f[5],f[7]
> }
> }
>
> ' "$1"
>
> -------
>
> gives the output of
>
> host $ ttt yak1 | more
> start
> 32645 2007-09-26 16:14:00 Opened
> 32645 09-26-2007 16:38 Picked
> 32645 09-26-2007 16:38 Approved
> 32645 09-27-2007 07:11 Complete
> 32646 2007-09-26 16:42:00 Opened
> 32646 09-26-2007 17:30 Picked
> 32646 09-26-2007 17:38 Approved
> 32646 09-27-2007 16:57 Complete
> 32647 2007-09-26 17:21:00 Opened
> 32647 09-26-2007 17:27 Picked
> 32647 09-27-2007 13:53 Approved
> 32647 09-27-2007 14:27 Complete
> etc <snip>
>
> Since we are dealing with Military time here I don't think a simple
> algorithm will do, it would be

to find how long the tickets are
> opened, since the date may change, and the "Opened" fields have a
> different date/time format, that adds a lot of complexity.

It's actually pretty easy with GNU awk's time functions.

> Presently I paste the output into Excel and it does a fair job of
> computing the time, but I would like to be able to have the script do
> it.
>
> desired example.
> 32645 2007-09-26 16:14:00 Opened 0:00
> 32645 09-26-2007 16:38 Picked 0:24
> 32645 09-26-2007 16:38 Approved 0:24
> 32645 09-27-2007 07:11 Complete 14:33
>
> Thanks,
> Cozzmo
>
> -------

Try this untested script (NOTE: it's GNU awk, not /usr/xpg4/bin/awk):

gawk '
BEGIN{ print "start" }
!/^\*\*\*.*Auto.Closed.*/ && !/^\*\*\*.*Closed/ && !/^$/ {
if (/^[1-6][0-9][0-9][0-9][0-9].*[a-z,A-Z]*$/) {
split($0,f,",")
ticket = f[1]
# time = f[6] f[7] = 2007-09-26 16:14:00
split(f[6]" "f[7],t,"[- :]")
startSecs = mktime(t[1]" "t[2]" "t[3]" "t[4]" "t[5]" "t[6])
print ticket,f[6],f[7],"Opened",0
} else if (/^\*\*\*.*Not.Approved*/) {
split($0,f," ")
print ticket,f[6],f[7],"Not-Approved"
} else if (/^\*\*\*.*Picked.Up.*/||/\*\*\*.*Complete.*/||/\*\*
\*.*Approved.*/) {
split($0,f," ")
# time = f[4] f[5] = 09-26-2007 16:38
split(f[4]" "f[5],t,"[- :]")
curSecs = mktime(t[3]" "t[1]" "t[2]" "t[4]" "t[5]" 0")
print ticket,f[4],f[5],f[7],curSecs - startSecs
}
}
' "$1"

That should print the number of seconds since the ticket was "Opened".
Reformating that into min:secs and doing the same for "Not-Approved", if
applicable, left as an exercise...

Ed.

Sponsored Links