Removing duplicates and excluding lines

Hi,

I am using the following awk command to produce a list of people's
names, removing duplicates, and excluding some names from the resulting
list.

The names are in names.txt, and those names to be excluded are taken
from exclude.txt.

awk "NR==FNR{f[$0]=1;next}!f[$0]++" exclude.txt names.txt

So, for example, with the following names.txt file:

John Smith
Dave Green
Steve West
Steve Smith
James Brown
Tom Smith
Tom Smith

and with an exclude.txt such as:

Dave Green

then the above awk command gives:

John Smith
Steve West
Steve Smith
James Brown
Tom Smith

What I would like to do also is to include only those names ending in
"Smith" in the result.  So ideally, I would like the result:

John Smith
Steve Smith
Tom Smith

But when I use:

awk /Smith/ "NR==FNR{f[$0]=1;next}!f[$0]++" exclude.txt names.txt

the duplicates are no longer removed (in this case "Tom Smith").  That
is, I get the (unwanted) result:

John Smith
Steve Smith
Tom Smith
Tom Smith

Please can you help.

Regards,
Jonny

Jonny wrote:
> Hi,
>
> I am using the following awk command to produce a list of people's
> names, removing duplicates, and excluding some names from the resulting
> list.
>
> The names are in names.txt, and those names to be excluded are taken
> from exclude.txt.
>
> awk "NR==FNR{f[$0]=1;next}!f[$0]++" exclude.txt names.txt
>
> So, for example, with the following names.txt file:
>
> John Smith
> Dave Green
> Steve West
> Steve Smith
> James Brown
> Tom Smith
> Tom Smith
>
> and with an exclude.txt such as:
>
> Dave Green
>
> then the above awk command gives:
>
> John Smith
> Steve West
> Steve Smith
> James Brown
> Tom Smith
>
> What I would like to do also is to include only those names ending in
> "Smith" in the result.  So ideally, I would like the result:

Including Smith only, automatically excludes everybody else.
You don't need an exclude file.

awk '/Smith/ && !a[$0]++' names.txt

>
> John Smith
> Steve Smith
> Tom Smith
>
> But when I use:
>
> awk /Smith/ "NR==FNR{f[$0]=1;next}!f[$0]++" exclude.txt names.txt
>
> the duplicates are no longer removed (in this case "Tom Smith").  That
> is, I get the (unwanted) result:
>
> John Smith
> Steve Smith
> Tom Smith
> Tom Smith
>

--
Regards,

---Robert

Le Tue, 19 Apr 2005 09:25:27 +0000, Jonny a écrit_:

> Hi,
>
> I am using the following awk command to produce a list of people's
> names, removing duplicates, and excluding some names from the resulting
> list.
>
> The names are in names.txt, and those names to be excluded are taken
> from exclude.txt.
>
> awk "NR==FNR{f[$0]=1;next}!f[$0]++" exclude.txt names.txt
>
> So, for example, with the following names.txt file:
>
> John Smith
> Dave Green
> Steve West
> Steve Smith
> James Brown
> Tom Smith
> Tom Smith
>
> and with an exclude.txt such as:
>
> Dave Green
>
> then the above awk command gives:
>
> John Smith
> Steve West
> Steve Smith
> James Brown
> Tom Smith
>
> What I would like to do also is to include only those names ending in
> "Smith" in the result.  So ideally, I would like the result:
>
> John Smith
> Steve Smith
> Tom Smith
>
> But when I use:
>
> awk /Smith/ "NR==FNR{f[$0]=1;next}!f[$0]++" exclude.txt names.txt
>
> the duplicates are no longer removed (in this case "Tom Smith").  That
> is, I get the (unwanted) result:
>
> John Smith
> Steve Smith
> Tom Smith
> Tom Smith

Something's odd there, what do you do of poor "Steve West" ?
Definitely not a Smith and not in your exlude.txt ???

A typo of you or something I didn't read correctly ???

Robert Katz wrote:

> Jonny wrote: 
>
> Including Smith only, automatically excludes everybody else.
> You don't need an exclude file.
>
>     awk '/Smith/ && !a[$0]++' names.txt
> 


Thanks for your reply, Robert.

What if I want to exclude Steve Smith and many other Smith's (too many
to list on the command line)?

Regards,
Jonny

Robert Katz wrote:

> Jonny wrote: 
>
> Including Smith only, automatically excludes everybody else.
> You don't need an exclude file.
>
>     awk '/Smith/ && !a[$0]++' names.txt
> 


Thanks for your reply, Robert.

What if I want to exclude Steve Smith and many other Smith's (too many
to list on the command line)?

Regards,
Jonny

Le Tue, 19 Apr 2005 15:38:16 +0200, Loki Harfagr a écrit_:


> A typo of you or something I didn't read correctly ???

Doh, forget it :D)
I read it again ;-)

Ed Morton wrote:
<snip>
> awk "NR==FNR{f[$0]=1;next}($2=="Smith")&&!f[$0]++" exclude.txt names.txt

Of course, you'll have to do whatever magic your system requires to
handle double quotes around strings ("Smith") since you're using double
quotes around your script.

Ed.

Ed Morton wrote:
>
>
> Jonny wrote:
> 
>
> <snip>
> 
>
>
>
> You do if there are some Smiths that you want excluded.

You got me there, Ed.

> 
>
>
> /Smith/ would find people whos first names are "Smith" and people who's
> first and last names start with "Smith", e.g. "Smithson". You need:
>
>     $2 == "Smith"
>

We've then got to make it more robust and account for middle names
or initials.  How about

$NF == "Smith"
 
>
>
> I don't know if the above syntax is valid in your environment. In mine
> (Unix), I'd have the /Smith/ inside the quotes along with the rest of
> the program.
> 
>
>
> Looks like it's taking the /Smith/ as your entire program and just
> finding all the Smiths. It's probably your quoting.
>
> Just do this:
>
> awk "NR==FNR{f[$0]=1;next}($2=="Smith")&&!f[$0]++" exclude.txt names.txt
>
>     Ed.

Ed Morton wrote:

> Ed Morton wrote:
> <snip> 
>
> Of course, you'll have to do whatever magic your system requires to
> handle double quotes around strings ("Smith") since you're using double
> quotes around your script.

Thanks Ed.

I'm using Windows 2000, so I just had to put a backslash before each of
the double quotes around Smith.

Regards,
Jonny