How to remove duplicate lines from a text file via command line

This post is just to serve as a tip if you want to remove duplicates from a text file using command line execution.

For the purpose of this demonstration I have a file as below

$ cat tux-file
Hello
World
Hello
World
World
Hello

Using sort command

Sort command provides a “-u” switch will can be used to easily exclude duplicates

$ sort -u tux-file
Hello
World

Use sort and uniq command to exclude duplicates

Using sort + uniq

$ sort tux-file | uniq
Hello
World

Conclusion

This post was intended to be a quick referral page to know the options available for duplicate strings handling on a file via Linux command line.

0 Comments on How to remove duplicate lines from a text file via command line

Leave a Comment

Your email address will not be published. Required fields are marked *

You Might Be Interested In