Ask Ubuntu
Ask Ubuntu

Questions tagged [uniq]

The uniq command deletes repeated lines in a file. The uniq command reads either standard input or a file specified by the InFile parameter

The uniq command deletes repeated lines in a file. The uniq command reads either standard input or a file specified by the InFile parameter. The command first compares adjacent lines and then removes the second and succeeding duplications of a line. Duplicated lines must be adjacent. (Before issuing the uniq command, use the sort command to make all duplicate lines adjacent.) Finally, the uniq command writes the resultant unique lines either to standard output or to the file specified by the OutFile parameter. The InFile and OutFile parameters must specify different files.

Source:IBM

19 questions
46
votes
4 answers

uniq command not working properly?

So I'm checking the md5 hash of my files with this as my output: 657cf4512a77bf47c39a0482be8e41e0 ./dupes2.txt 657cf4512a77bf47c39a0482be8e41e0 ./dupes.txt 8d60a927ce0f411ec94ac26a4785f749 ./derpina.txt 15f63928b8a1d5337137c38b5d66eed3 …
user2127726
  • 463
19
votes
8 answers

How to count occurrences of text in a file?

I have a log file sorted by IP addresses, I want to find the number of occurrences of each unique IP address. How can I do this with bash? Possibly listing the number of occurrences next to an ip, such as: 5.135.134.16 count: 5 13.57.220.172: count…
j0h
  • 15,365
14
votes
4 answers

Why did the command uniq -c put a whitespace at the beginning?

I have this code in a shell script: sort input | uniq -c | sort -nr > output The input file had no preceding white spaces, but the output does. How do I fix this? This is in bash
Jeremy Wik
  • 141
4
votes
1 answer

How to get only one item of the a permutation from a list of strings

I want to keep only one permutation among a set of strings and count the occurence of each permutation. To make things more clear, I want to transform (for instance) ab acf ba cfa acf into 2 ab 3 acf where 2 and 3 are the number of permutations…
user123456
  • 2,478
4
votes
2 answers

How can I find duplicate in the first column, then remove concerning whole lines ?

I have one xlsx file (110725x9 matrix) and I saved as type text (tab delemited) because I don't know whether Unix helps for xlsx files or not. Duplicates rows are always successive line by line. For example, suppose text file as follow. You will…
Suat Yazıcı
  • 45
  • 1
  • 7
3
votes
1 answer

How does uniq work?

Do not confuse this question to be a duplicate of "what is difference b/w sort -u and sort | uniq" This is in essence a word count program Confusion raised from the following command is reason for asking this question: …
juggernauthk108
  • 1,001
2
votes
1 answer

How to compare two (vague) file lists and print the duplicates?

I have two file lists. backup.txt and backup2.txt Some of the entries aren't exact, so it makes it difficult to find the duplicates with diff or…
mktoaster
  • 251
2
votes
1 answer

How to find duplicate lines in very large (65GB) text files?

I have 10 text files, that contain 65 characters of hex value in each line. Each text files is 6.5GB in size (having 99999999 lines) i.e.file1-6.5GB, file2-6.5GB, file3-6.5GB, file4-6.5GB, ... file10-6.5GB I need to find duplicate lines in all of…
maa
  • 91
1
vote
2 answers

Using uniq -f 1 gives unexpected results

After asking a question on ubuntuforums.org and not getting a satisfying answer, I've decided to ask the question again here on Ask Ubuntu. I need the answer to be very detailed. Specifically, I need to know which lines get compared every time a…
John_Patrick_Mason
  • 149
1
vote
1 answer

uniq --count command is yields incorrect result?

From the document said that the option --count going to prefix lines by the number of occurrences but after I do some test it seems doesn't work. $ echo -e "a\na\b\nc" |uniq --count 1 a 1 a 1 c Should the line 'a' occurs 2…
Smile.Hunter
  • 8,705
1
vote
1 answer

Why does uniq seem to keep some consecutive identical lines?

Without uniq: amin@ubuntu:~/Desktop$ cut -f 1 info.log | tail -n +2 | head -n -1 | sort Abol Abol Ahmad Akbar Arash Hadi Hamed Mahmood Maryam Maryam Mohsen NIma Rasool Sadegh Sepide Sepide With uniq: amin@ubuntu:~/Desktop$ cut -f 1 info.log |…
Amin
  • 91
  • 2
  • 7
0
votes
1 answer

FInd out duplicate lines

I have a file which contain following content. Hi abcd Hi abc hello hello xyz hello I want the find out duplicate lines as well as howmany times it is repeteing.My expected output is as below. 2 Hi 3 hello I have used following command alreay…
JalT
  • 163
0
votes
2 answers

Sorting two columns with sort and uniq

I have a file that needs sorting in a specific way but I can't figure out how to do it. I've tried the following command but it doesn't give me what I want: sort -t" " -k1,1 Original_File | uniq -d > Desired_Output Original_File: User1 US User1…
Roboman1723
  • 2,975
  • 8
  • 26
  • 32
0
votes
1 answer

How to use last to see how many times users logged in?

I need to see how many times users logged in. I used last | sort | uniq -c but it gave me bad data I need to sort them by number of times they logged in and only show username and number of time. for example: username number of logins userb …
Usermaxn
  • 247
  • 2
  • 10
0
votes
1 answer

Uniq --count doesn't work properly

I'm trying to solve a problem with counting IPs using uniq -c. My file after using file looks like this…
Learningmachine
  • 1
1
2