Print lines that have no duplicates in a file and preserve sort order linux

I tried sort file.txt | uniq -u it works, but output is sorted

You could take that output, and use it as a list of newline-delimited patterns with grep -f on the original file. Use -Fx to treat the patterns as whole line fixed strings (not regular expressions).

sort file.txt | uniq -u | grep -Fxf- file.txt

4
3

Here is a Ruby to do that:

ruby -lne 'BEGIN{cnt=Hash.new {|h,k| h[k] = 0} } 
cnt[$_]+=1
END{puts cnt.select{|k,v| v==1}.keys.join("\n") }
' file 

Prints:

4
3

Or, in one read of the file:

ruby -e 'puts $<.read.split(/\R+/).
            group_by{|x| x}.select{|k,v| v.length==1}.keys.join("\n")
' file 
# same output

Unlike awk, Ruby associative arrays maintain insertion order.

If you want a one pass awk you could do:

awk 'BEGIN{OFS="\t"}
{ if (seen[$0]++) delete order[$0]; else order[$0]=FNR } 
END { for ( e in order ) print order[e], e } ' file | sort -nk 1,1 | cut -f2-
# same output

(Thanks Ed Morton for a better awk!)

Using entirely Bash built-ins, you can do this in just a few lines:

declare -A SEEN=()
while IFS= read -r LINE; do
    (( ++SEEN[_$LINE] ))
done < file.txt
while IFS= read -r LINE; do
    if [[ ${SEEN[_$LINE]} -eq 1 ]]; then
        printf -- '%s\n' "$LINE"
    fi
done < file.txt

Note: The _$LINE as the subscript is to handle empty lines correctly.

Using Raku (formerly known as Perl_6)

~$ raku -ne 'BEGIN my %seen; 
             %seen.push: $_ => ++$; 
             END .key.put if .value.elems == 1 for %seen.sort: *.value.head;'  file

Raku is a programming language in the Perl-family that features high-level Unicode support. Above, Raku's -ne non-autoprinting linewise flags are used to act on an input file placed on the command line:

• BEGIN by declaring a %-sigiled hash (%seen),

• In the main body of the loop, each line is taken-in as key with ++$ an anonymous incrementing variable as value. Raku's => fat-arrow is the pair constructor.

• At the END iterate through the sorted hash and print out (put) the key only if .value.elems == 1. Sort is accomplished on *.value.head, i.e. the lowest line number obtained per key.

The Raku code above works fine on the OP's numeric input example. Since keys/values(line-numbers) can get confusing when bith are numeric, below is example input/output with alphabetic input:

Sample Input:

B
A
D
C
B
A

Sample Output (1):

D
C

If you really want to see behind the scenes, here's the END statement without filtering (i.e. END .say for %seen.sort: *.value.[0];). Note how the .value output order (first element) goes 1,2,3,4 :

Sample Output (2):

B => [1 5]
A => [2 6]
D => 3
C => 4

https:docs.raku.org

https:raku.org

Here's an approach using only awk, which reads the input only once, and yet doesn't store the entire file in memory:

• fo stores a line's first occurrence into an array. If the line isn't registered yet (!fo[$0]), save the current line number (fo[$0]=NR).
• fq counts the frequency of a line, which is incremented for every line read (fq[$0]++).
• Also, the yet unincremented value of fq[$0] is used as condition (which is not met on 0, i.e. the only value that would be incremented to the desired frequency of 1, and met otherwise due to an exceeding frequency) to abandon the corresponding register of first occurrence (delete fo[$0]).
• Eventually, fo contains only items of relevance (lines occurring not more than once), with the lines' contents as indices, and the line numbers of their first occurrences as values. So, to finish, only the array's indices need to be printed in ascending order of their numeric values. One way to achieve this would be by using asorti (available in GNU Awk 4+) with the proc instruction @val_num_asc to numerically sort by values in ascending order.

awk '
  !fo[$0]  {fo[$0]=NR}
  fq[$0]++ {delete fo[$0]}
  END      {asorti(fo,fo,"@val_num_asc"); for (i in fo) print fo[i]}
'

4
3