Skip to Content
gnux

uniqx

Report or omit repeated lines.

Mirrors GNU uniq. Run it as uniqx, or as gnux uniqx.

Synopsis

uniqx [OPTION]... [INPUT [OUTPUT]]

GNU reference

Description

uniqx filters adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output). With no options, matching lines are merged to the first occurrence. Only adjacent lines are compared, so a file is usually sorted first. -c prefixes each line with its number of occurrences, -d prints one copy of each repeated line, -u only the lines that are not repeated, and -D every line of each repeated group, with --all-repeated=METHOD (none, prepend, separate) putting an empty line before or between the groups. --group[=METHOD] (separate, prepend, append, both) prints every line with the groups delimited by an empty line, and cannot be combined with -c, -d, -D or -u; -D cannot be combined with -c either. The comparison skips the first N fields with -f (a field is a run of blanks then non-blanks), then N characters with -s, and compares at most N characters with -w. -z makes NUL the line delimiter, and the output always ends each line with it. Deviations from GNU: characters for -s and -w are bytes, as in GNU's C locale, and -i folds ASCII letters only. The obsolete -N and +N forms of -f and -s are not accepted. METHOD arguments may be abbreviated, as in GNU.

gnux's terminal view (BR-13): on a terminal, -c draws a histogram. Each count, in GNU's 7-column field, is colored by its share of the largest count, and a bar of that share stands before the line. The groups are written once the largest count is known: at the end of the input, or after the first 4096 groups, which are then drawn against the largest count among them, as is every group after. --pretty[=WHEN] and --theme=NAME, options of gnux's own, choose the view (GNUX_PRETTY, GNUX_THEME); piped or redirected output, an OUTPUT file and -z are always GNU's.

Permissions

Needs read permission on INPUT and search permission on the directories leading to it. When OUTPUT is given it is created or truncated, which needs write permission on it if it exists, or on its directory if it does not. INPUT itself is never modified.

Use cases

  • Removing duplicate lines from sorted output.
  • Counting how often each value occurs with sort and -c, a frequency table.
  • Listing only the values that appear more than once with -d.
  • Finding the values that appear exactly once with -u.
  • Grouping related lines visually with --group.
  • Comparing lines while ignoring a leading timestamp or ID with -f or -s.
  • Treating lines as equal when they share a prefix with -w.
  • Deduplicating NUL-separated file lists safely with -z.

Flags

  -D, --all-repeated METHOD[="none"]   print all duplicate lines; groups can be delimited with an empty line; METHOD is none (the default), prepend or separate
  -w, --check-chars N                  compare no more than N characters in lines
  -c, --count                          prefix lines by the number of occurrences
      --group METHOD[="separate"]      show all items, separating groups with an empty line; METHOD is separate (the default), prepend, append or both
      --help                           display this help and exit
  -i, --ignore-case                    ignore differences in case when comparing
      --pretty WHEN[="always"]         the gnux terminal view WHEN: 'always' (the default when WHEN is omitted), 'auto' (the default: on a terminal only), or 'never'; GNUX_PRETTY sets the default
  -d, --repeated                       only print duplicate lines, one for each group
  -s, --skip-chars N                   avoid comparing the first N characters
  -f, --skip-fields N                  avoid comparing the first N fields
      --theme NAME                     color the view with the built-in theme NAME (gnux theme picks and saves one); GNUX_THEME sets the default
  -u, --unique                         only print unique lines
      --version                        output version information and exit
  -z, --zero-terminated                line delimiter is NUL, not newline

Examples

Users report errors from your web server: see how its responses split between status codes, drawn as a histogram on the terminal

cutx -d' ' -f9 access.log | sortx | uniqx -c

Remove adjacent duplicate lines

uniqx names.txt

Write the result to another file

uniqx names.txt unique-names.txt

Count the occurrences of each line

uniqx -c sorted.txt
uniqx -d sorted.txt
uniqx -u sorted.txt
uniqx --all-repeated=separate sorted.txt
uniqx --group sorted.txt

Compare without regard to case

uniqx -i words.txt

Ignore the first field (a timestamp, say) when comparing

uniqx -f 1 events.log

Ignore the first 4 characters when comparing

uniqx -s 4 ids.txt

Treat lines that share their first 8 characters as equal

uniqx -w 8 commits.txt

Deduplicate a NUL-separated list

uniqx -z files.lst

Remove duplicates from a pipeline's output

catx a.txt b.txt | uniqx

Count the distinct lines of a file

uniqx sorted.txt | wcx -l

Deduplicate in the portable gnux shell

gnux shell -c 'catx sorted.txt | uniqx -c'

Options

Each option has a page of its own, with its usage and examples.

  • -D, --all-repeated: Print all duplicate lines; groups can be delimited with an empty line; METHOD is none (the default), prepend or separate.
  • -w, --check-chars: Compare no more than N characters in lines.
  • -c, --count: Prefix lines by the number of occurrences.
  • --group: Show all items, separating groups with an empty line; METHOD is separate (the default), prepend, append or both.
  • -i, --ignore-case: Ignore differences in case when comparing.
  • --pretty: The gnux terminal view WHEN: 'always' (the default when WHEN is omitted), 'auto' (the default: on a terminal only), or 'never'; GNUX_PRETTY sets the default.
  • -d, --repeated: Only print duplicate lines, one for each group.
  • -s, --skip-chars: Avoid comparing the first N characters.
  • -f, --skip-fields: Avoid comparing the first N fields.
  • --theme: Color the view with the built-in theme NAME (gnux theme picks and saves one); GNUX_THEME sets the default.
  • -u, --unique: Only print unique lines.
  • -z, --zero-terminated: Line delimiter is NUL, not newline.

Exit status

StatusMeaning
0success: every operand was processed
1failure: an operand could not be processed (diagnosed on standard error), or a command-line mistake

Help in the terminal

uniqx --help                # options, first examples, exit status
gnux help uniqx             # this page: the full manual
gnux help uniqx --OPTION    # one option in depth, with its examples

See also

On this page