uniqx
Report or omit repeated lines.
Mirrors GNU uniq. Run it as uniqx, or as gnux uniqx.
Synopsis
uniqx [OPTION]... [INPUT [OUTPUT]]GNU reference
Description
uniqx filters adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output). With no options, matching lines are merged to the first occurrence. Only adjacent lines are compared, so a file is usually sorted first. -c prefixes each line with its number of occurrences, -d prints one copy of each repeated line, -u only the lines that are not repeated, and -D every line of each repeated group, with --all-repeated=METHOD (none, prepend, separate) putting an empty line before or between the groups. --group[=METHOD] (separate, prepend, append, both) prints every line with the groups delimited by an empty line, and cannot be combined with -c, -d, -D or -u; -D cannot be combined with -c either. The comparison skips the first N fields with -f (a field is a run of blanks then non-blanks), then N characters with -s, and compares at most N characters with -w. -z makes NUL the line delimiter, and the output always ends each line with it. Deviations from GNU: characters for -s and -w are bytes, as in GNU's C locale, and -i folds ASCII letters only. The obsolete -N and +N forms of -f and -s are not accepted. METHOD arguments may be abbreviated, as in GNU.
gnux's terminal view (BR-13): on a terminal, -c draws a histogram. Each count, in GNU's 7-column field, is colored by its share of the largest count, and a bar of that share stands before the line. The groups are written once the largest count is known: at the end of the input, or after the first 4096 groups, which are then drawn against the largest count among them, as is every group after. --pretty[=WHEN] and --theme=NAME, options of gnux's own, choose the view (GNUX_PRETTY, GNUX_THEME); piped or redirected output, an OUTPUT file and -z are always GNU's.
Permissions
Needs read permission on INPUT and search permission on the directories leading to it. When OUTPUT is given it is created or truncated, which needs write permission on it if it exists, or on its directory if it does not. INPUT itself is never modified.
Use cases
- Removing duplicate lines from sorted output.
- Counting how often each value occurs with sort and
-c, a frequency table. - Listing only the values that appear more than once with
-d. - Finding the values that appear exactly once with
-u. - Grouping related lines visually with
--group. - Comparing lines while ignoring a leading timestamp or ID with
-for-s. - Treating lines as equal when they share a prefix with
-w. - Deduplicating NUL-separated file lists safely with
-z.
Flags
-D, --all-repeated METHOD[="none"] print all duplicate lines; groups can be delimited with an empty line; METHOD is none (the default), prepend or separate
-w, --check-chars N compare no more than N characters in lines
-c, --count prefix lines by the number of occurrences
--group METHOD[="separate"] show all items, separating groups with an empty line; METHOD is separate (the default), prepend, append or both
--help display this help and exit
-i, --ignore-case ignore differences in case when comparing
--pretty WHEN[="always"] the gnux terminal view WHEN: 'always' (the default when WHEN is omitted), 'auto' (the default: on a terminal only), or 'never'; GNUX_PRETTY sets the default
-d, --repeated only print duplicate lines, one for each group
-s, --skip-chars N avoid comparing the first N characters
-f, --skip-fields N avoid comparing the first N fields
--theme NAME color the view with the built-in theme NAME (gnux theme picks and saves one); GNUX_THEME sets the default
-u, --unique only print unique lines
--version output version information and exit
-z, --zero-terminated line delimiter is NUL, not newlineExamples
Users report errors from your web server: see how its responses split between status codes, drawn as a histogram on the terminal
cutx -d' ' -f9 access.log | sortx | uniqx -cRemove adjacent duplicate lines
uniqx names.txtWrite the result to another file
uniqx names.txt unique-names.txtCount the occurrences of each line
uniqx -c sorted.txtPrint only the lines that are repeated
uniqx -d sorted.txtPrint only the lines that are not repeated
uniqx -u sorted.txtPrint every repeated line, the groups separated by an empty line
uniqx --all-repeated=separate sorted.txtPrint all lines with each group of equal lines set apart
uniqx --group sorted.txtCompare without regard to case
uniqx -i words.txtIgnore the first field (a timestamp, say) when comparing
uniqx -f 1 events.logIgnore the first 4 characters when comparing
uniqx -s 4 ids.txtTreat lines that share their first 8 characters as equal
uniqx -w 8 commits.txtDeduplicate a NUL-separated list
uniqx -z files.lstRemove duplicates from a pipeline's output
catx a.txt b.txt | uniqxCount the distinct lines of a file
uniqx sorted.txt | wcx -lDeduplicate in the portable gnux shell
gnux shell -c 'catx sorted.txt | uniqx -c'Options
Each option has a page of its own, with its usage and examples.
- -D, --all-repeated: Print all duplicate lines; groups can be delimited with an empty line; METHOD is none (the default), prepend or separate.
- -w, --check-chars: Compare no more than N characters in lines.
- -c, --count: Prefix lines by the number of occurrences.
- --group: Show all items, separating groups with an empty line; METHOD is separate (the default), prepend, append or both.
- -i, --ignore-case: Ignore differences in case when comparing.
- --pretty: The
gnuxterminal view WHEN: 'always' (the default when WHEN is omitted), 'auto' (the default: on a terminal only), or 'never'; GNUX_PRETTY sets the default. - -d, --repeated: Only print duplicate lines, one for each group.
- -s, --skip-chars: Avoid comparing the first N characters.
- -f, --skip-fields: Avoid comparing the first N fields.
- --theme: Color the view with the built-in theme NAME (
gnuxtheme picks and saves one); GNUX_THEME sets the default. - -u, --unique: Only print unique lines.
- -z, --zero-terminated: Line delimiter is NUL, not newline.
Exit status
| Status | Meaning |
|---|---|
| 0 | success: every operand was processed |
| 1 | failure: an operand could not be processed (diagnosed on standard error), or a command-line mistake |
Help in the terminal
uniqx --help # options, first examples, exit status
gnux help uniqx # this page: the full manual
gnux help uniqx --OPTION # one option in depth, with its examples