terminal:scripting:text_processing
This is an old revision of the document!
Table of Contents
Text Processing Recipes
grep / sed / awk / jq / sort / cut / find / xargs: the stuff that answers “how do I…” faster than writing a script.
Deep dives: sed.md · awk.md · grep.md · jq.md · find-xargs.md · curl.md · regex.md (which flavor each tool uses)
find + xargs
find . -name '*.log' -mtime +7 -delete # old logs find . -type f -size +100M # big files find . -name '*.txt' -print0 | xargs -0 grep -l foo # safe for spaces find . -newer marker.txt # changed since file find . -type f -exec chmod 644 {} + # bulk chmod (+ = batched)
grep / rg
sudo apt install ripgrep # rg is faster and simpler than grep -r rg -n 'pattern' # recursive, line numbers, respects .gitignore rg -l --type-add 'conf:*.conf' -tconf 'listen' # file-type filter rg -A3 -B2 'error' log # context grep -vE '^\s*(#|$)' file # strip comments+blank lines
sed
sed -i 's/old/new/g' file # in-place sed -i.bak 's/x/y/' file # with backup sed -n '10,20p' file # lines 10-20 sed -n '/START/,/END/p' file # range by pattern sed 's/^/ /' file # indent sed '/^$/d' file # delete blank lines
awk
awk '{print $1}' # column 1 awk -F: '{print $1}' /etc/passwd # custom delimiter awk '$3 > 100 {print $1}' # condition awk '{s+=$1} END {print s}' # sum a column awk '!seen[$0]++' # dedupe, keep order awk 'NR==5' file # just line 5 df -h | awk '$5+0 > 80 {print}' # numeric compare
jq
sudo apt install jq jq . file.json # pretty-print jq '.items[].name' # nested field jq -r '.items[] | "\(.name) \(.size)"' # raw strings, format jq 'keys' # object keys jq 'length' # array length jq '.[] | select(.age > 30)' # filter jq -s 'add' *.json # slurp files into one array curl -s api | jq '.data' # from a pipe
sort / uniq / comm
sort -h du_output # human sizes sort -k2 -n # numeric by column 2 sort -u file # sort+dedupe sort file | uniq -c | sort -rn | head # frequency top comm -12 <(sort a) <(sort b) # lines in both diff -u <(cmd1) <(cmd2) # diff two commands' output
cut / paste / tr / column
cut -d: -f1,3 /etc/passwd echo "a b c" | tr ' ' '\n' # words to lines tr 'a-z' 'A-Z' < f # uppercase column -t -s, file.csv # align columns paste -d, a.txt b.txt # join lines side by side
Miscellaneous greats
wc -l file # count lines tee -a out.log # write+show xargs -I{} cp {} dest/ # template per item xargs -P4 -n1 cmd # parallel, one arg each comm/tabular viewing: less -S # no wrap, scroll right head -n -5 file # all but last 5 lines tac file | less # reverse order
CSV without a real CSV parser
mlr (miller) or csvkit (sudo apt install miller csvkit): proper CSV
tools that handle quoted commas, unlike cut -d,:
mlr --csv cut -f name,age in.csv csvcut -n in.csv && csvcut -c 2 in.csv | csvlook
See Also
terminal/scripting/text_processing.1792792621.txt.gz · Last modified: by 127.0.0.1