====== Text Processing Recipes ======

grep / sed / awk / jq / sort / cut / find / xargs: the stuff that answers
"how do I..." faster than writing a script.

Deep dives: [[terminal:scripting:sed|sed]] · [[terminal:scripting:awk|awk]] · [[terminal:scripting:grep|grep]] ·
[[terminal:scripting:jq|jq]] · [[terminal:scripting:find_xargs|find-xargs]] · [[terminal:scripting:curl|curl]] ·
[[terminal:scripting:regex|regex]] (which flavor each tool uses)

===== find + xargs =====

<code bash>
find . -name '*.log' -mtime +7 -delete              # old logs
find . -type f -size +100M                          # big files
find . -name '*.txt' -print0 | xargs -0 grep -l foo # safe for spaces
find . -newer marker.txt                            # changed since file
find . -type f -exec chmod 644 {} +                 # bulk chmod (+ = batched)
</code>

===== grep / rg =====

<code bash>
sudo apt install ripgrep        # rg is faster and simpler than grep -r
rg -n 'pattern'                 # recursive, line numbers, respects .gitignore
rg -l --type-add 'conf:*.conf' -tconf 'listen'    # file-type filter
rg -A3 -B2 'error' log          # context
grep -vE '^\s*(#|$)' file       # strip comments+blank lines
</code>

===== sed =====

<code bash>
sed -i 's/old/new/g' file             # in-place
sed -i.bak 's/x/y/' file              # with backup
sed -n '10,20p' file                  # lines 10-20
sed -n '/START/,/END/p' file          # range by pattern
sed 's/^/    /' file                  # indent
sed '/^$/d' file                      # delete blank lines
</code>

===== awk =====

<code bash>
awk '{print $1}'                      # column 1
awk -F: '{print $1}' /etc/passwd      # custom delimiter
awk '$3 > 100 {print $1}'             # condition
awk '{s+=$1} END {print s}'           # sum a column
awk '!seen[$0]++'                     # dedupe, keep order
awk 'NR==5' file                      # just line 5
df -h | awk '$5+0 > 80 {print}'       # numeric compare
</code>

===== jq =====

<code bash>
sudo apt install jq
jq . file.json                        # pretty-print
jq '.items[].name'                    # nested field
jq -r '.items[] | "\(.name) \(.size)"'   # raw strings, format
jq 'keys'                             # object keys
jq 'length'                           # array length
jq '.[] | select(.age > 30)'          # filter
jq -s 'add' *.json                    # slurp files into one array
curl -s api | jq '.data'              # from a pipe
</code>

===== sort / uniq / comm =====

<code bash>
sort -h du_output                     # human sizes
sort -k2 -n                           # numeric by column 2
sort -u file                          # sort+dedupe
sort file | uniq -c | sort -rn | head # frequency top
comm -12 <(sort a) <(sort b)          # lines in both
diff -u <(cmd1) <(cmd2)               # diff two commands' output
</code>

===== cut / paste / tr / column =====

<code bash>
cut -d: -f1,3 /etc/passwd
echo "a b c" | tr ' ' '\n'            # words to lines
tr 'a-z' 'A-Z' < f                    # uppercase
column -t -s, file.csv                # align columns
paste -d, a.txt b.txt                 # join lines side by side
</code>

===== Miscellaneous greats =====

<code bash>
wc -l file   # count lines          tee -a out.log   # write+show
xargs -I{} cp {} dest/               # template per item
xargs -P4 -n1 cmd                    # parallel, one arg each
comm/tabular viewing: less -S        # no wrap, scroll right
head -n -5 file                      # all but last 5 lines
tac file | less                      # reverse order
</code>

===== CSV without a real CSV parser =====

''mlr'' (miller) or ''csvkit'' (''sudo apt install miller csvkit''): proper CSV
tools that handle quoted commas, unlike ''cut -d,'':

<code bash>
mlr --csv cut -f name,age in.csv
csvcut -n in.csv && csvcut -c 2 in.csv | csvlook
</code>


===== See Also =====

  * [[terminal:scripting:start|Scripting Index]]
  * [[terminal:tools:start|CLI Tools]]
  * [[terminal:start|Return to terminal]]
  * [[start|Return to wiki home]]