awk

A tiny language for column/record processing. Pattern: pattern { action } per line. Ubuntu ships mawk (fast, smaller feature set) : sudo apt install gawk gets you the full GNU dialect (gensub, strftime, mktime, asort, BEGINFILE).

Model

awk '{print $1}'              # $0=whole line, $1..$NF=fields, NF=count
awk 'NR==5'                   # NR=line number (bare pattern = print)
awk -F: '{print $1}' /etc/passwd   # field separator
awk -F: -v OFS='|' '{print $1,$3}' # output separator

Built-ins: NR record#, NF field count, FS/OFS/RS/ORS, FILENAME, $0, $1..$NF, $(NF-1) (second-to-last).

Patterns

awk '$3 > 100'                       # numeric compare
awk '$1 ~ /foo/'                     # field matches regex (== !~ for not)
awk '/start/,/end/'                  # range pattern
awk 'NR>=10 && NR<=20'
awk 'length($0) > 80'                # long lines
awk 'END{print}'                     # last line

BEGIN / END: headers, totals

awk 'BEGIN{FS=":"} {s+=$3} END{print "total:",s}' /etc/passwd
ps aux | awk '{m+=$6} END{printf "%.1f MB\n", m/1024}'   # total RSS

Associative arrays: the main feature

awk '{count[$1]++} END{for(k in count) print k, count[k]}'   # histogram
awk '{seen[$0]++} !seen[$0]-1' ...                            # just: awk '!seen[$0]++' dedupes
awk '{sum[$1]+=$2} END{for(k in sum) print k":"sum[k]}'       # group+sum

String functions

length(x) substr(s,i,n) index(s,t) tolower(s) toupper(s)
split(s,a,sep)         # array from string
gsub(re,rep,str)       # global replace IN the string (returns count)
sub(re,rep,str)        # first only
sprintf(fmt,...)       # returns formatted (printf prints)
match(s,re)            # sets RSTART/RLENGTH

awk '{gsub(/"/,""); print}' csvish.txt
awk '{split($1,a,"."); print a[1]}'       # first dotted part
awk '{printf "%-20s %6.2f\n", $1, $2}'    # aligned columns

Multiline / CSV-ish handling

awk -v RS='' '{print NR": "$0}' f    # paragraph mode (blank-line records)
awk -F'";"' '{print $2}' file        # crude quoted-CSV: real CSV → miller

Recipes

ps aux | awk '$3>5 {print $2, $3, $11}'        # processes >5% CPU
df -h | awk 'NR>1 && $5+0>80 {print $6,$5}'    # mounts over 80%
awk '{s[$1]++} END{for(x in s) print s[x],x}' access.log | sort -rn | head
awk 'NR==FNR{a[$1]=1;next} !($1 in a)' a.txt b.txt   # lines in b not in a
awk '{print $NF}'                               # last field
seq 100 | awk '{sum+=$1} END{print sum/NR}'     # mean

Gotchas

Fields are 1-indexed, $0 is the whole line.
-F takes a regex: -F'[,:]' splits on either.
String vs number: $1 == “5” string compare, $1 == 5 numeric.
print adds ORS (newline); printf doesn't.
mawk lacks gensub/strftime: install gawk if a script needs them.

folk.zone Wiki

Table of Contents

awk