Python for Scripting

Setup

python3 --version              # already installed
sudo apt install python3-pip python3-venv

Ubuntu blocks pip install system-wide (PEP 668): use a venv:

python3 -m venv ~/.venvs/myproj
source ~/.venvs/myproj/bin/activate
pip install requests
deactivate

Or install distro packages: sudo apt install python3-requests python3-bs4. Quick escape hatch without a venv: pip install –user –break-system-packages pkg : fine for a personal laptop, worth knowing it bypasses the guard.

Script skeleton

#!/usr/bin/env python3
"""one-line description"""
import sys
 
def main() -> int:
    return 0
 
if __name__ == "__main__":
    sys.exit(main())

chmod +x, ./script.py. Or a #!/usr/bin/env -S python3 shebang.

Recipes

# argparse
import argparse
p = argparse.ArgumentParser()
p.add_argument("file"); p.add_argument("-n", "--dry-run", action="store_true")
args = p.parse_args()
 
# files
from pathlib import Path
for f in Path("~/notes").expanduser().rglob("*.md"):
    text = f.read_text(encoding="utf-8")
 
# json / csv / yaml
import json, csv
data = json.loads(Path("in.json").read_text())
rows = list(csv.DictReader(open("in.csv")))
# yaml: pip install pyyaml -> yaml.safe_load(text)
 
# subprocess
import subprocess
out = subprocess.run(["grep", "-c", "x", "f.txt"],
                     capture_output=True, text=True, check=True).stdout
 
# http (stdlib, no deps)
from urllib.request import urlopen
html = urlopen("https://example.com", timeout=10).read()
 
# http (requests, if installed)
import requests
r = requests.get("https://api.github.com", timeout=10)
r.raise_for_status(); j = r.json()
 
# env + exit codes
import os, sys
token = os.environ.get("TOKEN") or sys.exit("set TOKEN first")
 
# dates
from datetime import datetime, timedelta
stamp = datetime.now().strftime("%Y-%m-%d %H:%M")

REPL & debugging

python3                     # repl
python3 -i script.py        # run then drop into repl
python3 -m pdb script.py    # debugger
breakpoint()                # in code, python >=3.7

Useful stdlib modules (no pip needed)

pathlib shutil subprocess argparse json csv sqlite3 re datetime os/os.path sys textwrap logging http.server zipfile tarfile statistics collections itertools functools dataclasses

Quick servers & tools

python3 -m http.server 8000          # serve cwd over HTTP
python3 -m json.tool < file.json     # pretty-print/validate JSON
python3 -m venv --help
pip list / pip show pkg / pip install -r requirements.txt

When performance hurts

Startup is fine; CPU-bound loops are slow: use numpy for number work or

push the hot path to subprocess-ed tools (grep, ffmpeg).

python3 -O drops asserts. cProfile for real profiling:

python3 -m cProfile -s cumtime script.py | head -20

folk.zone Wiki

Table of Contents