lesson 114 CLI & Automation · intermediate

Run other programs with subprocess

Automation often means running tools that already exist, such as git, a backup command or a test runner, and acting on how they finished. subprocess.run does that and returns a CompletedProcess you can inspect.

Read and diagnose: this can't run in the browser, so you'll read real code and output and answer questions about it.

You'll be able to

Read a CompletedProcess: returncode, stdout, stderr, and check_returncode()
Explain why the command is a list of words and what capture_output, text and check change
Pass the runner in as a parameter so the logic around it can be exercised with a stand-in

The idea

On a computer, subprocess.run starts another program, waits for it to finish, and returns a CompletedProcess. A typical call:

subprocess.run(["git", "status", "--porcelain"], capture_output=True, text=True)

The command is a list of words: the program, then each argument. Python passes them to the program exactly as they are, so a file name with spaces or a ; in user input can't turn into a second command. capture_output=True keeps what the program printed instead of letting it go to the terminal, and text=True gives it back as text rather than bytes.

The result has four parts: args (the command), returncode (0 means success; anything else is the program's way of saying it failed), stdout and stderr. run does not raise when the program fails: you check returncode, or pass check=True to get a CalledProcessError instead. A CompletedProcess is an ordinary object you can also build yourself, which is what this example does to show what a successful and a failed git call give back.

Run each example to see what it prints. Running examples doesn't count toward progress.

example-1.py · Two results: success and failure

import subprocess

ok = subprocess.CompletedProcess(["git", "status", "--porcelain"], 0, stdout=" M report.py\n?? notes.txt\n", stderr="")
failed = subprocess.CompletedProcess(["git", "status"], 128, stdout="", stderr="fatal: not a git repository\n")

print(ok.returncode, ok.stdout.splitlines())
print(failed.returncode, failed.stderr.strip())
try:
    failed.check_returncode()
except subprocess.CalledProcessError as err:
    print("check failed with code", err.returncode, "for", err.cmd)

Starting a program needs an operating system that can start processes. A web page has none, so in this browser the call itself fails with OSError before any program runs. On your own computer the same line prints git's version.

example-2.py · Calling git from the browser

import subprocess

result = subprocess.run(["git", "--version"], capture_output=True, text=True)
print(result.stdout)

So write the logic around the call as a function that receives the runner: def current_branch(run=subprocess.run). In real use nothing changes, because the default is the real subprocess.run. In a test (and in this browser) you pass a stand-in: a small function that accepts the same arguments and returns a CompletedProcess with the answer you want to try. The code under test can't tell the difference, so its decisions (parsing the output, reacting to a failure) are exercised for real.

example-3.py · The current git branch, with a stand-in

import subprocess

def current_branch(run=subprocess.run):
    result = run(["git", "rev-parse", "--abbrev-ref", "HEAD"], capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError("git failed: " + result.stderr.strip())
    return result.stdout.strip()

def git_on_main(args, **options):
    print("asked to run:", args, options)
    return subprocess.CompletedProcess(args, 0, stdout="main\n", stderr="")

def git_outside_repo(args, **options):
    return subprocess.CompletedProcess(args, 128, stdout="", stderr="fatal: not a git repository\n")

print(current_branch(run=git_on_main))
try:
    current_branch(run=git_outside_repo)
except RuntimeError as err:
    print(err)

When to use it: calling tools that already do the job (git, a compressor, a database dump) from a Python script that decides what to run and what to do with the result.

Mistakes to avoid: building one command string from user input and passing shell=True (a name like x; rm -rf ~ becomes a second command); forgetting capture_output=True and then finding result.stdout is None; and ignoring returncode, so a failed backup looks like a successful one.

Review

subprocess.run([...], capture_output=True, text=True) returns a CompletedProcess: returncode, stdout, stderr.
0 means success; run doesn't raise on failure unless you ask with check=True.
Pass the command as a list of words, and take the runner as a parameter so the surrounding logic can run with a stand-in.
Next: a complete folder tool that plans its changes, shows them, applies them and reports.

Practise this

Tag a release with git