Python os and subprocess module

1. A Simple Way to Learn a New Python Module

Whenever you start learning a new Python module, a useful first step is dir(module_name). It gives you a quick overview of the functions, classes, constants, and attributes available inside that module. You do not need to learn everything at once. Instead, identify the parts that are useful for the problem you are trying to solve and explore them through practical examples.

For example, with the os module:

import os
print(dir(os))

A practical learning workflow is:

1. Import the module.

   import os

2. Explore what is available.

   dir(os)

3. Pick the useful functions instead of trying to memorize everything.

   os.getcwd()
   os.listdir()
   os.mkdir()
   os.remove()
   os.rename()
   os.walk()
   os.getenv()

4. Connect each function to a DevOps use case.
   • os.getenv() - read environment variables.
   • os.walk() - scan directories for logs or configuration files.
   • os.listdir() - inspect files in a deployment directory.
   • os.remove() - clean old temporary files.
   • os.path - validate and work with paths before an automation script runs.

Key idea: dir() is excellent for discovery, but you normally do not want to learn every item it returns. Focus on the small set of functions that solve real DevOps problems.

2. Check Whether a Path Exists: File or Directory?

One of the most common tasks in DevOps automation is checking whether a path exists and then identifying whether that path is a file or a directory. You may need this check before reading a configuration file, creating a directory, copying files, or running a deployment step. The same problem can be solved from the Linux command line, in a shell script, or in Python.

2.1 Linux command line

The Linux test command can check whether a path exists:

test -e /tmp/example && echo "Exists"

To check specifically for a regular file:

test -f /tmp/example && echo "It is a file"

To check specifically for a directory:

test -d /tmp/example && echo "It is a directory"

The bracket syntax is a common alternative:

[ -f /tmp/example ] && echo "File"
[ -d /tmp/example ] && echo "Directory"

You can also inspect the path with these commands:

ls -ld /tmp/example
file /tmp/example

2.2 Shell script

A small shell script can perform all three checks in one place:

#!/bin/bash

path="$1"

if [ ! -e "$path" ]; then
    echo "Path does not exist"
elif [ -f "$path" ]; then
    echo "It is a file"
elif [ -d "$path" ]; then
    echo "It is a directory"
else
    echo "It exists, but it is neither a regular file nor a directory"
fi

Run it by passing a path as the first argument:

./check.sh /etc/passwd
# Output: It is a file

./check.sh /etc
# Output: It is a directory

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/check_file_exist/check_file.sh

2.3 Python using os.path

The os.path functions provide the same checks from Python:

import os

path = "/etc/passwd"

if not os.path.exists(path):
    print("Path does not exist")
elif os.path.isfile(path):
    print("It is a file")
elif os.path.isdir(path):
    print("It is a directory")

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/check_file_exist/check_file.py

2.4 Python using pathlib

A more modern approach is pathlib. Instead of treating a path only as a string, pathlib represents it as a Path object and provides readable methods such as exists(), is_file(), and is_dir().

from pathlib import Path

path = Path("/etc/passwd")

if not path.exists():
    print("Path does not exist")
elif path.is_file():
    print("It is a file")
elif path.is_dir():
    print("It is a directory")

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/check_file_exist/check_file_pathlib.py

2.5 Quick comparison

The problem stays the same across Linux, shell scripting, and Python; only the syntax changes. For newer Python code, pathlib is often easier to read, while os.path remains common and useful.

2.6 Useful functions to explore next

With os:

• os.getcwd()
• os.listdir()
• os.mkdir()
• os.makedirs()
• os.remove()
• os.rename()
• os.walk()
• os.getenv()

With pathlib:

• Path.cwd()
• Path.home()
• Path.mkdir()
• Path.iterdir()
• Path.glob()
• Path.name
• Path.suffix
• Path.parent
• Path.exists()
• Path.is_file()
• Path.is_dir()

3. Check the Size of a File

Once you know that a path exists and is a file, the next useful question is often its size. In DevOps, file size matters when working with log files, backups, configuration files, artifacts, or temporary files. For example, you may want to detect a log file that has grown too large before rotating it, or verify the size of a backup before copying it.

3.1 Linux

Use ls with the -h option for a human-readable size:

ls -lh /tmp/example.log

The -h option means human-readable, so sizes are displayed in units such as KB, MB, or GB.

3.2 Python using os

import os

file = "/tmp/example.log"
size = os.path.getsize(file)
print(size)

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/check_file_size/check_file_size.py

os.path.getsize() returns the file size in bytes. For example, 1048576 bytes is 1 MB.

To convert the result to megabytes:

size_mb = os.path.getsize(file) / (1024 * 1024)
print(size_mb)

3.3 Python using pathlib

from pathlib import Path

file = Path("/tmp/example.log")
size = file.stat().st_size
print(size)

file.stat().st_size also returns the size in bytes.

3.4 Easy comparison

Most Python file-size functions return bytes. If you want KB, MB, or GB, convert the value yourself:

size = os.path.getsize(file)

print("KB:", size / 1024)
print("MB:", size / (1024 * 1024))
print("GB:", size / (1024 * 1024 * 1024))

DevOps exercise idea: find all files larger than a chosen size. This is a common task when troubleshooting logs and disk usage.

4. Search for a Particular File

Another common DevOps task is searching for a particular file when you do not know its exact location. This is useful for locating configuration files, log files, certificates, deployment artifacts, or other operational files.

4.1 Linux find command

Search /etc for a file named passwd:

find /etc -name "passwd"

Search from the current directory:

find . -name "config.yaml"

Perform a case-insensitive search:

find . -iname "config.yaml"

Search only for regular files:

find . -type f -name "config.yaml"

4.2 Python using os.walk()

os.walk() recursively walks through a directory tree. For each directory, it gives you the current root path, a list of subdirectories, and a list of files.

import os

search_path = "/etc"
file_name = "passwd"

for root, directories, files in os.walk(search_path):
    if file_name in files:
        print(os.path.join(root, file_name))

To understand what os.walk() returns, print the three values while it traverses the directory tree:

for root, directories, files in os.walk("/etc"):
    print(root)
    print(directories)
    print(files)

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/search_files/search_file_v1.py

4.3 A more practical search script

This version keeps track of whether the file was found and prints a clear message when no match exists:

import os

search_path = "/etc"
file_name = "passwd"
found = False

for root, directories, files in os.walk(search_path):
    if file_name in files:
        full_path = os.path.join(root, file_name)
        print(f"File found: {full_path}")
        found = True

if not found:
    print("File not found")

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/search_files/search_file_v2.py

4.4 Using pathlib

pathlib can perform a recursive search with rglob(). The name rglob stands for recursive glob.

from pathlib import Path

search_path = Path("/etc")

for file in search_path.rglob("passwd"):
    print(file)

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/search_files/search_file_pathlib.py

4.5 Quick comparison

5. Execute Linux Commands from Python

In DevOps automation, you often already know the Linux command you want to run. Instead of rewriting the command in pure Python, you can execute the existing command as part of a larger Python workflow. Common examples include checking disk usage, inspecting processes, verifying network information, or calling an existing command-line tool.

5.1 The subprocess module

A simple command can be executed with subprocess.run():

import subprocess

subprocess.run(["ls", "-l"])

Another example:

import subprocess

subprocess.run(["df", "-h"])

If you want to capture the command output inside Python, use capture_output=True and text=True:

import subprocess

result = subprocess.run(
    ["df", "-h"],
    capture_output=True,
    text=True
)

print(result.stdout)

What each part means:

• ["df", "-h"] - the command and its argument.
• capture_output=True - captures the command output instead of only displaying it in the terminal.
• text=True - returns the captured output as a normal Python string.
• result.stdout - contains the command's standard output.

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/blob/main/python_for_devops/linux_command/subprocess_command.py

5.2 os.system(): the older approach

The os module can also execute a command:

import os

os.system("df -h")

For DevOps scripts, subprocess is generally more useful because it makes it easier to capture output, inspect return codes, handle errors, and pass command arguments in a structured way.

The practical idea is simple: you already know how to perform many tasks directly in Linux. Sometimes the cleanest automation is to execute an existing Linux command from Python and then process its output.

6. Practical DevOps Scenario: Check a Log File and Inspect the System

Imagine that you are troubleshooting an application server. A useful automation script can combine the file and command concepts from the previous sections.

The script should:

1. Check whether /var/log/myapp/app.log exists.

2. Confirm that the path is actually a file.

3. Check the file size.

4. Print a warning if the file becomes too large.

5. Run a Linux command such as df -h to inspect disk usage.

Source example: https://github.com/ideaweaver-ai/cracking-the-genai-interview/tree/main/python_for_devops/check_logfile_inspect_system

This scenario is useful because it combines path validation, file inspection, file-size checks, and Linux command execution into one practical DevOps workflow.

7. What You Have Learned

• Use dir(module_name) to discover what a Python module provides.
• Use Linux test, os.path, or pathlib to check whether a path exists and whether it is a file or directory.
• Use os.path.getsize() or Path.stat().st_size to inspect file sizes in Python.
• Use find, os.walk(), or Path.rglob() to search for files.
• Use subprocess.run() when you need to execute Linux commands from Python and work with their output.
• Combine these tools to build practical DevOps troubleshooting and automation scripts.