PowerPoint Presentation

Python System Scripting.

[Audio] Welcome to "Mastering Pythonic Python"! This comprehensive course is designed to take your Python programming skills to the next level by focusing on writing code that not only works but also embodies the principles and best practices of Pythonic programming. Pythonic code is clean, elegant, and efficient, enabling you to write more readable, maintainable, and expressive programs..

[Audio] Python is a versatile programming language that can be used for various tasks, including system scripting. System scripting involves automating repetitive tasks, managing files and directories, interacting with the operating system, and more. Python provides several modules and libraries that make system scripting easy and efficient..

[Audio] Course Objectives The objective of this module is to provide an overview and practical examples of Python system scripting and administration. The module aimed to cover various topics, including working with files and directories, environment variables, running external commands, network programming, parsing arguments, creating filters to read text files, logging, and using third-party libraries. The goal was to equip the user with knowledge and examples to effectively utilize Python for system scripting and administration tasks..

[Audio] Course Outcomes Understanding of the os module and its functionalities for file and directory operations, environment variables, and launching external commands. Familiarity with the os.path module and its usage for path manipulation and working with directories and file paths. Knowledge of how to access and modify environment variables using the os.environ dictionary and os.getenv() function. Understanding of subprocess management and executing external commands using functions Awareness of network programming in Python using the built-in socket module for creating network connections and sending/receiving data. Understanding of techniques for parsing command-line arguments using the argparse module. Knowledge of creating filters to read text files, including techniques for line filtering, data extraction, and manipulation. Understanding of the logging module and its usage for recording and storing log messages for debugging and analysis. Awareness of the importance of third-party libraries and their usage in Python system scripting and administration tasks. Familiarity with examples and code snippets illustrating the concepts discussed throughout the chat..

[Audio] Course Prerequisites are Basic knowledge of Python programming Familiarity with programming concepts like variables, functions, and control flow.

[Audio] Python System Scripting: OS Services There are two modules related to system scripting in Python: os and os.path When it comes to system scripting in Python, the os module is one of the most important modules you'll work with. It provides a wide range of functions for interacting with the operating system, managing files and directories, and executing system commands. The os.path module, which is a submodule of os, provides functions for manipulating file paths.

[Audio] os Module: The os module provides a wide range of functions for interacting with the operating system. To Create a directory syntax is os dot make directory and directory path. To rename a file of directory syntax is od dot rename put old name then new name. To delete or remove any file the syntax is os dot remove keywork then name of file. To delete an empty directory syntax is or dot remove keyword and directory path. To Delete a directory and its contents recursively type shutil dor rmtree then directory_path. To get current working directory the syntax is os dot getcwd. To change current working directory the syntax is os dot chdir directory path..

[Audio] File Information: Os dot stat path name to Get file or directory information. Os dot path dot exists file_path to Check if a file exists or not. Os dot pathdot isfile path to Check if a path is a file. Os dot pathdot isdir path to Check if a path is a directory Environment Variables: Os dot getenv variable_name to Get the value of an environment variable. Os dot environ inside square bracket variable_name equal to 'value' will Set the value of an environment variable. System Commands Os dot system command will Run a system command. Subprocess dot check_output('command', shell=True): Run a system command and capture its output..

[Audio] The os.mkdir() function is used to create new directory It will create the new directory to the path in the string argument of the function in the D drive named folder Pythondir then QA Foldder. The os.getcwd() It returns the current working directory(CWD) of the file..

[Audio] os.popen() This function opens a file or from the command specified, and it returns a file object which is connected to a pipe..

[Audio] os.path Module: The os.path module, which is a submodule of os, provides functions for manipulating file paths. Here are some common functions it offers: Os dot path dot join (path1, path2): to Join two or more paths together. Os dot path dot basename('path'): to Get the base name (file or directory name) from a path. Os dot path dot dirname('path'): Get the directory name from a path. Os dot path dot exists ('path'): Check if a path exists. Os dot path dot isfile('path'): Check if a path points to a file. Os dot path dot isdir ('path'): Check if a path points to a directory. Os dot path dot abspath ('path'): Get the absolute path of a file or directory. Os dot path dot split('path'): Split a path into its directory and base name components. Os dot path dot splitext ('file_path'): Split a file path into its base name and extension. These modules provide a powerful set of functions to work with the operating system, manipulate file paths, and perform various system scripting tasks in Python..

[Audio] os.path.basename(path) : It is used to return the basename of the file . This function basically return the file name from the path given..

[Audio] os.path.dirname(path) : It is used to return the directory name from the path given. This function returns the name from the path except the path name..

[Audio] OS Services: Environment Variables Environment variables provide a great way to configure your Python application, eliminating the need to edit your source code when the configuration changes. Common configuration items that are often passed to application through environment variables are third-party API keys, network ports, database servers, and any custom options that your application may need to work properly. To set and get environment variables in Python you can just use the os module:.

[Audio] How to access environment variables from Python Using the os.environ dictionary In Python, the os.environ dictionary contains all the environment variables. The simplest way to retrieve a variable from inside your application is to use the standard dictionary syntax. For example, In this example: os.environ.get('MY_VARIABLE') retrieves the value of the environment variable named 'MY_VARIABLE'. If the variable is not found, it returns None. This method is useful when you want to handle the case where the variable is not set gracefully. os.environ['MY_VARIABLE'] = 'new_value' sets the value of the environment variable 'MY_VARIABLE' to 'new_value'. 'MY_VARIABLE' in os.environ checks if the environment variable 'MY_VARIABLE' is set. It returns True if the variable is present in the environment, and False otherwise. os.environ.items() returns a dictionary-like object containing all environment variables and their corresponding values. The loop prints each variable and its value. Remember to replace 'MY_VARIABLE' with the name of the actual environment variable you want to work with. The os.environ dictionary provides a convenient interface to interact with environment variables in Python..

[Audio] Using os.environ.get() In this example: os.environ.get('MY_VARIABLE') retrieves the value of the environment variable named 'MY_VARIABLE'. If the variable is found, its value is returned; otherwise, None is returned. The code checks if value is not None, indicating that the environment variable is set. It then prints the value of the variable. If value is None, it means the environment variable is not set, and a corresponding message is printed. Remember to replace 'MY_VARIABLE' with the name of the actual environment variable you want to retrieve. The os.environ.get() method allows you to safely retrieve the value of an environment variable without raising an exception if it doesn't exist..

[Audio] Adding a default value if the variable is not defined using the os.environ.get() method to provide a default value if the environment variable is not defined. In this example: os.environ.get('MY_VARIABLE', 'default_value') retrieves the value of the environment variable named 'MY_VARIABLE'. If the variable is found, its value is returned. If the variable is not defined, the default value 'default_value' is returned. The code prints the value of the variable, whether it is obtained from the environment or the default value. Remember to replace 'MY_VARIABLE' with the name of the actual environment variable you want to retrieve. By providing a default value as the second argument to os.environ.get(), you can handle the case where the environment variable is not defined gracefully..

[Audio] example that demonstrates the usage of os.getenv() to retrieve the value of an environment variable. In this example: os.getenv('MY_VARIABLE') retrieves the value of the environment variable named 'MY_VARIABLE'. If the variable is found, its value is returned. If the variable is not defined, None is returned. The code checks if value is not None, indicating that the environment variable is set. It then prints the value of the variable. If value is None, it means the environment variable is not set, and a corresponding message is printed. Remember to replace 'MY_VARIABLE' with the name of the actual environment variable you want to retrieve. The os.getenv() function allows you to retrieve the value of an environment variable, and you can handle the case where the variable is not defined gracefully by checking for None in the returned value..

[Audio] set environment variables In this section we'll check quick summary of how to set environment variables in a terminal or command prompt window. Unix and MacOS There are two basic ways to set an environment variable from a bash or zsh terminal session. One is using the export keyword: A variable that is set in this way will be passed on to any programs or scripts that you start from this terminal session. Keep in mind that environment variables are not saved anywhere outside of the context of the shell session, so they will go away when you close the terminal session. An alternative way to define an environment variable is to set it in the same line where the target application is executed: This second form has the benefit that the variable is only set in the environment space of the intended application..

[Audio] To set environment variables in Windows, you can use either the command line or the Windows system settings. Here are the steps for both methods: Method 1: Using Command Prompt Open a Command Prompt window. Press Win + R to open the Run dialog. Type "cmd" and press Enter. Set an environment variable using the set command. To set a temporary variable (available for the current session): To set a permanent variable (available across sessions): Replace VARIABLE_NAME (we use name python as variable name) with the name of the variable you want to set and "variable_value" (we use py as variable value) with the desired value. Note that for setx, the variable value should be enclosed in double quotes. Verify that the variable is set. To check if a variable is set, use the echo command: Echo return variable value in this case this is Py.

[Audio] To set environment variables in Windows, you can use either the command line or the Windows system settings. Here are the steps for both methods: Method 1: Using Command Prompt Open a Command Prompt window. Press Win + R to open the Run dialog. Type "cmd" and press Enter. Set an environment variable using the set command. To set a temporary variable (available for the current session): To set a permanent variable (available across sessions): Replace VARIABLE_NAME (we use name python as variable name) with the name of the variable you want to set and "variable_value" (we use py as variable value) with the desired value. Note that for setx, the variable value should be enclosed in double quotes. Verify that the variable is set. To check if a variable is set, use the echo command: Echo return variable value in this case this is Py Remember to replace VARIABLE_NAME with the actual name of the variable you want to set or verify. Setting environment variables allows you to configure specific values that can be accessed by programs or scripts running on your Windows system..

[Audio] With OS Services, Launching external commands with subprocess is a tricky approach, to understand and apply it is recommended to know about process and sub processes. Processes and subprocesses are fundamental concepts in operating systems and programming. A process refers to an executing instance of a computer program, while a subprocess is a child process spawned by another process. In Python, the subprocess module provides functionality to create, control, and interact with subprocesses. Processes A process is an executing program with its own memory space, resources, and state. Each process has a unique process identifier (PID) that identifies it within the operating system. Processes can run concurrently and independently of each other. Processes may communicate with each other through inter-process communication (IPC) mechanisms provided by the operating system. Examples of processes include the programs running on your computer, such as text editors, web browsers, and system services. A process refers to an executing instance of a computer program..

[Audio] Subprocesses A subprocess is a child process created and managed by another process. Subprocesses are often created to perform specific tasks or execute external commands. Subprocesses inherit certain attributes and resources from the parent process, such as the environment variables and file descriptors. Subprocesses can communicate with the parent process or other processes through standard input/output (stdin/stdout), pipes, or shared memory. In Python, the subprocess module provides functions and classes to create, control, and interact with subprocesses, such as subprocess.run(), subprocess.Popen(), and more. A subprocess is a child process spawned by another process. By utilizing the subprocess module, we can launch and interact with external programs, run system commands, capture output, and handle input/output streams. It provides a way to control and communicate with subprocesses from within your Python code, allowing you to incorporate external functionality seamlessly into your applications..

[Audio] the Python subprocess Module The Python subprocess module is for launching child processes. These processes can be anything from GUI applications to the shell. The parent-child relationship of processes is where the sub in the subprocess name comes from. When you use subprocess, Python is the parent that creates a new child process. What that new child process is, is up to you. Python subprocess was originally proposed and accepted for Python 2.4 as an alternative to using the os module. Some documented changes have happened as late as 3.8. The examples in this article were tested with Python 3.10.4, but you only need 3.8+ to follow along with this. Most of your interaction with the Python subprocess module will be via the run() function. This blocking function will start a process and wait until the new process exits before moving on. The documentation recommends using run() for all cases that it can handle. For edge cases where you need more control, the Popen class can be used. Popen is the underlying class for the whole subprocess module. All functions in the subprocess module are convenience wrappers around the Popen() constructor and its instance methods. Near the end of this tutorial, you'll dive into the Popen class. Note: If you're trying to decide whether you need subprocess or not, check out the section on deciding whether you need subprocess for your task. You may come across other functions like call(), check_call(), and check_output(), but these belong to the older subprocess API from Python 3.5 and earlier. Everything these three functions do can be replicated with the newer run() function. There's also a fair amount of redundancy in the subprocess module, meaning that there are various ways to achieve the same end goal. You won't be exploring all variations in this tutorial. What you will find, though, are robust techniques that should keep you on the right path..

[Audio] The subprocess module in Python provides functionality for creating, controlling, and interacting with subprocesses. It allows you to run external programs, execute system commands, capture their output, and handle input/output streams. The subprocess module provides several functions and classes for different use cases. Here are some commonly used ones: subprocess.run(args, ..., capture_output=False, text=False) Runs a command with arguments and waits for it to complete. Arguments are passed as a list or string. capture_output=True captures the command's output. text=True returns the output as a string. subprocess.Popen(args, ..., stdin=None, stdout=None, stderr=None) Spawns a new process and allows interaction with it. Arguments are passed as a list or string. stdin, stdout, and stderr define the input/output streams. subprocess.PIPE A constant that represents a pipe for connecting the input/output streams of subprocesses. subprocess.check_output(args, ..., text=False) Runs a command and returns its output as a byte string. Raises an exception if the command returns a non-zero exit status. subprocess.call(args, ..., shell=False) Runs a command and waits for it to complete. Returns the command's exit status. subprocess.check_call(args, ..., shell=False) Runs a command and waits for it to complete. Raises an exception if the command returns a non-zero exit status. These are just a few examples of the functions and constants provided by the subprocess module. There are additional options and parameters available for handling complex scenarios, such as managing input/output streams, redirecting input/output to files, setting environment variables, and more. When working with subprocesses, it's essential to consider security implications, such as properly handling user input and ensuring the safe execution of external commands to prevent vulnerabilities like command injection..

[Audio] To launch external commands and interact with them from a Python script, you can use the subprocess module. The subprocess module provides various functions and classes for working with external processes. Here's an example that demonstrates launching an external command using subprocess: In this example: subprocess.run() is used to execute the command. It takes a list of command arguments as the first argument. In this case, ['ls', '-l'] executes the ls -l command in a Unix-like environment to list the files in the current directory. The capture_output=True argument captures the command's output, which can be accessed later. The text=True argument ensures that the command's output is returned as a string (text) rather than bytes. After executing the command, you can access its output using the stdout attribute of the subprocess.run() result. In the example, result.stdout contains the output of the ls -l command, which is then printed. You can replace ['ls', '-l'] with any other command and its arguments to execute different commands. The subprocess module provides more advanced features for controlling and interacting with external processes, such as handling input, error handling, and more..

[Audio] How to traverse file system in Python ? Suppose we have given below file structure in our system, and we want to traverse all it's branches completely from top to bottom ? Answer is walking directory trees !! Using os.walk ().

[Audio] How does os.walk() work in python ? OS.walk() generate the file names in a directory tree by walking the tree either top-down or bottom-up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames). root : Prints out directories only from what you specified. dirs : Prints out sub-directories from root. files : Prints out all files from root and directories. Walking directory trees refers to the process of traversing through directories and their subdirectories to access and perform operations on files and folders within them. Python provides the os module along with the os.path module to accomplish this task efficiently. In this example: The os.walk() function is used to traverse the directory tree rooted at the specified root directory. It returns a generator that yields a tuple (dirpath, dirnames, filenames) for each directory it encounters. dirpath is the path to the current directory being traversed, dirnames is a list of subdirectory names within the current directory, and filenames is a list of filenames in the current directory. The example showcases how to process files and directories within each directory encountered during the walk. Inside the loop, os.path.join() is used to construct the complete paths for files and subdirectories by concatenating the dirpath with the respective filename or dirname. You can perform any desired operations on the file_path or dir_path as needed. Remember to replace '/path/to/root' with the actual path of the directory you want to start walking from. This example demonstrates how to print the files and directories encountered during the walk, but you can customize the code to perform any desired operations on them. Using the os.walk() function, you can easily traverse directory trees and access files and directories within them, enabling you to perform various tasks like copying files, analyzing directory structures, searching for specific files, and more..

[Audio] In Python, paths, directories, and filenames are essential concepts when working with file systems. Here's a brief explanation of each term: Path: A path is a string representation of the location of a file or directory in a file system. It describes the hierarchical structure and the necessary steps to navigate to a specific file or directory. A path can be either absolute or relative. Absolute Path: An absolute path specifies the complete location of a file or directory from the root of the file system. It typically starts with a drive letter (e.g., "C:\folder\file.txt" on Windows) or a leading slash (e.g., "/home/user/file.txt" on Unix-like systems). Relative Path: A relative path specifies the location of a file or directory relative to the current working directory. It doesn't start from the root of the file system but instead assumes the current directory as the starting point. For example, "folder/file.txt" refers to a file named "file.txt" inside a directory named "folder" located in the current working directory. Directory: A directory, also known as a folder, is a container for organizing files and subdirectories. It represents a location within the file system that can hold files and other directories. Directories have names and are organized in a hierarchical structure. They can contain files and subdirectories. Filename: A filename is the name assigned to a file within a directory. It represents the specific name of the file, excluding the path and any directory information. For example, in the path "/home/user/file.txt", "file.txt" is the filename..

[Audio] Common Functions in os.path When working with paths, directories, and filenames in Python, the os.path module provides various functions for manipulating and working with them. These functions include: os.path.join(path, *paths): Combines one or more paths to form a single path, taking into account the appropriate path separator for the operating system. os.path.abspath(path): Returns the absolute path of a given path. os.path.dirname(path): Returns the directory name of a path, excluding the filename. os.path.basename(path): Returns the filename portion of a path. os.path.exists(path): Checks if a path exists in the file system. os.path.isdir(path): Checks if a path is a directory. os.path.isfile(path): Checks if a path is a regular file. These are just a few examples of the functions provided by os.path module. By using these functions, you can manipulate and extract information from paths, directories, and filenames, making it easier to navigate and operate on files and directories in your Python programs..

[Audio] Example1: Combining Paths : The path separator varies depending on the OS. UNIX (including Mac) uses the slash /, while Windows uses the backslash \. Example2: Extracting Components : Use os.path.basename() to get the filename (basename) from a path string. Example3: Checking Existence Example4: Checking Directory or File.

[Audio] Working with file systems is a common task in many programming projects. Python provides a rich set of modules and functions to handle file operations, such as creating, reading, writing, and deleting files, as well as manipulating directories and file paths. In python, the file system contains the files and directories. To handle these files and directories python supports "os" module. Python has the "os" module, which provides us with many useful methods to work with directories (and files as well). The os module provides us the methods that are involved in file processing operations and directory processing like renaming, deleting, get current directory, changing directory etc. key concepts and operations when working with file systems in Python are : File Operations: File Paths and Directories: File and Directory Information: Working with File Objects: Exception Handling:.

[Audio] File Operations: Creating a File: Use the open() function with the "w" or "x" mode to create a new file. Reading from a File: Use the open() function with the "r" mode to read data from an existing file. Writing to a File: Use the open() function with the "w" or "a" mode to write or append data to a file. Closing a File: Call the close() method on the file object to release system resources. Deleting a File: Use the os.remove() function to delete a file..

[Audio] File Operations Example In this example: We start by creating a file named "example.txt" using the open() function with the "w" mode. We write the text "Hello, World!" to the file using the write() method and then close the file. Next, we open the file in "r" mode to read its content using the read() method. We store the content in the content variable and then close the file. We print the content to the console. We open the file in "a" mode to append a new line of text using the write() method. We append the line "This is an appended line." and close the file. We use the os.remove() function to delete the file. Finally, we check the existence of the file using the os.path.exists() function. If the file exists, we print a message indicating its existence; otherwise, we print a message indicating that it doesn't exist. This example demonstrates the basic file operations of creating, reading, appending, and deleting files, as well as checking their existence using Python's file handling capabilities. Remember to handle file operations with care and properly manage file resources by closing files after use..

[Audio] File Paths and Directories: Manipulating Paths: Use the os.path module to join, split, or manipulate file paths and directory names. Creating Directories: Use the os.mkdir() or os.makedirs() functions to create directories. Deleting Directories: Use the os.rmdir() or shutil.rmtree() functions to remove directories and their contents..

[Audio] In this example: We define a file path "path/to/file.txt". We use the os.path module to extract various components of the file path: the absolute path using os.path.abspath(), the directory name using os.path.dirname(), and the filename using os.path.basename(). We then create directories using the os.mkdir() function to create a single directory "new_directory", and os.makedirs() to create nested directories "nested/directory". The exist_ok=True parameter ensures that the directories are created even if they already exist. Next, we demonstrate deleting directories using os.rmdir() to remove a single empty directory, and os.removedirs() to remove nested directories. Note that directories must be empty to be deleted. Finally, we check the existence of the directory using os.path.exists(), and print a message indicating whether the directory exists or not. This example showcases the manipulation of file paths using os.path, creating and deleting directories, and checking their existence using Python's file system capabilities. Remember to handle file paths and directories carefully, and always consider proper error handling and validation when working with file systems..

[Audio] File and Directory Information: Checking Existence: Use the os.path.exists() function to check if a file or directory exists. Getting File Information: Use the os.stat() function or the os.path.getsize(), os.path.getmtime(), etc. functions to retrieve information about a file. Listing Directory Contents: Use the os.listdir() function or the os.scandir() function for more advanced directory listing..

[Audio] In this example: We start by obtaining file information. We use the os.path.getsize() function to get the size of a file in bytes, and os.path.getmtime() to get the last modified time of the file. We print the file path, size, and last modified time to the console. Next, we list the contents of a directory using os.listdir(). We iterate over the list of files, and for each file, we join the directory path and the file name using os.path.join() to get the complete file path. We then check if the path corresponds to a file using os.path.isfile(), and if it does, we print the file name. Finally, we check if a given path is a directory using os.path.isdir(), and print a message indicating whether it is a directory or not. This example demonstrates how to obtain file size, last modified time, and how to list files in a directory using Python's file system functions. It also shows how to check if a path is a directory. Remember to handle exceptions and ensure proper error handling when working with file and directory operations..

[Audio] Working with File Objects: Reading from Files: Use the read() or readline() methods to read data from a file object. Writing to Files: Use the write() or writelines() methods to write data to a file object..

[Audio] In this example: We create a file named "example.txt" using the open() function with the "w" mode. We write multiple lines of text to the file using the write() method, and then close the file. Next, we open the file in "r" mode to read its content using the read() method. We store the content in the content variable and then close the file. We print the file content to the console. We open the file again, this time in "a" mode to append additional text to the file using the write() method. We append the line "Appending additional text." and close the file. Lastly, we open the file in "r" mode once more and demonstrate reading the file line by line using a loop. Each line is printed after stripping any trailing whitespace. This example showcases how to open files in different modes (writing, reading, appending), how to write and read content from files, and how to iterate over file objects line by line. Remember to properly close file objects after use to release system resources and ensure data integrity..

[Audio] Exception Handling: When working with file operations, it's important to handle exceptions properly, such as FileNotFoundError, PermissionError, or IOError, to gracefully deal with potential errors that may occur during file handling. Remember to handle file operations with care, especially when dealing with sensitive data or performing operations that can modify or delete files. Always close files after use and implement error handling to ensure the reliability and integrity of your code. Python's standard library offers additional modules for more specialized file operations, such as shutil for file and directory operations, glob for file pattern matching, and tempfile for working with temporary files. By utilizing the various modules and functions available in Python, you can effectively work with file systems, perform file operations, manipulate directories, and manage file paths to accomplish a wide range of tasks in your Python programs..

[Audio] In this example: We try to open a file named "nonexistent_file.txt" in "r" mode. Inside the try block, we read the content of the file and print it to the console. If any exception occurs during these operations, the program will jump to the appropriate except block. The first except block handles the FileNotFoundError specifically, which is raised when the file does not exist. It prints a message indicating that the file does not exist. The second except block handles the IOError (a subclass of OSError), which represents input/output-related errors. It catches and handles any other input/output-related errors that may occur while reading the file. The third except block handles any other exceptions that are not specifically caught by the previous blocks. It provides a generic error message to handle unexpected errors. The finally block is executed regardless of whether an exception occurs or not. It is useful for cleaning up resources or performing final actions. In this case, it simply prints a completion message. This example demonstrates the use of exception handling to gracefully handle potential errors when working with file operations. It showcases specific exception handling, generic exception handling, and the finally block for finalizing actions. Remember to handle exceptions appropriately in your code to ensure the reliability and stability of your program..

[Audio] Course Prerequisites are Basic knowledge of Python programming Familiarity with programming concepts like variables, functions, and control flow.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Network Programming Objectives Review principles of networking Contrast TCP and UDP features Show how Python programs access networking functionality Give examples of client and server program structures Demonstrate some Python network libraries Give pointers to other network functionality.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The idea behind this workshop is to quickly get you to the stage where you can write useful network-based Python applications. We start out by looking at an overview of networking in the TCP/IP world, and then examine the basic functionality Python offers for network programming, through the so-called sockets interface. Then we move on to looking at some of the canned libraries you can use for specific purposes..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The ISO spent a long time developing a standardized open systems interconnection (OSI) model, based on seven layers. This model was the basis of early networking systems using X.25 as the network layer. As we will see later, TCP/IP uses a simplified model more appropriate to modern transmission systems. This reduced protocol overhead is one of the major reasons for TCP/IP's success in the Open Systems world..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Protocol layering is really quite easy to understand. The advantage is that different layers can cope with different functions. It would be a very difficult world if a file transfer application had to know whether it was running over an Etherenet or a token ring network and take different actions accordingly. Fortunately the physical network is several layers down from the application code, which is more or less completely isolated from such issues. We usually think of the application data as traveling "down the stack" to emerge, at the bottom, as a transmission across some network interface. When the transmission reaches its destination it flows "up the stack" again until it is delievered, with all lower-level headers stripped off, to the receiving application process..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 A TCP/IP application must provide its own presentation- and session-layer services if it needs to use them. Many times the data that are interchanged are simple enough not to require presentation layer services, and the interactions are short so no session-layer services are needed. The TCP/IP application layer therefore corresponds to the top three layers of the OSI model. From an application point of view the distinction between datalink and physical layer is irrelevant. They are both collapsed into a single subnetwork layer. The really important point is that the application can treat the transport-layer API as its way to communicate with remote processes. The layered complexity is essentially invisible to the application code, and is called on by the transport layer libraries without any action being required by the application.. The subnetwork layer data is actually an IP datagram, whose data is a TCP or UDP segment, whose data is an application protocol data unit..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Another advantage of TCP/IP is the incredible adaptability of the network layer. It's hard to think of a transmission medium that hasn't been used to carry IP datagrams in some network or other! Two major transport layer protocols (TCP and UDP) are used, though IP can carry many others, including ISO protocols in some cases. Most applications are designed to run over one or the other of these two major transport protocols, but certain applications such as NFS (the Network File System) and DNS (the Domain Name System) are designed to use either transport layer. Usually TCP is used over wide-area links, and UDP is used over LANs, which are typically rather more reliable. When we talk about "TCP/IP" we actually assume the presence of all the components shown above, and more besides..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The network layer of TCP/IP makes no guarantees of delivery: at any hop along the way a packet may be discarded, either because transmission errors have been detected or simply because the receiving equipment does not have the capacity to process it. In such circumstances the originating host may or may not receive an ICMP (Internet Control Message Protocol) message detailing the reason for non-delivery. ICMP messages can be useful in helping hosts to change to more appropriate behavior, but are not always actioned even when raised. Since different physical network use different addressing schemes it's important that IP provides a unified scheme independent of the underlying hardware. This is crucial when building large internets (an "internet" with a small "I" is any interconnected collection of IP networks). You can find a detailed description of IP addressing at http://www.holdenweb.com/students/3comip.pdf.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 You can think of UDP as similar to the postal service – packets go out, they can be lost along the way if an intermediate router runs out of resources or if they are mangled by noise, no acknowledgements are issued. UDP applications are therefore usually simpler messaging-style applications, where the only recovery action will be a fixed number of retries. DNS is a good example -- although it can use TCP, most DNS traffic is carried by UDP. If a server does not reply, or if the reply (or the request) gets lost, the consequences are usually not tragic. If you fail to resolve a host name because of such an error you (as a user) will usually be quite happy to try again, and this failure will not have been significant. As you will see later, a server binds to a specific port when it starts up. Originally a client had to "just know" what port the server would be listening on, and so default port numbers were allocated: 80 for HTTP, 25 for SMTP and so on. Later on, services like the PortMapper were introduced so that a server could use any port, and register it with the Portmapper – the clients would start with a PortMapper enquiry to find out which port their server was listening on..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 A TCP conversation requires specific endpoints, so you can think of it as more like a telephone call: one system calls another, and has to get a reply before communication can take place. Because of the need for defined endpoints, TCP has no broadcast capability, unlike UDP. Because TCP is a reliable protocol the possibility of error is rather low (though not zero) – errors are detected and corrected in the transport layer. This makes it relatively easy to use, since the applications can assume the data they send will, eventually, be received in the absence of a complete connectivity failure. TCP is a much more sophisticated protocol than UDP and includes both error detection and correction, relieving applications of housekeeping tasks which would otherwise quickly become burdensome. There is no need to transmit data is any particular block size, and an application is free to send as few as one byte or as much as several megabytes at a time. The transport layer will take responsibility for buffering this data and sending it out as a stream of packets of an appropriate size..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Although not all client/server systems are based around the idea of a passive and an active opener this is currently the dominant paradigm for networking, used by FTP, Telnet, DNS, SMTP, SNMP and many others. The client and the server will typically use the socket API, originally devised as part of the bsd Unix implementation, to interact with the networking features of their host operating system. A socket gives the application code access to the network interface using convenient calls to transmit and receive data. Python standardizes access to sockets to eliminate the need to worry about platform differences -- network programming is remarkably consistent among Linux, UNIX and Windows. The fact that a client port number is guaranteed unique on its host allows traffic to be multiplexed and demultiplexed over shared network paths. Even when two telnet clients on the same host are in touch with telnet servers on the same host the client address/port number combination is unique, allowing the traffic to be delivered to the correct process..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 When UDP is used, the server creates a socket and binds address(es) and a port number to it. The server then waits for incoming data (remember: UDP is connectionless). The clients also create a socket, then they bind it to the appropriate interface – typically allowing the transport layer to choose an epehemeral (short-lived) port number rather than specifying a particular port. The client sends data to the server, which awakes from its blocked state and starts to compute its response. Meantime the client has issued a recvfrom() using the same address it sent the data to, and is blocked awaiting the response which should eventually arrive from the server. When the server sends its result back, it goes to the address and port number the incoming data was received from. The server then loops around to wait for another request. The arrival of the server's data unblocks the client, which can then continue. This is something of a simplification: using a library based on the select() system call it is possible to use sockets in a non-blocking fashion. This does complicate the code somewhat, however..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 It's more usual to import the whole socket library and use qualified names, but the from statement is a convenient way to access only specific names from a module. I did this to keep code lines on the slide shorter, and hence more readable. The following code is an equivalent but rather more conventional way to create the socket: import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) This achieves exactly the same ends. Which style you use is largely a matter of taste and readability. The remainder of the code will run unchanged since the other features it uses are attributes of the socket, and are accessed the same way no matter how the socket was created. Note carefully that the bind() call takes a single argument, a tuple containing an IP address string and a port number. If the IP address is the empty string then the code will bind to all interfaces, which is how most servers actually start up. The example above is a little more secure, since only local processes can connect via the local loopback interface. The code on this slide is on your CD as udpserv1.py if you want to run it now..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The client specifies port number zero to indicate that it simply wants an ephemeral port – this is more efficient than attempting to use a specific port number because the port requested might already be in use and then the bind() call would fail. The getsocketname() call tells the user the address and port number being used for the client end of the communication. While this isn't an essential part of the program it's useful debugging data. The client simply sends the data and (usually) receives a reply from the server. This particular program is somewhat inadequate in terms of error checking: if the server's response is somehow lost then the client will hang forever. The socket library was recently (2.3) updated to include timeout features that previously had only been available in third-party additions..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This simple exercise allows you to verify that your networking setup is OK and that Python is making things happen as it should. The programs are very simple, with no error checking at all. When you run the client with no server listening it will hang, waiting forever for a non-existent server to reply. Clearly this situation isn't ideal when everything is supposed to run unattended  Note that these programs will run on almost an platform that Python supports. The socket layer is very robust, and considerably eases portability problems in networked applications. Once I found out how easy Ptyhon was to use in this way I entirely abandoned any idea of using C or C++. No hair shirts for me … You could also try to run two copies of the server. What happens then?.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Please try to overlook the fact that certain inhabitants of Pythonia appear to find it necessary to drag in obscure references to Monty Python's Flying Circus. The name is in no way meaningful. The creosote module shows how useful UDP can be for messaging-style applications. Debugging messages can be emitted by a program, and it doesn't matter whether anything is listening or not. You could even use IP address 255.255.255.255 (the local broadcast address) as the destination for reports, or the directed broadcast address of your local subnet. In either case, any machine on the LAN that was running a creosote debugger would pick uo the broadcasts and display them!.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This is pretty much the output routine from the Mr. Creosote module. The definition line shows how Python allows you to define default values. In this case the (destination) host defaults to the machine the code is running on, and the (destination) port number to a value previously defined globally in the module. The bind() call requests an ephemeral port usable on any interface. The while loop sends chunks (whose size, BUFSIZE, is also a module parameter) which are successively deleted from the message. The loop terminates when the message is reduced to an empty string..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This slide is a simplification of the real code, which has logging abilities that I have omitted clarity. Again the port defaults to a value preset elsewhere in the module. The function binds to all local addresses on the given port number (the empty string) and proceeds to receive messages on that socket. The print statement uses backticks to produce a readable representation of the message even if it contains binary characters, and strips the quotes off with the [1:-1] slicing. The while 1 loop is typical of server code: a good server will run forever. In the case of the recvfrom() call the argument sets the maximum message size than can be received..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 You may not necessarily have a laptop with you today, but I am hoping that enough of you will have brought them with you to make exercises practical. All the code is available on CD or via the web, so you will be able to run the exercises when you get back home anyway. To assist those who aren't able to work hands-on I will demonstrate how code can be written and run on my own laptop, and field questions interactively. Because a current focus is integration of Windows and open source technologies I will be running the exercise code on a Windows 2000 laptop, sometimes in a command window and sometimes under Cygwin. The exercises have all been carefully tested on Linux, Windows and Cygwin, and so you shouldn't have any portability problems. That's one of Python's strengths!.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 A connection-oriented server creates a socket, binds it to one or more local ports on which it will listen for connections, and then puts the socket into the listening state to wait for incoming connections. At this point the server process blocks until a connection request arrives. A client creates its own socket, usually without specifying any particular port number, and then connects to the endpoint the server is listening on. The server's accept() call returns a new socket that the server can use to send data across this particular connection.. The two parties then exchange data using read() and write() calls. The major limitation of this structure is the non-overlapped nature of the request handling in the server. Theoretically it's possible for the server to use its original socket to listen for further requests while the current request is being handled, but that isn't shown here. You will learn how to overcome this limitation using standard library classes..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Connection-oriented servers are a little more complex because the connection allows clients and servers to interact across multiple send() and recv() calls. The server blocks in accept() until a server connects. The return value from accept() is a tuple consisting of a socket and the client address (which is the usual (address, port) tuple). The server can, if it chooses, use multitasking techniques such as creating a new thread or forking a new process to allow it to handle several concurrent connections. Either solution allows the connection to be processed while the main control loop returns to execute another accept() and deal with the next client connection. Since each connection generates a new server-side socket there is no conflict between the different conversations, and the server can continue to use the listen()ing socket to listen for incoming connections while it serves already-connected clients..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This very simple client just sends a single message and receives a single response. More typical code will send a request and use its understanding of the application protocol to determine when the response to that request has been completed. Protocols like HTTP 1.0 use a separate connection for each request. Protocols like telnet can exchange thousands of messages before the connection is closed, and the code tends to be more complex iin that case. Under normal circumstances a recv() call guarantees that at least one byte of data will be returned. When a program seems the empty string (zero bytes) returned from the recv() it knows that the other end has terminated the connection by calling close() on its socket..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Although there is no formal presentation layer in the TCP/IP suite, certain data types are so commonly interchanged that it is useful to be able to transmit them in network-neutral format. This makes it easy for little-endian and big-endian machines to communicate with each other. l stands for long, a 32-bit value, and s signifies a 16-bit short. If you need to communicate IP addresses as a part of your application, the inet_*() functions allow you to do so efficiently..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 These utility functions are useful ways to access the domain name of the host your are running on, and other DNS services. A typical use would be as follows: >>> import socket >>> socket.gethostbyaddr("10.0.0.10") ('prom01.holdenweb.com', [], ['10.0.0.10']) This shows interactive use of the Python interpreter, which is handy for debugging purposes. If you simply enter an expression, the interpreter prints out the resulting value..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Sometimes if the protocol involves variable-length strings it's easier to use the file-based functions like readline() and write() to handle I/O. Calling s.makefile() on a socket s yields an object sufficiently "file-like" that it allows you to write your programs in this more familiar style. Otherwise it's sometimes necessary to assemble input strings from a sequence of recv() calls, and add line terminators to the strings you pass to send(). The socket-based file object can be closed without closing the underlying socket. This paradigm also makes it easier to adapt existing code, written to handle files, to network applications..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The idea of this exercise is to show two programs: one acts as a server, and each time it receives a connection it repeatedly reads strings from the client, returning the same string in upper case. The second simply acts as a client, reading strings from the user, sending them to the server and echoing the server's output back until an empty line is entered, at which time it closes the connection to the server and terminates. This is a somewhat fragile set-up for a number of reasons. The first is that if a client terminates without properly closing its server connection the server may not detect this situation, and will continue to wait for input. The effect of this fault is magnified by the server's inability to handle multiple connections – it won't accept a second connection until the first one terminates. We'll see how to get around this problem shortly..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Mostly we are concerned with IP v4 sockets and the related services, although IPv6 is coming, and Python 2.3 is expanding socket support to include it. Unix named pipes are available if you need them, and work in more or less the same ways as the IP version 4 sockets we'll be using in the class..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Most of the services we are interested in will be TCP-based, and therefore use SOCK_STREAM sockets. We do look briefly at SOCK_DGRAM sockets. SOCK_RAW sockets have been the basis of (for example) ping programs to generate and handle ICMP traffic, but this is beyond the scope of this presentation. Still. It's always useful to know that you can get access to the network hardware if you need it – most times you don't, but typically when you need such access you really need it..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 If you are an experienced C network programmer you will be familiar with the socket library provided to programmers in that language, which has been widely ported to various OS platforms. Although most of the features of these libraries are available from the socket modue, it is surprising how much you can do without using more than the basics outlined in this tutorial. We are focusing on the features you need to get your network applications up and running – the rest can follow later. There is a wealth of Python code already in the public domain that you can use as examples..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 A big problem with network programming is: what do you do when the expected responses aren't received? In the case of TCP connection attempts, if no reply is received to the initial connection attempt it isn't unusual for the adaptive timeout to keep retrying at exponentially larger intervals, with the result that no exception occurs for over an hour. UDP sockets will hang indefinitely if they are awaiting a message that somehow gets lost, or is never transmitted. Timeouts are therefore a valuable way to ensure that failures are detected in a timely manner, and I encourage you to use this feature, which has been added to the standard library in the most recent release. If you are using an older release then you should download the timeoutsocket module from http://www.timo-tasi.org/python/timeoutsocket.py Having made you aware of the problems we are going to ignore them for the rest of the class – timeouts, although invaluable in practice, can complicate the code to the extent that useful programs take more than one slide .

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The SocketServer module is the basis for (among other things) a family of HTTP servers which are also a part of the standard Python library. It offers a framework that allows you to write some quite sophisticated servers, and is very useful for rapid network experimentation. You should be aware, though, that robustness and throughput are both somewhat below professional servers, so you should not make this the basis of production servers unless you are fairly certain the load will be light. Having said that, the ability to build an experimental server with a (very) few lines of code is invaluable, so SocketServer should definitely be a part of your programming vocabulary if you do a lot of protocol development or similar work. Python's conciseness and readability are a real asset here. Python's ability to handle multiple inheritance is also useful -- the module lets you add the ability to handle each request in a separate thread or a separate process by adding so-called mixin classes to your server's base classes. This is particularly useful for connection-oriented servers, which would otherwise stall additional clients until an existing session was completely terminated, as we saw in the earlier demonstration..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The SocketServer framework is quite easy to use and yet, as the HTTP server modules we examine later will show, it can be the basis of some very powerful functionality. The UDP server code treats each incoming packet as a standalone request, which is pretty much what you expect with UDP – there's no such thing as a connection, and so all the request's data has to arrive at once. The address you provide is normally a two-element tuple: the first element is a string containing the address you want to bind to (the empty string means "bind to all addresses") and the port number to listen on. This approach is common to almost all Python socket code. For the handler you provide a class, and the server framework creates a brand new instance of that class each time a new request comes in. [Technically you provide a callable, but when you call it the result must be an object that possesses all required methods and attributes: in practice it's easiest to provide a subclass of the SocketServer.BaseRequestHandler class provided with the library]. Usually all you have to define in your subclass is your own handle() method..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The key to SocketServer programs is to get the handle() method right. It isn't difficult, as the framework is fairly intelligent and does most of what needs to be done..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 To give an immediate flavor of the utility of this library, we present an example of both a connectionless and a connection-oriented server. The service is simply that of translating the case of text. As the slides point out, though, the code is easily adapted to providing other functionality – the server framework gives you pretty much everything you need..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This server example shows the basic way to reply to a UDP client. UCHandler is an upper-case handler class. Its handle() method gets the data and the return address from the instance's request attribute. It replies to the client using the socket's sendto() method. Python strings are objects with their own methods. The upper() method of a string simply returns the upper-case conversion of the string. It would be easy to replace the data.upper() expression with a more complicate function of data to implement different handler functionality..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Since we can't make connections using UDP, each input generates a datagram to the server, and the client then waits for the server's return datagram. In practice, of course, it would be rather better to set a timeout on the receive so that the program doesn't hang forever if something goes wrong..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The TCP server structure is somewhat different from the UDP server because the handle() method has to handle a complete connection from beginning to end, potentially involving a number of interations between the client and the server. So it implements a loop, which requires a terminating condition. This server's handle() method terminates when it receives a blank line from the client. It would be more usual to terminate when a zero-length input was received, as this is the conventional indication that the client has closed the socket from its end. However that would need a client that closes the socket, so the empty line is rather easier to deal with on the client side..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 As you can see, the client connects to the server and then repeatedly reads data from the user. In line with the server's requirements, the client sends the blank line that terminates the input before closing the socket. The Python socket is supposed to be closed automatically when the program ends, but it is akways safe to explicitly close the socket, and it's better practice than relying on something that may or may not happen..

Python System Scripting: Network Programming : Exercise 4: SocketServer Usage.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Clearly a server will be able to handle more work if processing one request doesn't hold others up. The basic SocketServer server, however, only handles a single request at a time. Fortunately the solution is ready to hand – the SocketServer module's author has already thought about this problem, and provided mixin classes to give forking and threading as alternative server behaviors..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This diagram shows what needs to happen to allow parallel handling of concurrent requests. In the case of a forking server process, the forked child handles the request and the parent loops around to accept() another connection. In the case of a threading server the two threads coexist in the same process, and the process shares its CPU allocation between the new thread that processes the request and the original thread, which loops around to accept() another connection. The essential point is that the server no longer has to handle one connection completely before it can handle the next request. This isn't so important for UDP servers, where the request and response tend to be short and sweet. In the TCP world, however, a "request" is actually a connection that can be exceedingly long-lived – think of a Telnet or ssh session, for example..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This is a very powerful way to make your servers asynchronous. In the forking and threading variants, service for one request becomes independent of the main server loop because the handler gets its own process (in the forking model) or thread (in the threading model). Once the server has started the request handler it is free to loop round again and accept another request almost immediately. A measure of Python's power is how easily it is possible to adapt the standard servers – the ThreadingTCPServer and ThreadingTCPServer classes are actually quite simple, as you will see on the next slide..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 You might think the mix-in classes would be complex, but this is not the case. Forking adds maybe thirty lines of code, and threading around twenty lines. The mix-in classes override the process_request() method of their client classes, and so their definition is used in preference to the definition in the server class (TCPServer or UDPServer). This inheritance model makes it relatively easy to add orthogonal functionality to classes. The forking variants do not work on Windows platforms because the os.fork() primitive is unavailable. The threading variants work on all common platforms..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Another, different way to specify asynchronous processing uses these modules. They are based on the use of the select() system call to define "events" that can occur asynchronously, and each active or lstening socket is a "channel". A basic asyncore server program will create a single channel which is its listening socket, and then call the asyncore.loop() function. As connections come in the server will add further channels to handle events related to these connections. Events cause the loop to call channel methods such as handle_read() and handle_write(), which cause data transfer between the server and the remote client. The asynchat module adds the ability to generate events when particular strings appear in the incoming data stream, which is useful for handling protocols that use delimiters rather than fixed-length strings..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This shows that it isn't too difficult to adapt the original servers so they can handle multiple concurrent connections. SocketServer limits Windows to multi-threading solutions, but on Unix you have the choice of multithreading or process forking..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The news is realtively good if you want to use the more common protocols. Lots of batteries included, although it would always be nice to see more client modules incorporated into the core..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Since Python is inherently object-oriented, it makes sense to encapsulate the connection with the server as an instance of some class. Then you have no problems about where to store connection state, and it's easy to generate multiple connections in the same program – if you do that, of course, you can get problems unless each client has its own thread. As we've seen, adding asynchronous behavior is relatively easy on the server side, because mixin classes are provided for explicit support. It's less straightforward on the client side, but it's less frequently required too, so it still tends not to be a problem. The third-party twisted framework is worth investigating (http://www.twistedmatrix.com/) if you have a need for an event-driven netwrok programming framework; the twisted code has been used to implement a wide variety of client- and server-side network functionality..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The sendmail() method either raises an exception (if none of the recipients is accepted) or returns a dictionary. Each dictionary entry has an unacceptable recipient address as a key, and the value associated with thast key is a tuple consisting of the numeric error code and the error message returned by the SMTP server. For a completely successful send the returned dictionary will be empty. This makes it relatively easy to decode the results of an email transmission..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Note that the addresses of the sender and recipients aren't taken from the message headers, but from the arguments to the sendmail() method. While the two sets of addresses are usually the same, this isn't invariably the case. This is one reason why spam is so common now – the SMTP protocol was never intended to cope with people who lied about who they are!.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Since it's entirely possible that the transmission will generate exceptions, it's always safest to allow for them. The sample code in this presentation isn't the best example of defensive programming, but the slides would have been rather too busy if all possible exceptions had been handled. At least it does report both partial and total failure..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 POP is probably the most popular mail user agent protocol. IMAP is coming up fast, but it's a much more complex protocol, and well outside the scope of anything shorter than a book. Once you have connected to a server and authenticated yourself, you can find out how much mail is waiting in your mailbox..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 You then retrieve the messages one at a time, and can mark them for deletion or not, as the case may be. The POP protocol specifies that the mailbox doesn't get updated until the client terminates the session. That's why you sometimes get duplicate messages when a network connection goes down during a POP3 session..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Again the error checking is a little vestigial, but it's important that you realise how easy it is for Python to deal with the kind of problem that network code can generate. At the end of this first page of the code, p is an authenticated POP3 connection waiting and ready to go..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The server returns a list of message descriptions as the second element of the result of the list() method. The for loop iterates over this list of message descriptions to process the whole content of the mailbox. Each message description is itself made up of a message number and a size. These are saved in an unpacking assignment, and the message number is used to retrieve the message. The message is reconstituted as a file-like StringIO object to allow it to be parsed using the standard rfc822 library (more modern code would use the email library). Some of the message's details are then printed out. This shows you how easy it is to handle email in Python – a complete mailbox processor in less than thirty lines of code!.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 FTP is a rather more complex protocol, so the coverage here is limited to the bare essentials. I focus on the things that I found most difficult to comprehend when I was a Python beginner..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The retrieval methods use a callback paradigm – you call the retrieval method, providing a callback function which the method's code will call whenever it has data to dispose of. The callback can be a plain function, or any other Python callable. When binary files are transferred then typically the callback will be a function (or the bound method of an object) that writes each chunk out to a local file as it is received..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Text retrieval is similar to binary retrieval, with the exception that the callback is actiavted for each received line. The storage methods use a file that you have already opened to provide the transfer content, and simply read the file as appropriate and necessary. One of the weaknesses of the FTP module is that you have to know enough about the FTP protocol to be able to formulate the commands to the server, which isn't strictly necessary. There's a more complicated FTP example at http://www.holdenweb.com/Python/PDCode/ftpStream.py This shows transfers of different sizes, and explains how it's relatively easy to layer the behavior you want over a Python library whose interface isn't convenient for you..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 A rather simple example, but at least it shoes you how you can use the library to perform real work. Note that a Writer instance is callable – what actually gets called is its __call__() special method..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 There's a large variety of web-related libraries. Many of the techniques you need are the same as those used in the earlier examples..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The urllib code can handle just about anything that a modern browser can handle, though it isn't going to support secure communications any time soon. You need urllib2 for that. Ongoing efforts in the Python community continue to try to make web services more accessible. There are also SOAP libraries, and lots of XML handling code of various types..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The interactive interpreter is a very easy way to find out how the various modules work., and you are encouraged to try things out to assist your understanding from the documentation. Here we see a simple exposition of urllib..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The newer library is better organized, which means it is more flexible (for example, people have managed to handle https communications with it), but this makes it correspondingly more difficult to use.If urllib can handle your needs it's the best option for a quick solution. For a better-engineered solution, however, you will find urllib2 more satisfactory..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 All the core Python web servers are based on the same server code. The facilities of the server are enhanced by adding methods to the request handlers. For example, to handle POST requests the handler has to provide a do_post method..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 When I first published this code, sombody mailed me to compain that writing a web server couldn't possibly be that simple! It does seem quite amazing that three (logical) lines of code will get you on the web. The Python tutor group spent some time playing with this code. It was fascinating for language beginners to be able to test their own web content with such a small piece of code. Although the server has some limitations it's ideal for local use. Your browser gets to do real HTTP (1.0) interactions!.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 The code of the CGI module is rather complicated, mostly because of the need to maintain backward compatibility with older versions and because it needs to handle the rare complexity of forms with client-side files embedded. Since this complexity isn't needed by most applications, I focus on the bits you will need most of the time. Again, you will find that you can write CGI code quite easily in Python under a variety of web servers – the beauty of a standard environment!.

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 Strictly speakling, each value stored in the FieldStorage can be another instance of FieldStorage. Often it will instead be an instance of the (signature-compatible but considerably less complex) MiniFieldStorage class, internal to the cgi module..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 There's nothing worse than a mysterious web application error that just produces a blank page. Unfortunately this can often be what the user sees if you just let a Python failure generate a traceback on standard error – the traceback gets recorded in the server logs, but this isn't ideal, especially when the server isn't directly under your control. The cgitb module saves you from such embarrassments. Here you see the traceback from a deliberately-provoked divide-by-zero error, somewhat truncated for space reasons..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004 This is about as short and simple as a CGI can be. It dumps the value of three named form fields to the page output. Note that like all CGIs it has to produce appropriate HTTP headers. The Python triple-quote notation is very useful for generating string constants with multi-line values. You also see here a couple of examples of string formatting, modelled after the UNIX C library's sprintf family of functions. The left-hand operand of the % operator is the format string, the right-hand operand is a tuple of values to be substituted. If you only have one value for substitution you don't even need to use a tuple, as the second example at the end of the code shows..

[Audio] Python Network Programming LinuxWorld, New York, January 20, 2004.

[Audio] Course Prerequisites are Basic knowledge of Python programming Familiarity with programming concepts like variables, functions, and control flow.

[Audio] Scripting for System Administration: Running external programs How can you execute an external program or call a system command from within a Python program? Solution is just simple Running external programs is a common task in system administration scripting. Python provides several modules and functions to execute external commands and programs..

[Audio] Let's check os dot system(): function for executing external command The os dot system() function allows you to execute a command in the system shell. It returns the exit status of the command. In this example: The os.system() function is used to execute external commands in the system shell. The first command 'ls' lists the files in the current directory. The second command 'cp ' copies a file from the source path to the destination path. Note the usage of an f-string to insert the values of the source and destination variables into the command. When you run the script, the specified commands will be executed in the system shell. The output or result of the commands will be displayed directly in the console. Please note that while os.system() is a simple way to run external commands, it has some limitations and security considerations. It's recommended to use subprocess module for more advanced control and flexibility when executing external commands in your Python scripts..

[Audio] subprocess.run(): The subprocess.run() function is a more powerful and flexible way to run external commands. It allows yo u to control input/output, handle errors, and capture command output. In this example: The subprocess.run() function is used to execute external commands. The first command ['ls'] lists the files in the current directory. The capture_output=True argument captures the command output, and text=True decodes the output as text. The second command ['cp', source, destination] copies a file from the source path to the destination path. The result of the command is captured, and the return code is checked to determine if the command executed successfully. Based on the return code, appropriate messages are printed to indicate the success or failure of the command. When you run the script, the specified commands will be executed, and the output or result of the commands will be captured and displayed in the console. subprocess.run() provides more control over executing external commands compared to os.system(). It allows you to handle input/output, capture command output, check the return code, and more. Additionally, it supports capturing both the standard output and standard error streams, providing more detailed information about the command execution. Remember to handle exceptions that may occur during the execution of the external command to ensure robustness in your script..

[Audio] subprocess.Popen(): The subprocess.Popen() class provides even more control over executing external commands. It allows you to interact with the running process, communicate with it, and handle input/output streams. In this example: The subprocess.Popen() class is used to create a subprocess and run the external command. The first command ['ls', '-l'] lists the files in the current directory with detailed information. The stdout=subprocess.PIPE argument captures the command output, and text=True decodes the output as text. The second command ['cp', source, destination] copies a file from the source path to the destination path. The stdout=subprocess.PIPE and stderr=subprocess.PIPE arguments capture the standard output and standard error streams of the command, respectively. The communicate() method is called on the process object to capture the output streams of the command. The output is stored in the output variable for the first command and in the stdout and stderr variables for the second command. Based on the return code of the process (process.returncode), appropriate messages are printed to indicate the success or failure of the command. When you run the script, the specified commands will be executed, and the output or result of the commands will be captured and displayed in the console. subprocess.Popen() provides more control over executing external commands compared to os.system() and subprocess.run(). It allows you to interact with the running process, capture output streams, handle input/output, and more. It provides more flexibility for complex scenarios where you need fine-grained control over the execution and communication with the external command. Remember to handle exceptions that may occur during the execution of the external command and handle the output streams (stdout and stderr) appropriately in your script..

[Audio] shlex module: The shlex module helps in parsing command-line arguments into shell-like syntax. It provides the shlex.split() function to split a command into a list of arguments. In this example: We import the shlex module. The shlex.split() function is used to tokenize the command-line string or any shell-like syntax. In the first example, the command_line string represents a command with arguments. shlex.split() splits the string into individual tokens based on whitespace, considering quotes as necessary to preserve arguments with spaces. The result is a list of tokens. The second example demonstrates tokenizing a string containing quoted text. The string_with_quotes is split into tokens, including the quoted text as a single token. When you run the script, the command-line string or the input string will be tokenized using shlex.split(), and the resulting tokens will be printed. The shlex module provides various functions and classes for different use cases, such as shlex.quote() for quoting strings, shlex.join() for joining tokens into a string, and shlex.split() for tokenizing. It also allows you to customize the parsing behavior by using the shlex.shlex class. The shlex module is particularly useful when dealing with command-line arguments or shell-like syntax in your scripts, as it handles the complexities of tokenizing and preserving quoted text..

[Audio] When writing scripts for system administration, it's common to accept command-line arguments to make the script more flexible and customizable. Python provides several modules and libraries to parse command-line arguments easily. Here are some popular options for parsing arguments in Python: argparse module sys.argv Third-party libraries.

[Audio] argparse module: The argparse module provides a powerful and flexible way to parse command-line arguments. It allows you to define arguments, specify their types, add descriptions, and handle different scenarios. The argparse module in Python provides a convenient way to parse command-line arguments and options. It simplifies the process of building command-line interfaces for your scripts. Here's an example that demonstrates the usage of the argparse module In this example: We import the argparse module. We create an instance of the ArgumentParser class, which will be used to define and parse the command-line arguments. We add arguments to the parser using the add_argument() method. In this case, we add a positional argument 'integers' that accepts one or more integers, and an optional argument '--verbose' (or -v) that indicates whether to display additional output. We parse the arguments using the parse_args() method, which returns an object containing the values of the parsed arguments. We perform the desired operation on the parsed arguments. In this example, we calculate the sum of the integers provided. We display the result based on the value of the '--verbose' argument. If it is specified, we provide more detailed output; otherwise, we simply display the result..

[Audio] sys.argv The sys.argv list contains the command-line arguments passed to the script. It allows you to access arguments directly but requires manual parsing and handling. In this example: We import the sys module. We check the number of arguments passed to the script using len(sys.argv). If there are fewer than 2 arguments (the script name itself and at least one additional argument), we display a usage message and exit the script. We access the first command-line argument, sys.argv[1], and assign it to the variable name. We display a personalized greeting using the value of name. The sys.argv list contains the script name (sys.argv[0]) as the first element, followed by any additional command-line arguments. You can access and process these arguments as needed in your script. Please note that sys.argv returns command-line arguments as strings by default. If you need to parse arguments of different types or handle more complex argument scenarios, consider using the argparse module, as shown in the previous example..

[Audio] Third-party libraries There are several third-party libraries available, such as click, docopt, and fire, which provide more advanced and user-friendly argument parsing capabilities. These libraries often offer additional features like automatic help generation, support for subcommands, and more. In this example, we're using the requests library to send an HTTP GET request to a specified URL (https://api.qacademy.tech/users). Here's a breakdown of the code: We import the requests library. We use the get() function from the requests module to send a GET request to the specified URL. We check the status code of the response using response.status_code. If it is 200 (indicating a successful request), we proceed. We access the response content using response.json(), assuming the response contains JSON data. If the response contains other content types (like HTML), you may use different methods to access the content accordingly. We print the response data (parsed JSON) to the console. If the status code is not 200, indicating an error, we print the status code. The requests library simplifies the process of making HTTP requests and provides a wide range of features, such as handling different request methods, setting headers, handling authentication, working with cookies, and more. It is widely used in Python for interacting with web APIs and fetching data from external sources. Before using third-party libraries, make sure they are installed in your Python environment. You can typically install them using the pip package manager. For example, to install requests, you can run pip install requests in your command prompt or terminal. Remember to refer to the documentation of the specific third-party library you are using for more information on its features, options, and usage..

[Audio] To create filters for reading text files in Python, you can use various techniques to process and extract specific information from the file. The ability to read and process text files is a common task in programming. Creating filters to read text files allows us to extract, manipulate, and analyze data efficiently. we will explore various techniques and strategies for creating filters to read text files in Python..

[Audio] Opening and Reading Text Files: The first step is to open the text file using the built-in open() function. We specify the file path and the mode (e.g., read mode 'r'). Once the file is opened, we can use various methods to read its contents, such as read(), readline(), or readlines(). Demonstrate examples of each method and discuss their differences and use cases..

[Audio] Filtering Lines in a Text File: Often, we need to filter specific lines based on certain conditions. We can iterate over the lines of the file and apply filtering logic to extract relevant information. Show examples of filtering lines based on patterns, keywords, or specific criteria. Discuss the use of regular expressions (re module) for advanced pattern matching..

[Audio] Processing Data in Text Files: Once the lines are filtered, we can process the extracted data further. Discuss techniques for splitting lines into fields or tokens using string manipulation or regular expressions. Explore methods to convert data types, perform calculations, or apply data transformations. Illustrate examples such as counting occurrences, calculating averages, or generating summary statistics..

[Audio] Writing Filtered Output to Files: After processing the text file, we may want to write the filtered output to a new file. Open a new file using the open() function with the appropriate write mode ('w'). Iterate over the filtered lines and write them to the new file using the write() method. Close the file when finished writing..

[Audio] Command-Line Filters: Command-line filters allow us to process text files directly from the command line. Demonstrate how to create Python scripts that accept command-line arguments and process text files accordingly. Discuss the use of the argparse module to handle command-line arguments and options. Creating filters to read text files is a powerful technique for data extraction and manipulation. Python provides a rich set of tools and modules to facilitate this process. By mastering these techniques, you can efficiently process text files, extract meaningful information, and automate data analysis tasks..

[Audio] Let's check python code with for Creating filters to read text files covering all options Assuming we have a text file named data.txt with the contents The code above will filter and print only the lines that contain the word 'Python' in them: In this example: We use the open() function to open the text file in read mode. The file is read line by line using a for loop. We apply a filtering condition using the re.search() function from the re module. It searches for the pattern 'Python' in each line. If a line contains the word 'Python', it is printed using the print() function, after stripping any leading or trailing whitespace. You can modify the filtering condition to match different patterns or criteria based on your specific requirements. Additionally, you can perform various operations on the filtered lines, such as data extraction, transformation, or calculations, depending on the desired functionality. Remember to adjust the file path and name in the open() function to match the location of your actual text file..

[Audio] Logging is a means of tracking events that happen when some software runs. Logging is important for software developing, debugging, and running. If you don't have any logging record and your program crashes, there are very few chances that you detect the cause of the problem. And if you detect the cause, it will consume a lot of time. With logging, you can leave a trail of breadcrumbs so that if something goes wrong, we can determine the cause of the problem. There are a number of situations like if you are expecting an integer, you have been given a float and you can a cloud API, the service is down for maintenance, and much more. Such problems are out of control and are hard to determine in such one is Why print statement is not Pythonic? Some developers use the concept of printing the statements to validate if the statements are executed correctly or if some error has occurred. But printing is not a good idea. It may solve your issues for simple scripts but for complex scripts, the printing approach will fail. Python has a built-in module logging which allows writing status messages to a file or any other output streams. The file can contain information on which part of the code is executed and what problems have arisen..

[Audio] The Logging Module The logging module in Python is a ready-to-use and powerful module that is designed to meet the needs of beginners as well as enterprise teams. It is used by most of the third-party Python libraries, so you can integrate your log messages with the ones from those libraries to produce a homogeneous log for your application. With the logging module imported, you can use something called a "logger" to log messages that you want to see. By default, there are 5 standard levels indicating the severity of events. Each has a corresponding method that can be used to log events at that level of severity. The defined levels, in order of increasing severity, are the following: Python Logging Levels There are five built-in levels of the log message. Debug: These are used to give Detailed information, typically of interest only when diagnosing problems. Info: These are used to confirm that things are working as expected Warning: These are used as an indication that something unexpected happened, or is indicative of some problem in the near future Error: This tells that due to a more serious problem, the software has not been able to perform some function Critical: This tells serious error, indicating that the program itself may be unable to continue running If required, developers have the option to create more levels but these are sufficient enough to handle every possible situation. Each built-in level has been assigned its numeric value. The logging module is packed with several features. It has several constants, classes, and methods. The items with all caps are constant, the capitalized items are classes and the items which start with lowercase letters are methods..

[Audio] There are several logger objects offered by the base Handler itself. Logger.info(msg): This will log a message with level INFO on this logger. Logger.warning(msg): This will log a message with a level WARNING on this logger. Logger.error(msg): This will log a message with level ERROR on this logger. Logger.critical(msg): This will log a message with level CRITICAL on this logger. Logger.log(lvl,msg): This will Log a message with integer level lvl on this logger. Logger.exception(msg): This will log a message with level ERROR on this logger. Logger.setLevel(lvl): This function sets the threshold of this logger to lvl. This means that all the messages below this level will be ignored. Logger.addFilter(filt): This adds a specific filter fit into this logger. Logger.removeFilter(filt): This removes a specific filter fit into this logger. Logger.filter(record): This method applies the logger's filter to the record provided and returns True if the record is to be processed. Else, it will return False. Logger.addHandler(hdlr): This adds a specific handler hdlr to this logger. Logger.removeHandler(hdlr) : This removes a specific handler hdlr into this logger. Logger.hasHandlers(): This checks if the logger has any handler configured or not..

[Audio] Python Logging Basics The basics of using the logging module to record the events in a file are very simple. For that, simply import the module from the library. Create and configure the logger. It can have several parameters. But importantly, pass the name of the file in which you want to record the events. Here the format of the logger can also be set. By default, the file works in append mode but we can change that to write mode if required. Also, the level of the logger can be set which acts as the threshold for tracking based on the numeric values assigned to each level. There are several attributes that can be passed as parameters. The list of all those parameters is given in Python Library. The user can choose the required attribute according to the requirement. After that, create an object and use the various methods as shown in the example..

[Audio] In this example, we perform the following steps: Import the logging module: We import the logging module, which is part of Python's standard library. Configure the logging: We use the basicConfig() function to configure the logging. In this example, we specify the log file name as 'app.log', set the logging level to DEBUG, and define the log message format using the format parameter. Log messages: We log various types of messages using different logging levels. The messages are logged to the specified log file ('app.log') according to the configured level. Output: When you run the code, it will create a log file named 'app.log' in the same directory as the script. The log file will contain the logged messages, each with a timestamp, logging level, and the provided message. Each log entry includes a timestamp, the logging level (DEBUG, INFO, WARNING, ERROR, or CRITICAL), and the provided log message. The logging module provides more advanced features for customization, such as log file rotation, log handlers, log filtering, and integration with external libraries. You can refer to the Python documentation for more information on how to utilize the logging module effectively for your specific needs..

[Audio] OS Services: We explored the os module, which provides functions for interacting with the operating system, such as file and directory operations, environment variables, and launching external commands. File and Directory Operations: We learned how to work with files and directories using the os.path module, including examples of checking file existence, getting file information, manipulating paths, and creating directories. Environment Variables: We covered the usage of environment variables in Python, including accessing and modifying them using the os.environ dictionary and the os.getenv() function. We also demonstrated how to add default values for environment variables that are not defined..

[Audio] Subprocess and External Commands: We explored the subprocess module, which allows running external commands from within a Python script. We covered different functions like os.system(), subprocess.run(), and subprocess.Popen(), along with examples and their respective outputs. Network Programming: We briefly touched upon network programming and introduced the built-in socket module, which provides the functionality to create network connections, send and receive data over TCP/IP. Scripting for System Administration: We discussed various concepts related to scripting for system administration, including running external programs, parsing arguments, creating filters to read text files, and logging. Third-party Libraries: We highlighted the importance of third-party libraries and demonstrated examples using popular libraries like requests for making HTTP requests..

[Audio] In this lab, you will explore Python system scripting and administration techniques. You will learn how to interact with the operating system, perform file and directory operations, manipulate environment variables, run external commands, work with network programming, parse command-line arguments, create filters to read text files, and utilize logging. Through hands-on exercises and practical examples, you will gain the necessary skills to automate tasks, manage system resources, and streamline administrative processes using Python. Lab Objectives: Familiarize yourself with the os module for interacting with the operating system. Learn the various file and directory operations using the os.path module. Understand how to work with environment variables and modify them using the os.environ dictionary. Execute external commands and capture their output using the subprocess module. Explore network programming concepts and utilize the socket module for creating network connections. Practice parsing command-line arguments using the argparse module. Implement filters to read and process text files for data extraction and manipulation. Gain proficiency in logging to record and analyze program output. Utilize third-party libraries to extend the functionality of your Python scripts. Apply the acquired knowledge through hands-on exercises and practical examples..

Task 1: File and Directory Operations Create a Python script to perform common file operations such as creating, deleting, and renaming files. Implement directory operations like creating directories and listing directory contents.

Task 7: Logging Implement logging in a script to record and store log messages with different logging levels. Utilize log handlers and formatters to customize the log output..