Package ganeti :: Module utils
[hide private]
[frames] | no frames]

Module utils

source code

Ganeti utility module.

This module holds functions that can be used in both daemons (all) and the command line scripts.

Classes [hide private]
  RunResult
Holds the result of running external programs.
  HostInfo
Class implementing resolver and hostname functionality
  FileLock
Utility class for file locks.
  SignalHandler
Generic signal handler class.
  FieldSet
A simple field set.
Functions [hide private]
RunResult
RunCmd(cmd, env=None, output=None, cwd='/')
Execute a (shell) command.
source code
tuple
_RunCmdPipe(cmd, env, via_shell, cwd)
Run a command and return its output.
source code
int
_RunCmdFile(cmd, env, via_shell, output, cwd)
Run a command and save its output to a file.
source code
 
RemoveFile(filename)
Remove a file ignoring some errors.
source code
 
RenameFile(old, new, mkdir=False, mkdir_mode=0750)
Renames a file.
source code
str
_FingerprintFile(filename)
Compute the fingerprint of a file.
source code
dict
FingerprintFiles(files)
Compute fingerprints for a list of files.
source code
 
CheckDict(target, template, logname=None)
Ensure a dictionary has a required set of keys.
source code
 
ForceDictType(target, key_types, allowed_values=None)
Force the values of a dict to have certain types.
source code
boolean
IsProcessAlive(pid)
Check if a given pid exists on the system.
source code
int
ReadPidFile(pidfile)
Read a pid from a file.
source code
None or str
MatchNameComponent(key, name_list)
Try to match a name against a list.
source code
dict
ListVolumeGroups()
List volume groups and their size
source code
boolean
BridgeExists(bridge)
Check whether the given bridge exists in the system
source code
list
NiceSort(name_list)
Sort a list of strings based on digit and non-digit groupings.
source code
 
TryConvert(fn, val)
Try to convert a value ignoring errors.
source code
a regular expression match object
IsValidIP(ip)
Verifies the syntax of an IPv4 address.
source code
boolean
IsValidShellParam(word)
Verifies is the given word is safe from the shell's p.o.v.
source code
str
BuildShellCmd(template, *args)
Build a safe shell command line from the given arguments.
source code
str
FormatUnit(value, units)
Formats an incoming number of MiB with the appropriate unit.
source code
 
ParseUnit(input_string)
Tries to extract number and scale from the given string.
source code
 
AddAuthorizedKey(file_name, key)
Adds an SSH public key to an authorized_keys file.
source code
 
RemoveAuthorizedKey(file_name, key)
Removes an SSH public key from an authorized_keys file.
source code
 
SetEtcHostsEntry(file_name, ip, hostname, aliases)
Sets the name of an IP address and hostname in /etc/hosts.
source code
 
AddHostToEtcHosts(hostname)
Wrapper around SetEtcHostsEntry.
source code
 
RemoveEtcHostsEntry(file_name, hostname)
Removes a hostname from /etc/hosts.
source code
 
RemoveHostFromEtcHosts(hostname)
Wrapper around RemoveEtcHostsEntry.
source code
str
CreateBackup(file_name)
Creates a backup of a file.
source code
str
ShellQuote(value)
Quotes shell argument according to POSIX.
source code
str
ShellQuoteArgs(args)
Quotes a list of shell arguments.
source code
 
TcpPing(target, port, timeout=10, live_port_needed=False, source=None)
Simple ping implementation using TCP connect(2).
source code
bool
OwnIpAddress(address)
Check if the current host has the the given IP address.
source code
list
ListVisibleFiles(path)
Returns a list of visible files in a directory.
source code
 
GetHomeDir(user, default=None)
Try to get the homedir of the given user.
source code
str
NewUUID()
Returns a random UUID.
source code
str
GenerateSecret()
Generates a random secret.
source code
 
EnsureDirs(dirs)
Make required directories, if they don't exist.
source code
str
ReadFile(file_name, size=None)
Reads a file.
source code
None or int
WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None)
(Over)write a file atomically.
source code
int
FirstFree(seq, base=0)
Returns the first non-existing integer from seq.
source code
 
all(seq, pred=bool)
Returns True if pred(x) is True for every element in the iterable
source code
 
any(seq, pred=bool)
Returns True if pred(x) is True for at least one element in the iterable
source code
list
UniqueSequence(seq)
Returns a list with unique elements.
source code
boolean
IsValidMac(mac)
Predicate to check if a MAC address is valid.
source code
boolean
TestDelay(duration)
Sleep for a fixed amount of time.
source code
 
_CloseFDNoErr(fd, retries=5)
Close a file descriptor ignoring errors.
source code
 
CloseFDs(noclose_fds=None)
Close file descriptors.
source code
int
Daemonize(logfile)
Daemonize the current process.
source code
str
DaemonPidFileName(name)
Compute a ganeti pid file absolute path
source code
 
WritePidFile(name)
Write the current process pidfile.
source code
 
RemovePidFile(name)
Remove the current process pidfile.
source code
 
KillProcess(pid, signal_=signal.SIGTERM, timeout=30, waitpid=False)
Kill a process given by its pid.
source code
str or None
FindFile(name, search_path, test=os.path.exists)
Look for a filesystem object in a given path.
source code
None or str
CheckVolumeGroupSize(vglist, vgname, minsize)
Checks if the volume group list is valid.
source code
 
SplitTime(value)
Splits time as floating point number into a tuple.
source code
 
MergeTime(timetuple)
Merges a tuple into time as a floating point number.
source code
int
GetNodeDaemonPort()
Get the node daemon port for this cluster.
source code
 
SetupLogging(logfile, debug=False, stderr_logging=False, program="", multithreaded=False)
Configures the logging module.
source code
 
TailFile(fname, lines=20)
Return the last lines from a file.
source code
str
SafeEncode(text)
Return a 'safe' version of a source string.
source code
 
CommaJoin(names)
Nicely join a set of identifiers.
source code
 
LockedMethod(fn)
Synchronized object access decorator.
source code
 
LockFile(fd)
Locks a file using POSIX locks.
source code
Variables [hide private]
  sha1 = sha.new
  _locksheld = []
  _re_shell_unquoted = re.compile('^[-.,=:/_+@A-Za-z0-9]+$')
  debug_locks = False
  no_fork = False

Imports: os, time, subprocess, re, socket, tempfile, shutil, errno, pwd, itertools, select, fcntl, resource, logging, signal, StringIO, sha, errors, constants


Function Details [hide private]

RunCmd(cmd, env=None, output=None, cwd='/')

source code 

Execute a (shell) command.

The command should not read from its standard input, as it will be closed.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - Additional environment
  • output (str) - if desired, the output of the command can be saved in a file instead of the RunResult instance; this parameter denotes the file name (if not None)
  • cwd (string) - if specified, will be used as the working directory for the command; the default will be /
Returns: RunResult
RunResult instance
Raises:

_RunCmdPipe(cmd, env, via_shell, cwd)

source code 

Run a command and return its output.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - The environment to use
  • via_shell (bool) - if we should run via the shell
  • cwd (string) - the working directory for the program
Returns: tuple
(out, err, status)

_RunCmdFile(cmd, env, via_shell, output, cwd)

source code 

Run a command and save its output to a file.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - The environment to use
  • via_shell (bool) - if we should run via the shell
  • output (str) - the filename in which to save the output
  • cwd (string) - the working directory for the program
Returns: int
the exit status

RemoveFile(filename)

source code 

Remove a file ignoring some errors.

Remove a file, ignoring non-existing ones or directories. Other errors are passed.

Parameters:
  • filename (str) - the file to be removed

RenameFile(old, new, mkdir=False, mkdir_mode=0750)

source code 

Renames a file.

Parameters:
  • old (string) - Original path
  • new (string) - New path
  • mkdir (bool) - Whether to create target directory if it doesn't exist
  • mkdir_mode (int) - Mode for newly created directories

_FingerprintFile(filename)

source code 

Compute the fingerprint of a file.

If the file does not exist, a None will be returned instead.

Parameters:
  • filename (str) - the filename to checksum
Returns: str
the hex digest of the sha checksum of the contents of the file

FingerprintFiles(files)

source code 

Compute fingerprints for a list of files.

Parameters:
  • files (list) - the list of filename to fingerprint
Returns: dict
a dictionary filename: fingerprint, holding only existing files

CheckDict(target, template, logname=None)

source code 

Ensure a dictionary has a required set of keys.

For the given dictionaries target and template, ensure target has all the keys from template. Missing keys are added with values from template.

Parameters:
  • target (dict) - the dictionary to update
  • template (dict) - the dictionary holding the default values
  • logname (str or None) - if not None, causes the missing keys to be logged with this name

ForceDictType(target, key_types, allowed_values=None)

source code 

Force the values of a dict to have certain types.

Parameters:
  • target (dict) - the dict to update
  • key_types (dict) - dict mapping target dict keys to types in constants.ENFORCEABLE_TYPES
  • allowed_values (list) - list of specially allowed values

IsProcessAlive(pid)

source code 

Check if a given pid exists on the system.

Parameters:
  • pid (int) - the process ID to check
Returns: boolean
True if the process exists

Note: zombie status is not handled, so zombie processes will be returned as alive

ReadPidFile(pidfile)

source code 

Read a pid from a file.

Parameters:
  • pidfile (string) - path to the file containing the pid
Returns: int
The process id, if the file exists and contains a valid PID, otherwise 0

MatchNameComponent(key, name_list)

source code 

Try to match a name against a list.

This function will try to match a name like test1 against a list like ['test1.example.com', 'test2.example.com', ...]. Against this list, 'test1' as well as 'test1.example' will match, but not 'test1.ex'. A multiple match will be considered as no match at all (e.g. 'test1' against ['test1.example.com', 'test1.example.org']).

Parameters:
  • key (str) - the name to be searched
  • name_list (list) - the list of strings against which to search the key
Returns: None or str
None if there is no match or if there are multiple matches, otherwise the element from the list which matches

ListVolumeGroups()

source code 

List volume groups and their size

Returns: dict
Dictionary with keys volume name and values the size of the volume

BridgeExists(bridge)

source code 

Check whether the given bridge exists in the system

Parameters:
  • bridge (str) - the bridge name to check
Returns: boolean
True if it does

NiceSort(name_list)

source code 

Sort a list of strings based on digit and non-digit groupings.

Given a list of names ['a1', 'a10', 'a11', 'a2'] this function will sort the list in the logical order ['a1', 'a2', 'a10', 'a11'].

The sort algorithm breaks each name in groups of either only-digits or no-digits. Only the first eight such groups are considered, and after that we just use what's left of the string.

Parameters:
  • name_list (list) - the names to be sorted
Returns: list
a copy of the name list sorted with our algorithm

TryConvert(fn, val)

source code 

Try to convert a value ignoring errors.

This function tries to apply function fn to val. If no ValueError or TypeError exceptions are raised, it will return the result, else it will return the original value. Any other exceptions are propagated to the caller.

Parameters:
  • fn (callable) - function to apply to the value
  • val - the value to be converted
Returns:
The converted value if the conversion was successful, otherwise the original value.

IsValidIP(ip)

source code 

Verifies the syntax of an IPv4 address.

This function checks if the IPv4 address passes is valid or not based on syntax (not IP range, class calculations, etc.).

Parameters:
  • ip (str) - the address to be checked
Returns: a regular expression match object
a regular expression match object, or None if the address is not valid

IsValidShellParam(word)

source code 

Verifies is the given word is safe from the shell's p.o.v.

This means that we can pass this to a command via the shell and be sure that it doesn't alter the command line and is passed as such to the actual command.

Note that we are overly restrictive here, in order to be on the safe side.

Parameters:
  • word (str) - the word to check
Returns: boolean
True if the word is 'safe'

BuildShellCmd(template, *args)

source code 

Build a safe shell command line from the given arguments.

This function will check all arguments in the args list so that they are valid shell parameters (i.e. they don't contain shell metacharacters). If everything is ok, it will return the result of template % args.

Parameters:
  • template (str) - the string holding the template for the string formatting
Returns: str
the expanded command line

FormatUnit(value, units)

source code 

Formats an incoming number of MiB with the appropriate unit.

Parameters:
  • value (int) - integer representing the value in MiB (1048576)
  • units (char) - the type of formatting we should do:
    • 'h' for automatic scaling
    • 'm' for MiBs
    • 'g' for GiBs
    • 't' for TiBs
Returns: str
the formatted value (with suffix)

ParseUnit(input_string)

source code 

Tries to extract number and scale from the given string.

Input must be in the format NUMBER+ [DOT NUMBER+] SPACE* [UNIT]. If no unit is specified, it defaults to MiB. Return value is always an int in MiB.

AddAuthorizedKey(file_name, key)

source code 

Adds an SSH public key to an authorized_keys file.

Parameters:
  • file_name (str) - path to authorized_keys file
  • key (str) - string containing key

RemoveAuthorizedKey(file_name, key)

source code 

Removes an SSH public key from an authorized_keys file.

Parameters:
  • file_name (str) - path to authorized_keys file
  • key (str) - string containing key

SetEtcHostsEntry(file_name, ip, hostname, aliases)

source code 

Sets the name of an IP address and hostname in /etc/hosts.

Parameters:
  • file_name (str) - path to the file to modify (usually /etc/hosts)
  • ip (str) - the IP address
  • hostname (str) - the hostname to be added
  • aliases (list) - the list of aliases to add for the hostname

AddHostToEtcHosts(hostname)

source code 

Wrapper around SetEtcHostsEntry.

Parameters:

RemoveEtcHostsEntry(file_name, hostname)

source code 

Removes a hostname from /etc/hosts.

IP addresses without names are removed from the file.

Parameters:
  • file_name (str) - path to the file to modify (usually /etc/hosts)
  • hostname (str) - the hostname to be removed

RemoveHostFromEtcHosts(hostname)

source code 

Wrapper around RemoveEtcHostsEntry.

Parameters:
  • hostname (str) - hostname that will be resolved and its full and shot name will be removed from constants.ETC_HOSTS

CreateBackup(file_name)

source code 

Creates a backup of a file.

Parameters:
  • file_name (str) - file to be backed up
Returns: str
the path to the newly created backup
Raises:

ShellQuote(value)

source code 

Quotes shell argument according to POSIX.

Parameters:
  • value (str) - the argument to be quoted
Returns: str
the quoted value

ShellQuoteArgs(args)

source code 

Quotes a list of shell arguments.

Parameters:
  • args (list) - list of arguments to be quoted
Returns: str
the quoted arguments concatenated with spaces

TcpPing(target, port, timeout=10, live_port_needed=False, source=None)

source code 

Simple ping implementation using TCP connect(2).

Check if the given IP is reachable by doing attempting a TCP connect to it.

Parameters:
  • target (str) - the IP or hostname to ping
  • port (int) - the port to connect to
  • timeout (int) - the timeout on the connection attempt
  • live_port_needed (boolean) - whether a closed port will cause the function to return failure, as if there was a timeout
  • source (str or None) - if specified, will cause the connect to be made from this specific source address; failures to bind other than EADDRNOTAVAIL will be ignored

OwnIpAddress(address)

source code 

Check if the current host has the the given IP address.

Currently this is done by TCP-pinging the address from the loopback address.

Parameters:
  • address (string) - the address to check
Returns: bool
True if we own the address

ListVisibleFiles(path)

source code 

Returns a list of visible files in a directory.

Parameters:
  • path (str) - the directory to enumerate
Returns: list
the list of all files not starting with a dot

GetHomeDir(user, default=None)

source code 

Try to get the homedir of the given user.

The user can be passed either as a string (denoting the name) or as an integer (denoting the user id). If the user is not found, the 'default' argument is returned, which defaults to None.

NewUUID()

source code 

Returns a random UUID.

Returns: str

Note: This is a Linux-specific method as it uses the /proc filesystem.

GenerateSecret()

source code 

Generates a random secret.

This will generate a pseudo-random secret, and return its sha digest (so that it can be used where an ASCII string is needed).

Returns: str
a sha1 hexdigest of a block of 64 random bytes

EnsureDirs(dirs)

source code 

Make required directories, if they don't exist.

Parameters:
  • dirs (list of (string, integer)) - list of tuples (dir_name, dir_mode)

ReadFile(file_name, size=None)

source code 

Reads a file.

Parameters:
  • size (None or int) - Read at most size bytes
Returns: str
the (possibly partial) content of the file

WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None)

source code 

(Over)write a file atomically.

The file_name and either fn (a function taking one argument, the file descriptor, and which should write the data to it) or data (the contents of the file) must be passed. The other arguments are optional and allow setting the file mode, owner and group, and the mtime/atime of the file.

If the function doesn't raise an exception, it has succeeded and the target file has the new contents. If the function has raised an exception, an existing target file should be unmodified and the temporary file should be removed.

Parameters:
  • file_name (str) - the target filename
  • fn (callable) - content writing function, called with file descriptor as parameter
  • data (str) - contents of the file
  • mode (int) - file mode
  • uid (int) - the owner of the file
  • gid (int) - the group of the file
  • atime (int) - a custom access time to be set on the file
  • mtime (int) - a custom modification time to be set on the file
  • close (boolean) - whether to close file after writing it
  • prewrite (callable) - function to be called before writing content
  • postwrite (callable) - function to be called after writing content
Returns: None or int
None if the 'close' parameter evaluates to True, otherwise the file descriptor
Raises:

FirstFree(seq, base=0)

source code 

Returns the first non-existing integer from seq.

The seq argument should be a sorted list of positive integers. The first time the index of an element is smaller than the element value, the index will be returned.

The base argument is used to start at a different offset, i.e. [3, 4, 6] with offset=3 will return 5.

Example: [0, 1, 3] will return 2.

Parameters:
  • seq (sequence) - the sequence to be analyzed.
  • base (int) - use this value as the base index of the sequence
Returns: int
the first non-used index in the sequence

UniqueSequence(seq)

source code 

Returns a list with unique elements.

Element order is preserved.

Parameters:
  • seq (sequence) - the sequence with the source elements
Returns: list
list of unique elements from seq

IsValidMac(mac)

source code 

Predicate to check if a MAC address is valid.

Checks whether the supplied MAC address is formally correct, only accepts colon separated format.

Parameters:
  • mac (str) - the MAC to be validated
Returns: boolean
True is the MAC seems valid

TestDelay(duration)

source code 

Sleep for a fixed amount of time.

Parameters:
  • duration (float) - the sleep duration
Returns: boolean
False for negative value, True otherwise

_CloseFDNoErr(fd, retries=5)

source code 

Close a file descriptor ignoring errors.

Parameters:
  • fd (int) - the file descriptor
  • retries (int) - how many retries to make, in case we get any other error than EBADF

CloseFDs(noclose_fds=None)

source code 

Close file descriptors.

This closes all file descriptors above 2 (i.e. except stdin/out/err).

Parameters:
  • noclose_fds (list or None) - if given, it denotes a list of file descriptor that should not be closed

Daemonize(logfile)

source code 

Daemonize the current process.

This detaches the current process from the controlling terminal and runs it in the background as a daemon.

Parameters:
  • logfile (str) - the logfile to which we should redirect stdout/stderr
Returns: int
the value zero

DaemonPidFileName(name)

source code 

Compute a ganeti pid file absolute path

Parameters:
  • name (str) - the daemon name
Returns: str
the full path to the pidfile corresponding to the given daemon name

WritePidFile(name)

source code 

Write the current process pidfile.

The file will be written to constants.RUN_GANETI_DIR/name.pid

Parameters:
  • name (str) - the daemon name to use
Raises:

RemovePidFile(name)

source code 

Remove the current process pidfile.

Any errors are ignored.

Parameters:
  • name (str) - the daemon name used to derive the pidfile name

KillProcess(pid, signal_=signal.SIGTERM, timeout=30, waitpid=False)

source code 

Kill a process given by its pid.

Parameters:
  • pid (int) - The PID to terminate.
  • signal_ (int) - The signal to send, by default SIGTERM
  • timeout (int) - The timeout after which, if the process is still alive, a SIGKILL will be sent. If not positive, no such checking will be done
  • waitpid (boolean) - If true, we should waitpid on this process after sending signals, since it's our own child and otherwise it would remain as zombie

FindFile(name, search_path, test=os.path.exists)

source code 

Look for a filesystem object in a given path.

This is an abstract method to search for filesystem object (files, dirs) under a given search path.

Parameters:
  • name (str) - the name to look for
  • search_path (str) - location to start at
  • test (callable) - a function taking one argument that should return True if the a given object is valid; the default value is os.path.exists, causing only existing files to be returned
Returns: str or None
full path to the object if found, None otherwise

CheckVolumeGroupSize(vglist, vgname, minsize)

source code 

Checks if the volume group list is valid.

The function will check if a given volume group is in the list of volume groups and has a minimum size.

Parameters:
  • vglist (dict) - dictionary of volume group names and their size
  • vgname (str) - the volume group we should check
  • minsize (int) - the minimum size we accept
Returns: None or str
None for success, otherwise the error message

SplitTime(value)

source code 

Splits time as floating point number into a tuple.

Parameters:
  • value (int or float) - Time in seconds
Returns:
Tuple containing (seconds, microseconds)

MergeTime(timetuple)

source code 

Merges a tuple into time as a floating point number.

Parameters:
  • timetuple (tuple) - Time as tuple, (seconds, microseconds)
Returns:
Time as a floating point number expressed in seconds

GetNodeDaemonPort()

source code 

Get the node daemon port for this cluster.

Note that this routine does not read a ganeti-specific file, but instead uses socket.getservbyname to allow pre-customization of this parameter outside of Ganeti.

Returns: int

SetupLogging(logfile, debug=False, stderr_logging=False, program="", multithreaded=False)

source code 

Configures the logging module.

Parameters:
  • logfile (str) - the filename to which we should log
  • debug (boolean) - whether to enable debug messages too or only those at INFO and above level
  • stderr_logging (boolean) - whether we should also log to the standard error
  • program (str) - the name under which we should log messages
  • multithreaded (boolean) - if True, will add the thread name to the log file
Raises:
  • EnvironmentError - if we can't open the log file and stderr logging is disabled

TailFile(fname, lines=20)

source code 

Return the last lines from a file.

Parameters:
  • fname - the file name
  • lines (int) - the (maximum) number of lines to return

Note: this function will only read and parse the last 4KB of the file; if the lines are very long, it could be that less than the requested number of lines are returned

SafeEncode(text)

source code 

Return a 'safe' version of a source string.

This function mangles the input string and returns a version that should be safe to display/encode as ASCII. To this end, we first convert it to ASCII using the 'backslashreplace' encoding which should get rid of any non-ASCII chars, and then we process it through a loop copied from the string repr sources in the python; we don't use string_escape anymore since that escape single quotes and backslashes too, and that is too much; and that escaping is not stable, i.e. string_escape(string_escape(x)) != string_escape(x).

Parameters:
  • text (str or unicode) - input data
Returns: str
a safe version of text

CommaJoin(names)

source code 

Nicely join a set of identifiers.

Parameters:
  • names - set, list or tuple
Returns:
a string with the formatted results

LockedMethod(fn)

source code 

Synchronized object access decorator.

This decorator is intended to protect access to an object using the object's own lock which is hardcoded to '_lock'.

LockFile(fd)

source code 

Locks a file using POSIX locks.

Parameters:
  • fd (int) - the file descriptor we need to lock