Package ganeti :: Module utils
[hide private]
[frames] | no frames]

Module utils

source code

Ganeti utility module.

This module holds functions that can be used in both daemons (all) and the command line scripts.

Classes [hide private]
  RunResult
Holds the result of running external programs.
  ShellWriter
Helper class to write scripts with indentation.
  FdConditionWaiterHelper
Retry helper for WaitForFdCondition.
  LogFileHandler
Log handler that doesn't fallback to stderr.
  RetryTimeout
Retry loop timed out.
  RetryAgain
Retry again.
  _RetryDelayCalculator
Calculator for increasing delays.
  FileLock
Utility class for file locks.
  LineSplitter
Splits data chunks into lines separated by newline.
  SignalWakeupFd
  SignalHandler
Generic signal handler class.
  FieldSet
A simple field set.
Functions [hide private]
 
_BuildCmdEnvironment(env, reset)
Builds the environment for an external program.
source code
RunResult
RunCmd(cmd, env=None, output=None, cwd="/", reset_env=False, interactive=False)
Execute a (shell) command.
source code
 
SetupDaemonEnv(cwd="/", umask=077)
Setup a daemon's environment.
source code
 
SetupDaemonFDs(output_file, output_fd)
Setups up a daemon's file descriptors.
source code
int
StartDaemon(cmd, env=None, cwd="/", output=None, output_fd=None, pidfile=None)
Start a daemon process after forking twice.
source code
 
_StartDaemonChild(errpipe_read, errpipe_write, pidpipe_read, pidpipe_write, args, env, cwd, output, fd_output, pidfile)
Child process for starting daemon.
source code
 
WriteErrorToFD(fd, err)
Possibly write an error message to a fd.
source code
tuple
_RunCmdPipe(cmd, env, via_shell, cwd, interactive)
Run a command and return its output.
source code
int
_RunCmdFile(cmd, env, via_shell, output, cwd)
Run a command and save its output to a file.
source code
 
SetCloseOnExecFlag(fd, enable)
Sets or unsets the close-on-exec flag on a file descriptor.
source code
 
SetNonblockFlag(fd, enable)
Sets or unsets the O_NONBLOCK flag on on a file descriptor.
source code
 
RetryOnSignal(fn, *args, **kwargs)
Calls a function again if it failed due to EINTR.
source code
list of tuples
RunParts(dir_name, env=None, reset_env=False)
Run Scripts or programs in a directory
source code
 
RemoveFile(filename)
Remove a file ignoring some errors.
source code
 
RemoveDir(dirname)
Remove an empty directory.
source code
 
RenameFile(old, new, mkdir=False, mkdir_mode=0750)
Renames a file.
source code
 
Makedirs(path, mode=0750)
Super-mkdir; create a leaf directory and all intermediate ones.
source code
 
ResetTempfileModule()
Resets the random name generator of the tempfile module.
source code
str
_FingerprintFile(filename)
Compute the fingerprint of a file.
source code
dict
FingerprintFiles(files)
Compute fingerprints for a list of files.
source code
 
ForceDictType(target, key_types, allowed_values=None)
Force the values of a dict to have certain types.
source code
string
_GetProcStatusPath(pid)
Returns the path for a PID's proc status file.
source code
boolean
IsProcessAlive(pid)
Check if a given pid exists on the system.
source code
set
_ParseSigsetT(sigset)
Parse a rendered sigset_t value.
source code
string
_GetProcStatusField(pstatus, field)
Retrieves a field from the contents of a proc status file.
source code
bool
IsProcessHandlingSignal(pid, signum, status_path=None)
Checks whether a process is handling a signal.
source code
int
ReadPidFile(pidfile)
Read a pid from a file.
source code
 
ReadLockedPidFile(path)
Reads a locked PID file.
source code
None or str
MatchNameComponent(key, name_list, case_sensitive=True)
Try to match a name against a list.
source code
 
ValidateServiceName(name)
Validate the given service name.
source code
dict
ListVolumeGroups()
List volume groups and their size
source code
boolean
BridgeExists(bridge)
Check whether the given bridge exists in the system
source code
list
NiceSort(name_list)
Sort a list of strings based on digit and non-digit groupings.
source code
 
TryConvert(fn, val)
Try to convert a value ignoring errors.
source code
boolean
IsValidShellParam(word)
Verifies is the given word is safe from the shell's p.o.v.
source code
str
BuildShellCmd(template, *args)
Build a safe shell command line from the given arguments.
source code
str
FormatUnit(value, units)
Formats an incoming number of MiB with the appropriate unit.
source code
 
ParseUnit(input_string)
Tries to extract number and scale from the given string.
source code
list of int
ParseCpuMask(cpu_mask)
Parse a CPU mask definition and return the list of CPU IDs.
source code
 
AddAuthorizedKey(file_obj, key)
Adds an SSH public key to an authorized_keys file.
source code
 
RemoveAuthorizedKey(file_name, key)
Removes an SSH public key from an authorized_keys file.
source code
 
SetEtcHostsEntry(file_name, ip, hostname, aliases)
Sets the name of an IP address and hostname in /etc/hosts.
source code
 
AddHostToEtcHosts(hostname, ip)
Wrapper around SetEtcHostsEntry.
source code
 
RemoveEtcHostsEntry(file_name, hostname)
Removes a hostname from /etc/hosts.
source code
 
RemoveHostFromEtcHosts(hostname)
Wrapper around RemoveEtcHostsEntry.
source code
 
TimestampForFilename()
Returns the current time formatted for filenames.
source code
str
CreateBackup(file_name)
Creates a backup of a file.
source code
str
ShellQuote(value)
Quotes shell argument according to POSIX.
source code
str
ShellQuoteArgs(args)
Quotes a list of shell arguments.
source code
list
ListVisibleFiles(path)
Returns a list of visible files in a directory.
source code
 
GetHomeDir(user, default=None)
Try to get the homedir of the given user.
source code
str
NewUUID()
Returns a random UUID.
source code
str
GenerateSecret(numbytes=20)
Generates a random secret.
source code
 
EnsureDirs(dirs)
Make required directories, if they don't exist.
source code
str
ReadFile(file_name, size=-1)
Reads a file.
source code
None or int
WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None)
(Over)write a file atomically.
source code
 
GetFileID(path=None, fd=None)
Returns the file 'id', i.e.
source code
boolean
VerifyFileID(fi_disk, fi_ours)
Verifies that two file IDs are matching.
source code
 
SafeWriteFile(file_name, file_id, **kwargs)
Wraper over WriteFile that locks the target file.
source code
 
ReadOneLineFile(file_name, strict=False)
Return the first non-empty line from a file.
source code
int
FirstFree(seq, base=0)
Returns the first non-existing integer from seq.
source code
int or None
SingleWaitForFdCondition(fdobj, event, timeout)
Waits for a condition to occur on the socket.
source code
int or None
WaitForFdCondition(fdobj, event, timeout)
Waits for a condition to occur on the socket.
source code
list
UniqueSequence(seq)
Returns a list with unique elements.
source code
str
NormalizeAndValidateMac(mac)
Normalizes and check if a MAC address is valid.
source code
boolean
TestDelay(duration)
Sleep for a fixed amount of time.
source code
 
_CloseFDNoErr(fd, retries=5)
Close a file descriptor ignoring errors.
source code
 
CloseFDs(noclose_fds=None)
Close file descriptors.
source code
 
Mlockall(_ctypes=ctypes)
Lock current process' virtual address space into RAM.
source code
int
Daemonize(logfile)
Daemonize the current process.
source code
str
DaemonPidFileName(name)
Compute a ganeti pid file absolute path
source code
 
EnsureDaemon(name)
Check for and start daemon if not alive.
source code
 
StopDaemon(name)
Stop daemon
source code
int
WritePidFile(pidfile)
Write the current process pidfile.
source code
 
RemovePidFile(name)
Remove the current process pidfile.
source code
 
KillProcess(pid, signal_=signal.SIGTERM, timeout=30, waitpid=False)
Kill a process given by its pid.
source code
str or None
FindFile(name, search_path, test=os.path.exists)
Look for a filesystem object in a given path.
source code
None or str
CheckVolumeGroupSize(vglist, vgname, minsize)
Checks if the volume group list is valid.
source code
 
SplitTime(value)
Splits time as floating point number into a tuple.
source code
 
MergeTime(timetuple)
Merges a tuple into time as a floating point number.
source code
 
SetupLogging(logfile, debug=0, stderr_logging=False, program="", multithreaded=False, syslog=constants.SYSLOG_USAGE, console_logging=False)
Configures the logging module.
source code
 
IsNormAbsPath(path)
Check whether a path is absolute and also normalized
source code
 
PathJoin(*args)
Safe-join a list of path components.
source code
 
TailFile(fname, lines=20)
Return the last lines from a file.
source code
 
FormatTimestampWithTZ(secs)
Formats a Unix timestamp with the local timezone.
source code
 
_ParseAsn1Generalizedtime(value)
Parses an ASN1 GENERALIZEDTIME timestamp as used by pyOpenSSL.
source code
 
GetX509CertValidity(cert)
Returns the validity period of the certificate.
source code
 
_VerifyCertificateInner(expired, not_before, not_after, now, warn_days, error_days)
Verifies certificate validity.
source code
 
VerifyX509Certificate(cert, warn_days, error_days)
Verifies a certificate for LUVerifyCluster.
source code
string
SignX509Certificate(cert, key, salt)
Sign a X509 certificate.
source code
 
_ExtractX509CertificateSignature(cert_pem)
Helper function to extract signature from X509 certificate.
source code
tuple; (OpenSSL.crypto.X509, string)
LoadSignedX509Certificate(cert_pem, key)
Verifies a signed X509 certificate.
source code
 
Sha1Hmac(key, text, salt=None)
Calculates the HMAC-SHA1 digest of a text.
source code
bool
VerifySha1Hmac(key, text, digest, salt=None)
Verifies the HMAC-SHA1 digest of a text.
source code
str
SafeEncode(text)
Return a 'safe' version of a source string.
source code
string
UnescapeAndSplit(text, sep=",")
Split and unescape a string based on a given separator.
source code
 
CommaJoin(names)
Nicely join a set of identifiers.
source code
tuple; (value in dictionary, matched groups as list)
FindMatch(data, name)
Tries to find an item in a dictionary matching a name.
source code
int
BytesToMebibyte(value)
Converts bytes to mebibytes.
source code
int
CalculateDirectorySize(path)
Calculates the size of a directory recursively.
source code
list of tuples
GetMounts(filename=constants.PROC_MOUNTS)
Returns the list of mounted filesystems.
source code
int
GetFilesystemStats(path)
Returns the total and free space on a filesystem.
source code
bool
RunInSeparateProcess(fn, *args)
Runs a function in a separate process.
source code
bool
IgnoreProcessNotFound(fn, *args, **kwargs)
Ignores ESRCH when calling a process-related function.
source code
 
IgnoreSignals(fn, *args, **kwargs)
Tries to call a function ignoring failures due to EINTR.
source code
 
LockFile(fd)
Locks a file using POSIX locks.
source code
 
FormatTime(val)
Formats a time value.
source code
string
FormatSeconds(secs)
Formats seconds for easier reading.
source code
 
ReadWatcherPauseFile(filename, now=None, remove_after=3600)
Reads the watcher pause file.
source code
 
Retry(fn, delay, timeout, args=None, wait_fn=time.sleep, _time_fn=time.time)
Call a function repeatedly until it succeeds.
source code
 
GetClosedTempfile(*args, **kwargs)
Creates a temporary file and returns its path.
source code
 
GenerateSelfSignedX509Cert(common_name, validity)
Generates a self-signed X509 certificate.
source code
 
GenerateSelfSignedSslCert(filename, common_name=constants.X509_CERT_CN, validity=constants.X509_CERT_DEFAULT_VALIDITY)
Legacy function to generate self-signed X509 certificate.
source code
 
SignalHandled(signums)
Signal Handled decoration.
source code
Variables [hide private]
  ctypes = None
  _locksheld = []
  _re_shell_unquoted = re.compile('^[-.,=:/_+@A-Za-z0-9]+$')
  debug_locks = False
  no_fork = False
when set to True, RunCmd is disabled
  _RANDOM_UUID_FILE = "/proc/sys/kernel/random/uuid"
  HEX_CHAR_RE = r"[a-zA-Z0-9]"
  VALID_X509_SIGNATURE_SALT = re.compile("^%s+$" % HEX_CHAR_RE, ...
  X509_SIGNATURE = re.compile(r"^%s:\s*(?P<salt>%s+)/(?P<sign>%s...
  _VALID_SERVICE_NAME_RE = re.compile("^[-_.a-zA-Z0-9]{1,128}$")
  UUID_RE = re.compile('^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-' '...
  _MCL_CURRENT = 1
  _MCL_FUTURE = 2
  _MAC_CHECK = re.compile("^([0-9a-f]{2}:){5}[0-9a-f]{2}$", re.I)
MAC checker regexp
  RETRY_REMAINING_TIME = object()
Special delay to specify whole remaining timeout

Imports: os, sys, time, subprocess, re, socket, tempfile, shutil, errno, pwd, itertools, select, fcntl, resource, logging, signal, OpenSSL, datetime, calendar, hmac, collections, StringIO, errors, constants, compat


Function Details [hide private]

RunCmd(cmd, env=None, output=None, cwd="/", reset_env=False, interactive=False)

source code 

Execute a (shell) command.

The command should not read from its standard input, as it will be closed.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - Additional environment variables
  • output (str) - if desired, the output of the command can be saved in a file instead of the RunResult instance; this parameter denotes the file name (if not None)
  • cwd (string) - if specified, will be used as the working directory for the command; the default will be /
  • reset_env (boolean) - whether to reset or keep the default os environment
  • interactive (boolean) - weather we pipe stdin, stdout and stderr (default behaviour) or run the command interactive
Returns: RunResult
RunResult instance
Raises:

SetupDaemonEnv(cwd="/", umask=077)

source code 

Setup a daemon's environment.

This should be called between the first and second fork, due to setsid usage.

Parameters:
  • cwd - the directory to which to chdir
  • umask - the umask to setup

SetupDaemonFDs(output_file, output_fd)

source code 

Setups up a daemon's file descriptors.

Parameters:
  • output_file - if not None, the file to which to redirect stdout/stderr
  • output_fd - if not None, the file descriptor for stdout/stderr

StartDaemon(cmd, env=None, cwd="/", output=None, output_fd=None, pidfile=None)

source code 

Start a daemon process after forking twice.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - Additional environment variables
  • cwd (string) - Working directory for the program
  • output (string) - Path to file in which to save the output
  • output_fd (int) - File descriptor for output
  • pidfile (string) - Process ID file
Returns: int
Daemon process ID
Raises:

WriteErrorToFD(fd, err)

source code 

Possibly write an error message to a fd.

Parameters:
  • fd (None or int (file descriptor)) - if not None, the error will be written to this fd
  • err - string, the error message

_RunCmdPipe(cmd, env, via_shell, cwd, interactive)

source code 

Run a command and return its output.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - The environment to use
  • via_shell (bool) - if we should run via the shell
  • cwd (string) - the working directory for the program
  • interactive (boolean) - Run command interactive (without piping)
Returns: tuple
(out, err, status)

_RunCmdFile(cmd, env, via_shell, output, cwd)

source code 

Run a command and save its output to a file.

Parameters:
  • cmd (string or list) - Command to run
  • env (dict) - The environment to use
  • via_shell (bool) - if we should run via the shell
  • output (str) - the filename in which to save the output
  • cwd (string) - the working directory for the program
Returns: int
the exit status

SetCloseOnExecFlag(fd, enable)

source code 

Sets or unsets the close-on-exec flag on a file descriptor.

Parameters:
  • fd (int) - File descriptor
  • enable (bool) - Whether to set or unset it.

SetNonblockFlag(fd, enable)

source code 

Sets or unsets the O_NONBLOCK flag on on a file descriptor.

Parameters:
  • fd (int) - File descriptor
  • enable (bool) - Whether to set or unset it

RunParts(dir_name, env=None, reset_env=False)

source code 

Run Scripts or programs in a directory

Parameters:
  • dir_name (string) - absolute path to a directory
  • env (dict) - The environment to use
  • reset_env (boolean) - whether to reset or keep the default os environment
Returns: list of tuples
list of (name, (one of RUNDIR_STATUS), RunResult)

RemoveFile(filename)

source code 

Remove a file ignoring some errors.

Remove a file, ignoring non-existing ones or directories. Other errors are passed.

Parameters:
  • filename (str) - the file to be removed

RemoveDir(dirname)

source code 

Remove an empty directory.

Remove a directory, ignoring non-existing ones. Other errors are passed. This includes the case, where the directory is not empty, so it can't be removed.

Parameters:
  • dirname (str) - the empty directory to be removed

RenameFile(old, new, mkdir=False, mkdir_mode=0750)

source code 

Renames a file.

Parameters:
  • old (string) - Original path
  • new (string) - New path
  • mkdir (bool) - Whether to create target directory if it doesn't exist
  • mkdir_mode (int) - Mode for newly created directories

Makedirs(path, mode=0750)

source code 

Super-mkdir; create a leaf directory and all intermediate ones.

This is a wrapper around os.makedirs adding error handling not implemented before Python 2.5.

ResetTempfileModule()

source code 

Resets the random name generator of the tempfile module.

This function should be called after os.fork in the child process to ensure it creates a newly seeded random generator. Otherwise it would generate the same random parts as the parent process. If several processes race for the creation of a temporary file, this could lead to one not getting a temporary name.

_FingerprintFile(filename)

source code 

Compute the fingerprint of a file.

If the file does not exist, a None will be returned instead.

Parameters:
  • filename (str) - the filename to checksum
Returns: str
the hex digest of the sha checksum of the contents of the file

FingerprintFiles(files)

source code 

Compute fingerprints for a list of files.

Parameters:
  • files (list) - the list of filename to fingerprint
Returns: dict
a dictionary filename: fingerprint, holding only existing files

ForceDictType(target, key_types, allowed_values=None)

source code 

Force the values of a dict to have certain types.

Parameters:
  • target (dict) - the dict to update
  • key_types (dict) - dict mapping target dict keys to types in constants.ENFORCEABLE_TYPES
  • allowed_values (list) - list of specially allowed values

_GetProcStatusPath(pid)

source code 

Returns the path for a PID's proc status file.

Parameters:
  • pid (int) - Process ID
Returns: string

IsProcessAlive(pid)

source code 

Check if a given pid exists on the system.

Parameters:
  • pid (int) - the process ID to check
Returns: boolean
True if the process exists

Note: zombie status is not handled, so zombie processes will be returned as alive

_ParseSigsetT(sigset)

source code 

Parse a rendered sigset_t value.

This is the opposite of the Linux kernel's fs/proc/array.c:render_sigset_t function.

Parameters:
  • sigset (string) - Rendered signal set from /proc/$pid/status
Returns: set
Set of all enabled signal numbers

_GetProcStatusField(pstatus, field)

source code 

Retrieves a field from the contents of a proc status file.

Parameters:
  • pstatus (string) - Contents of /proc/$pid/status
  • field (string) - Name of field whose value should be returned
Returns: string

IsProcessHandlingSignal(pid, signum, status_path=None)

source code 

Checks whether a process is handling a signal.

Parameters:
  • pid (int) - Process ID
  • signum (int) - Signal number
Returns: bool

ReadPidFile(pidfile)

source code 

Read a pid from a file.

Parameters:
  • pidfile (string) - path to the file containing the pid
Returns: int
The process id, if the file exists and contains a valid PID, otherwise 0

ReadLockedPidFile(path)

source code 

Reads a locked PID file.

This can be used together with StartDaemon.

Parameters:
  • path (string) - Path to PID file
Returns:
PID as integer or, if file was unlocked or couldn't be opened, None

MatchNameComponent(key, name_list, case_sensitive=True)

source code 

Try to match a name against a list.

This function will try to match a name like test1 against a list like ['test1.example.com', 'test2.example.com', ...]. Against this list, 'test1' as well as 'test1.example' will match, but not 'test1.ex'. A multiple match will be considered as no match at all (e.g. 'test1' against ['test1.example.com', 'test1.example.org']), except when the key fully matches an entry (e.g. 'test1' against ['test1', 'test1.example.com']).

Parameters:
  • key (str) - the name to be searched
  • name_list (list) - the list of strings against which to search the key
  • case_sensitive (boolean) - whether to provide a case-sensitive match
Returns: None or str
None if there is no match or if there are multiple matches, otherwise the element from the list which matches

ValidateServiceName(name)

source code 

Validate the given service name.

Parameters:
  • name (number or string) - Service name or port specification

ListVolumeGroups()

source code 

List volume groups and their size

Returns: dict
Dictionary with keys volume name and values the size of the volume

BridgeExists(bridge)

source code 

Check whether the given bridge exists in the system

Parameters:
  • bridge (str) - the bridge name to check
Returns: boolean
True if it does

NiceSort(name_list)

source code 

Sort a list of strings based on digit and non-digit groupings.

Given a list of names ['a1', 'a10', 'a11', 'a2'] this function will sort the list in the logical order ['a1', 'a2', 'a10', 'a11'].

The sort algorithm breaks each name in groups of either only-digits or no-digits. Only the first eight such groups are considered, and after that we just use what's left of the string.

Parameters:
  • name_list (list) - the names to be sorted
Returns: list
a copy of the name list sorted with our algorithm

TryConvert(fn, val)

source code 

Try to convert a value ignoring errors.

This function tries to apply function fn to val. If no ValueError or TypeError exceptions are raised, it will return the result, else it will return the original value. Any other exceptions are propagated to the caller.

Parameters:
  • fn (callable) - function to apply to the value
  • val - the value to be converted
Returns:
The converted value if the conversion was successful, otherwise the original value.

IsValidShellParam(word)

source code 

Verifies is the given word is safe from the shell's p.o.v.

This means that we can pass this to a command via the shell and be sure that it doesn't alter the command line and is passed as such to the actual command.

Note that we are overly restrictive here, in order to be on the safe side.

Parameters:
  • word (str) - the word to check
Returns: boolean
True if the word is 'safe'

BuildShellCmd(template, *args)

source code 

Build a safe shell command line from the given arguments.

This function will check all arguments in the args list so that they are valid shell parameters (i.e. they don't contain shell metacharacters). If everything is ok, it will return the result of template % args.

Parameters:
  • template (str) - the string holding the template for the string formatting
Returns: str
the expanded command line

FormatUnit(value, units)

source code 

Formats an incoming number of MiB with the appropriate unit.

Parameters:
  • value (int) - integer representing the value in MiB (1048576)
  • units (char) - the type of formatting we should do:
    • 'h' for automatic scaling
    • 'm' for MiBs
    • 'g' for GiBs
    • 't' for TiBs
Returns: str
the formatted value (with suffix)

ParseUnit(input_string)

source code 

Tries to extract number and scale from the given string.

Input must be in the format NUMBER+ [DOT NUMBER+] SPACE* [UNIT]. If no unit is specified, it defaults to MiB. Return value is always an int in MiB.

ParseCpuMask(cpu_mask)

source code 

Parse a CPU mask definition and return the list of CPU IDs.

CPU mask format: comma-separated list of CPU IDs or dash-separated ID ranges Example: "0-2,5" -> "0,1,2,5"

Parameters:
  • cpu_mask (str) - CPU mask definition
Returns: list of int
list of CPU IDs

AddAuthorizedKey(file_obj, key)

source code 

Adds an SSH public key to an authorized_keys file.

Parameters:
  • file_obj (str or file handle) - path to authorized_keys file
  • key (str) - string containing key

RemoveAuthorizedKey(file_name, key)

source code 

Removes an SSH public key from an authorized_keys file.

Parameters:
  • file_name (str) - path to authorized_keys file
  • key (str) - string containing key

SetEtcHostsEntry(file_name, ip, hostname, aliases)

source code 

Sets the name of an IP address and hostname in /etc/hosts.

Parameters:
  • file_name (str) - path to the file to modify (usually /etc/hosts)
  • ip (str) - the IP address
  • hostname (str) - the hostname to be added
  • aliases (list) - the list of aliases to add for the hostname

AddHostToEtcHosts(hostname, ip)

source code 

Wrapper around SetEtcHostsEntry.

Parameters:
  • hostname (str) - a hostname that will be resolved and added to constants.ETC_HOSTS
  • ip (str) - The ip address of the host

RemoveEtcHostsEntry(file_name, hostname)

source code 

Removes a hostname from /etc/hosts.

IP addresses without names are removed from the file.

Parameters:
  • file_name (str) - path to the file to modify (usually /etc/hosts)
  • hostname (str) - the hostname to be removed

RemoveHostFromEtcHosts(hostname)

source code 

Wrapper around RemoveEtcHostsEntry.

Parameters:
  • hostname (str) - hostname that will be resolved and its full and shot name will be removed from constants.ETC_HOSTS

TimestampForFilename()

source code 

Returns the current time formatted for filenames.

The format doesn't contain colons as some shells and applications them as separators.

CreateBackup(file_name)

source code 

Creates a backup of a file.

Parameters:
  • file_name (str) - file to be backed up
Returns: str
the path to the newly created backup
Raises:

ShellQuote(value)

source code 

Quotes shell argument according to POSIX.

Parameters:
  • value (str) - the argument to be quoted
Returns: str
the quoted value

ShellQuoteArgs(args)

source code 

Quotes a list of shell arguments.

Parameters:
  • args (list) - list of arguments to be quoted
Returns: str
the quoted arguments concatenated with spaces

ListVisibleFiles(path)

source code 

Returns a list of visible files in a directory.

Parameters:
  • path (str) - the directory to enumerate
Returns: list
the list of all files not starting with a dot
Raises:

GetHomeDir(user, default=None)

source code 

Try to get the homedir of the given user.

The user can be passed either as a string (denoting the name) or as an integer (denoting the user id). If the user is not found, the 'default' argument is returned, which defaults to None.

NewUUID()

source code 

Returns a random UUID.

Returns: str

Note: This is a Linux-specific method as it uses the /proc filesystem.

GenerateSecret(numbytes=20)

source code 

Generates a random secret.

This will generate a pseudo-random secret returning an hex string (so that it can be used where an ASCII string is needed).

Parameters:
  • numbytes - the number of bytes which will be represented by the returned string (defaulting to 20, the length of a SHA1 hash)
Returns: str
an hex representation of the pseudo-random sequence

EnsureDirs(dirs)

source code 

Make required directories, if they don't exist.

Parameters:
  • dirs (list of (string, integer)) - list of tuples (dir_name, dir_mode)

ReadFile(file_name, size=-1)

source code 

Reads a file.

Parameters:
  • size (int) - Read at most size bytes (if negative, entire file)
Returns: str
the (possibly partial) content of the file

WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None)

source code 

(Over)write a file atomically.

The file_name and either fn (a function taking one argument, the file descriptor, and which should write the data to it) or data (the contents of the file) must be passed. The other arguments are optional and allow setting the file mode, owner and group, and the mtime/atime of the file.

If the function doesn't raise an exception, it has succeeded and the target file has the new contents. If the function has raised an exception, an existing target file should be unmodified and the temporary file should be removed.

Parameters:
  • file_name (str) - the target filename
  • fn (callable) - content writing function, called with file descriptor as parameter
  • data (str) - contents of the file
  • mode (int) - file mode
  • uid (int) - the owner of the file
  • gid (int) - the group of the file
  • atime (int) - a custom access time to be set on the file
  • mtime (int) - a custom modification time to be set on the file
  • close (boolean) - whether to close file after writing it
  • prewrite (callable) - function to be called before writing content
  • postwrite (callable) - function to be called after writing content
Returns: None or int
None if the 'close' parameter evaluates to True, otherwise the file descriptor
Raises:

GetFileID(path=None, fd=None)

source code 

Returns the file 'id', i.e. the dev/inode and mtime information.

Either the path to the file or the fd must be given.

Parameters:
  • path - the file path
  • fd - a file descriptor
Returns:
a tuple of (device number, inode number, mtime)

VerifyFileID(fi_disk, fi_ours)

source code 

Verifies that two file IDs are matching.

Differences in the inode/device are not accepted, but and older timestamp for fi_disk is accepted.

Parameters:
  • fi_disk - tuple (dev, inode, mtime) representing the actual file data
  • fi_ours - tuple (dev, inode, mtime) representing the last written file data
Returns: boolean

SafeWriteFile(file_name, file_id, **kwargs)

source code 

Wraper over WriteFile that locks the target file.

By keeping the target file locked during WriteFile, we ensure that cooperating writers will safely serialise access to the file.

Parameters:
  • file_name (str) - the target filename
  • file_id (tuple) - a result from GetFileID

ReadOneLineFile(file_name, strict=False)

source code 

Return the first non-empty line from a file.

Parameters:
  • strict (boolean) - if True, abort if the file has more than one non-empty line

FirstFree(seq, base=0)

source code 

Returns the first non-existing integer from seq.

The seq argument should be a sorted list of positive integers. The first time the index of an element is smaller than the element value, the index will be returned.

The base argument is used to start at a different offset, i.e. [3, 4, 6] with offset=3 will return 5.

Example: [0, 1, 3] will return 2.

Parameters:
  • seq (sequence) - the sequence to be analyzed.
  • base (int) - use this value as the base index of the sequence
Returns: int
the first non-used index in the sequence

SingleWaitForFdCondition(fdobj, event, timeout)

source code 

Waits for a condition to occur on the socket.

Immediately returns at the first interruption.

Parameters:
  • fdobj (integer or object supporting a fileno() method) - entity to wait for events on
  • event (integer) - ORed condition (see select module)
  • timeout (float or None) - Timeout in seconds
Returns: int or None
None for timeout, otherwise occured conditions

WaitForFdCondition(fdobj, event, timeout)

source code 

Waits for a condition to occur on the socket.

Retries until the timeout is expired, even if interrupted.

Parameters:
  • fdobj (integer or object supporting a fileno() method) - entity to wait for events on
  • event (integer) - ORed condition (see select module)
  • timeout (float or None) - Timeout in seconds
Returns: int or None
None for timeout, otherwise occured conditions

UniqueSequence(seq)

source code 

Returns a list with unique elements.

Element order is preserved.

Parameters:
  • seq (sequence) - the sequence with the source elements
Returns: list
list of unique elements from seq

NormalizeAndValidateMac(mac)

source code 

Normalizes and check if a MAC address is valid.

Checks whether the supplied MAC address is formally correct, only accepts colon separated format. Normalize it to all lower.

Parameters:
  • mac (str) - the MAC to be validated
Returns: str
returns the normalized and validated MAC.
Raises:

TestDelay(duration)

source code 

Sleep for a fixed amount of time.

Parameters:
  • duration (float) - the sleep duration
Returns: boolean
False for negative value, True otherwise

_CloseFDNoErr(fd, retries=5)

source code 

Close a file descriptor ignoring errors.

Parameters:
  • fd (int) - the file descriptor
  • retries (int) - how many retries to make, in case we get any other error than EBADF

CloseFDs(noclose_fds=None)

source code 

Close file descriptors.

This closes all file descriptors above 2 (i.e. except stdin/out/err).

Parameters:
  • noclose_fds (list or None) - if given, it denotes a list of file descriptor that should not be closed

Mlockall(_ctypes=ctypes)

source code 

Lock current process' virtual address space into RAM.

This is equivalent to the C call mlockall(MCL_CURRENT|MCL_FUTURE), see mlock(2) for more details. This function requires ctypes module.

Raises:

Daemonize(logfile)

source code 

Daemonize the current process.

This detaches the current process from the controlling terminal and runs it in the background as a daemon.

Parameters:
  • logfile (str) - the logfile to which we should redirect stdout/stderr
Returns: int
the value zero

DaemonPidFileName(name)

source code 

Compute a ganeti pid file absolute path

Parameters:
  • name (str) - the daemon name
Returns: str
the full path to the pidfile corresponding to the given daemon name

WritePidFile(pidfile)

source code 

Write the current process pidfile.

Parameters:
  • pidfile (sting) - the path to the file to be written
Returns: int
the file descriptor of the lock file; do not close this unless you want to unlock the pid file
Raises:
  • errors.LockError - if the pid file already exists and points to a live process

RemovePidFile(name)

source code 

Remove the current process pidfile.

Any errors are ignored.

Parameters:
  • name (str) - the daemon name used to derive the pidfile name

KillProcess(pid, signal_=signal.SIGTERM, timeout=30, waitpid=False)

source code 

Kill a process given by its pid.

Parameters:
  • pid (int) - The PID to terminate.
  • signal_ (int) - The signal to send, by default SIGTERM
  • timeout (int) - The timeout after which, if the process is still alive, a SIGKILL will be sent. If not positive, no such checking will be done
  • waitpid (boolean) - If true, we should waitpid on this process after sending signals, since it's our own child and otherwise it would remain as zombie

FindFile(name, search_path, test=os.path.exists)

source code 

Look for a filesystem object in a given path.

This is an abstract method to search for filesystem object (files, dirs) under a given search path.

Parameters:
  • name (str) - the name to look for
  • search_path (str) - location to start at
  • test (callable) - a function taking one argument that should return True if the a given object is valid; the default value is os.path.exists, causing only existing files to be returned
Returns: str or None
full path to the object if found, None otherwise

CheckVolumeGroupSize(vglist, vgname, minsize)

source code 

Checks if the volume group list is valid.

The function will check if a given volume group is in the list of volume groups and has a minimum size.

Parameters:
  • vglist (dict) - dictionary of volume group names and their size
  • vgname (str) - the volume group we should check
  • minsize (int) - the minimum size we accept
Returns: None or str
None for success, otherwise the error message

SplitTime(value)

source code 

Splits time as floating point number into a tuple.

Parameters:
  • value (int or float) - Time in seconds
Returns:
Tuple containing (seconds, microseconds)

MergeTime(timetuple)

source code 

Merges a tuple into time as a floating point number.

Parameters:
  • timetuple (tuple) - Time as tuple, (seconds, microseconds)
Returns:
Time as a floating point number expressed in seconds

SetupLogging(logfile, debug=0, stderr_logging=False, program="", multithreaded=False, syslog=constants.SYSLOG_USAGE, console_logging=False)

source code 

Configures the logging module.

Parameters:
  • logfile (str) - the filename to which we should log
  • debug (integer) - if greater than zero, enable debug messages, otherwise only those at INFO and above level
  • stderr_logging (boolean) - whether we should also log to the standard error
  • program (str) - the name under which we should log messages
  • multithreaded (boolean) - if True, will add the thread name to the log file
  • syslog (string) - one of 'no', 'yes', 'only':
    • if no, syslog is not used
    • if yes, syslog is used (in addition to file-logging)
    • if only, only syslog is used
  • console_logging (boolean) - if True, will use a FileHandler which falls back to the system console if logging fails
Raises:
  • EnvironmentError - if we can't open the log file and syslog/stderr logging is disabled

IsNormAbsPath(path)

source code 

Check whether a path is absolute and also normalized

This avoids things like /dir/../../other/path to be valid.

PathJoin(*args)

source code 

Safe-join a list of path components.

Requirements:

  • the first argument must be an absolute path
  • no component in the path must have backtracking (e.g. /../), since we check for normalization at the end
Parameters:
  • args - the path components to be joined
Raises:
  • ValueError - for invalid paths

TailFile(fname, lines=20)

source code 

Return the last lines from a file.

Parameters:
  • fname - the file name
  • lines (int) - the (maximum) number of lines to return

Note: this function will only read and parse the last 4KB of the file; if the lines are very long, it could be that less than the requested number of lines are returned

_ParseAsn1Generalizedtime(value)

source code 

Parses an ASN1 GENERALIZEDTIME timestamp as used by pyOpenSSL.

Parameters:
  • value (string) - ASN1 GENERALIZEDTIME timestamp

GetX509CertValidity(cert)

source code 

Returns the validity period of the certificate.

Parameters:
  • cert (OpenSSL.crypto.X509) - X509 certificate object

_VerifyCertificateInner(expired, not_before, not_after, now, warn_days, error_days)

source code 

Verifies certificate validity.

Parameters:
  • expired (bool) - Whether pyOpenSSL considers the certificate as expired
  • not_before (number or None) - Unix timestamp before which certificate is not valid
  • not_after (number or None) - Unix timestamp after which certificate is invalid
  • now (number) - Current time as Unix timestamp
  • warn_days (number or None) - How many days before expiration a warning should be reported
  • error_days (number or None) - How many days before expiration an error should be reported

VerifyX509Certificate(cert, warn_days, error_days)

source code 

Verifies a certificate for LUVerifyCluster.

Parameters:
  • cert (OpenSSL.crypto.X509) - X509 certificate object
  • warn_days (number or None) - How many days before expiration a warning should be reported
  • error_days (number or None) - How many days before expiration an error should be reported

SignX509Certificate(cert, key, salt)

source code 

Sign a X509 certificate.

An RFC822-like signature header is added in front of the certificate.

Parameters:
  • cert (OpenSSL.crypto.X509) - X509 certificate object
  • key (string) - Key for HMAC
  • salt (string) - Salt for HMAC
Returns: string
Serialized and signed certificate in PEM format

LoadSignedX509Certificate(cert_pem, key)

source code 

Verifies a signed X509 certificate.

Parameters:
  • cert_pem (string) - Certificate in PEM format and with signature header
  • key (string) - Key for HMAC
Returns: tuple; (OpenSSL.crypto.X509, string)
X509 certificate object and salt

Sha1Hmac(key, text, salt=None)

source code 

Calculates the HMAC-SHA1 digest of a text.

HMAC is defined in RFC2104.

Parameters:
  • key (string) - Secret key
  • text (string)

VerifySha1Hmac(key, text, digest, salt=None)

source code 

Verifies the HMAC-SHA1 digest of a text.

HMAC is defined in RFC2104.

Parameters:
  • key (string) - Secret key
  • digest (string) - Expected digest
  • text (string)
Returns: bool
Whether HMAC-SHA1 digest matches

SafeEncode(text)

source code 

Return a 'safe' version of a source string.

This function mangles the input string and returns a version that should be safe to display/encode as ASCII. To this end, we first convert it to ASCII using the 'backslashreplace' encoding which should get rid of any non-ASCII chars, and then we process it through a loop copied from the string repr sources in the python; we don't use string_escape anymore since that escape single quotes and backslashes too, and that is too much; and that escaping is not stable, i.e. string_escape(string_escape(x)) != string_escape(x).

Parameters:
  • text (str or unicode) - input data
Returns: str
a safe version of text

UnescapeAndSplit(text, sep=",")

source code 

Split and unescape a string based on a given separator.

This function splits a string based on a separator where the separator itself can be escape in order to be an element of the elements. The escaping rules are (assuming coma being the separator):

  • a plain , separates the elements
  • a sequence \\, (double backslash plus comma) is handled as a backslash plus a separator comma
  • a sequence \, (backslash plus comma) is handled as a non-separator comma
Parameters:
  • text (string) - the string to split
  • text (string) - the separator
  • sep (string)
Returns: string
a list of strings

CommaJoin(names)

source code 

Nicely join a set of identifiers.

Parameters:
  • names - set, list or tuple
Returns:
a string with the formatted results

FindMatch(data, name)

source code 

Tries to find an item in a dictionary matching a name.

Callers have to ensure the data names aren't contradictory (e.g. a regexp that matches a string). If the name isn't a direct key, all regular expression objects in the dictionary are matched against it.

Parameters:
  • data (dict) - Dictionary containing data
  • name (string) - Name to look for
Returns: tuple; (value in dictionary, matched groups as list)

BytesToMebibyte(value)

source code 

Converts bytes to mebibytes.

Parameters:
  • value (int) - Value in bytes
Returns: int
Value in mebibytes

CalculateDirectorySize(path)

source code 

Calculates the size of a directory recursively.

Parameters:
  • path (string) - Path to directory
Returns: int
Size in mebibytes

GetMounts(filename=constants.PROC_MOUNTS)

source code 

Returns the list of mounted filesystems.

This function is Linux-specific.

Parameters:
  • filename - path of mounts file (/proc/mounts by default)
Returns: list of tuples
list of mount entries (device, mountpoint, fstype, options)

GetFilesystemStats(path)

source code 

Returns the total and free space on a filesystem.

Parameters:
  • path (string) - Path on filesystem to be examined
Returns: int
tuple of (Total space, Free space) in mebibytes

RunInSeparateProcess(fn, *args)

source code 

Runs a function in a separate process.

Note: Only boolean return values are supported.

Parameters:
  • fn (callable) - Function to be called
Returns: bool
Function's result

IgnoreProcessNotFound(fn, *args, **kwargs)

source code 

Ignores ESRCH when calling a process-related function.

ESRCH is raised when a process is not found.

Returns: bool
Whether process was found

LockFile(fd)

source code 

Locks a file using POSIX locks.

Parameters:
  • fd (int) - the file descriptor we need to lock

FormatTime(val)

source code 

Formats a time value.

Parameters:
  • val (float or None) - the timestamp as returned by time.time()
Returns:
a string value or N/A if we don't have a valid timestamp

FormatSeconds(secs)

source code 

Formats seconds for easier reading.

Parameters:
  • secs (number) - Number of seconds
Returns: string
Formatted seconds (e.g. "2d 9h 19m 49s")

ReadWatcherPauseFile(filename, now=None, remove_after=3600)

source code 

Reads the watcher pause file.

Parameters:
  • filename (string) - Path to watcher pause file
  • now (None, float or int) - Current time as Unix timestamp
  • remove_after (int) - Remove watcher pause file after specified amount of seconds past the pause end time

Retry(fn, delay, timeout, args=None, wait_fn=time.sleep, _time_fn=time.time)

source code 

Call a function repeatedly until it succeeds.

The function fn is called repeatedly until it doesn't throw RetryAgain anymore. Between calls a delay, specified by delay, is inserted. After a total of timeout seconds, this function throws RetryTimeout.

delay can be one of the following:

  • callable returning the delay length as a float
  • Tuple of (start, factor, limit)
  • RETRY_REMAINING_TIME to sleep until the timeout expires (this is useful when overriding wait_fn to wait for an external event)
  • A static delay as a number (int or float)
Parameters:
  • fn (callable) - Function to be called
  • delay - Either a callable (returning the delay), a tuple of (start, factor, limit) (see _RetryDelayCalculator), RETRY_REMAINING_TIME or a number (int or float)
  • timeout (float) - Total timeout
  • wait_fn (callable) - Waiting function
Returns:
Return value of function

GenerateSelfSignedX509Cert(common_name, validity)

source code 

Generates a self-signed X509 certificate.

Parameters:
  • common_name (string) - commonName value
  • validity (int) - Validity for certificate in seconds

GenerateSelfSignedSslCert(filename, common_name=constants.X509_CERT_CN, validity=constants.X509_CERT_DEFAULT_VALIDITY)

source code 

Legacy function to generate self-signed X509 certificate.

Parameters:
  • filename (str) - path to write certificate to
  • common_name (string) - commonName value
  • validity (int) - validity of certificate in number of days

SignalHandled(signums)

source code 

Signal Handled decoration.

This special decorator installs a signal handler and then calls the target function. The function must accept a 'signal_handlers' keyword argument, which will contain a dict indexed by signal number, with SignalHandler objects as values.

The decorator can be safely stacked with iself, to handle multiple signals with different handlers.

Parameters:
  • signums (list) - signals to intercept

Variables Details [hide private]

VALID_X509_SIGNATURE_SALT

Value:
re.compile("^%s+$" % HEX_CHAR_RE, re.S)

X509_SIGNATURE

Value:
re.compile(r"^%s:\s*(?P<salt>%s+)/(?P<sign>%s+)$" %(re.escape(constant\
s.X509_CERT_SIGNATURE_HEADER), HEX_CHAR_RE, HEX_CHAR_RE), re.S | re.I)

UUID_RE

Value:
re.compile('^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-' '[a-f0-9]{4}-[a-f0-\
9]{12}$')