module documentation

Utility functions for I/O.

Class FileStatHelper Helper to store file handle's fstat.
Class TemporaryFileManager Stores the list of files to be deleted and removes them on demand.
Function BytesToMebibyte Converts bytes to mebibytes.
Function CalculateDirectorySize Calculates the size of a directory recursively.
Function CanRead Returns True if the user can access (read) the file.
Function CreateBackup Creates a backup of a file.
Function DaemonPidFileName Compute a ganeti pid file absolute path
Function EnforcePermission Enforces that given path has given permissions.
Function EnsureDirs Make required directories, if they don't exist.
Function ErrnoOrStr Format an EnvironmentError exception.
Function FindFile Look for a filesystem object in a given path.
Function GetFileID Returns the file 'id', i.e. the dev/inode and mtime information.
Function GetFilesystemStats Returns the total and free space on a filesystem.
Function IsBelowDir Check whether a path is below a root dir.
Function IsNormAbsPath Check whether a path is absolute and also normalized
Function IsUrl Check whether a path is a HTTP URL.
Function IsUserInGroup Returns True if the user belongs to the group.
Function ListVisibleFiles Returns a list of visible files in a directory.
Function Makedirs Super-mkdir; create a leaf directory and all intermediate ones.
Function MakeDirWithPerm Enforces that given path is a dir and has given mode, uid and gid set.
Function NewUUID Returns a random UUID.
Function OpenTTY Returns a text I/O object pointing a TTY (/dev/tty by default)
Function PathJoin Safe-join a list of path components.
Function ReadBinaryFile Reads a binary file.
Function ReadFile Reads a text file.
Function ReadLockedPidFile Reads a locked PID file.
Function ReadOneLineFile Return the first non-empty line from a file.
Function ReadPidFile Read a pid from a file.
Function ReadWatcherPauseFile Reads the watcher pause file.
Function RemoveDir Remove an empty directory.
Function RemoveFile Remove a file ignoring some errors.
Function RenameFile Renames a file.
Function SafeWriteFile Wraper over WriteFile that locks the target file.
Function TailFile Return the last lines from a file.
Function TimestampForFilename Returns the current time formatted for filenames.
Function VerifyFileID Verifies that two file IDs are matching.
Function WriteFile (Over)write a file atomically.
Function WritePidFile Write the current process pidfile.
Constant KEEP_PERMS_VALUES Undocumented
Constant KP_ALWAYS Undocumented
Constant KP_IF_EXISTS Undocumented
Constant KP_NEVER Undocumented
Constant URL_RE Undocumented
Function _ParsePidFileContents Tries to extract a process ID from a PID file's content.
Constant _LOST_AND_FOUND Undocumented
def BytesToMebibyte(value):

Converts bytes to mebibytes.

Parameters
value:intValue in bytes
Returns
intValue in mebibytes
def CalculateDirectorySize(path):

Calculates the size of a directory recursively.

Parameters
path:stringPath to directory
Returns
intSize in mebibytes
def CanRead(username, filename):

Returns True if the user can access (read) the file.

Parameters
username:stringthe name of the user
filename:stringthe name of the file
Returns
boolUndocumented
def CreateBackup(file_name):

Creates a backup of a file.

Parameters
file_name:strfile to be backed up
Returns
strthe path to the newly created backup
Raises
errors.ProgrammerErrorfor invalid file names
def DaemonPidFileName(name):

Compute a ganeti pid file absolute path

Parameters
name:strthe daemon name
Returns
strthe full path to the pidfile corresponding to the given daemon name
def EnforcePermission(path, mode, uid=None, gid=None, must_exist=True, _chmod_fn=os.chmod, _chown_fn=os.chown, _stat_fn=os.stat):

Enforces that given path has given permissions.

Parameters
pathThe path to the file
modeThe mode of the file
uidThe uid of the owner of this file
gidThe gid of the owner of this file
must_existSpecifies if non-existance of path will be an error
_chmod_fnchmod function to use (unittest only)
_chown_fnchown function to use (unittest only)
_stat_fnUndocumented
def EnsureDirs(dirs):

Make required directories, if they don't exist.

Parameters
dirs:list of (string, integer)list of tuples (dir_name, dir_mode)
def ErrnoOrStr(err):

Format an EnvironmentError exception.

If the err argument has an errno attribute, it will be looked up and converted into a textual E... description. Otherwise the string representation of the error will be returned.

Parameters
err:EnvironmentErrorthe exception to format
def FindFile(name, search_path, test=os.path.exists):

Look for a filesystem object in a given path.

This is an abstract method to search for filesystem object (files, dirs) under a given search path.

Parameters
name:strthe name to look for
search_path:iterable of stringlocations to start at
test:callablea function taking one argument that should return True if the a given object is valid; the default value is os.path.exists, causing only existing files to be returned
Returns
str or Nonefull path to the object if found, None otherwise
def GetFileID(path=None, fd=None):

Returns the file 'id', i.e. the dev/inode and mtime information.

Either the path to the file or the fd must be given.

Parameters
paththe file path
fda file descriptor
Returns
a tuple of (device number, inode number, mtime)
def GetFilesystemStats(path):

Returns the total and free space on a filesystem.

Parameters
path:stringPath on filesystem to be examined
Returns
inttuple of (Total space, Free space) in mebibytes
def IsBelowDir(root, other_path):

Check whether a path is below a root dir.

This works around the nasty byte-byte comparison of commonprefix.

def IsNormAbsPath(path):

Check whether a path is absolute and also normalized

This avoids things like /dir/../../other/path to be valid.

def IsUrl(path):

Check whether a path is a HTTP URL.

def IsUserInGroup(uid, gid):

Returns True if the user belongs to the group.

Parameters
uid:intthe user id
gid:intthe group id
Returns
boolUndocumented
def ListVisibleFiles(path, _is_mountpoint=os.path.ismount):

Returns a list of visible files in a directory.

Parameters
path:strthe directory to enumerate
_is_mountpointUndocumented
Returns
listthe list of all files not starting with a dot
Raises
ProgrammerErrorif path is not an absolue and normalized path
def Makedirs(path, mode=488):

Super-mkdir; create a leaf directory and all intermediate ones.

This is a wrapper around os.makedirs adding error handling not implemented before Python 2.5.

def MakeDirWithPerm(path, mode, uid, gid, _lstat_fn=os.lstat, _mkdir_fn=os.mkdir, _perm_fn=EnforcePermission):

Enforces that given path is a dir and has given mode, uid and gid set.

Parameters
pathThe path to the file
modeThe mode of the file
uidThe uid of the owner of this file
gidThe gid of the owner of this file
_lstat_fnStat function to use (unittest only)
_mkdir_fnmkdir function to use (unittest only)
_perm_fnpermission setter function to use (unittest only)
def NewUUID():

Returns a random UUID.

Returns
strUndocumented
Note
This is a Linux-specific method as it uses the /proc filesystem.
def OpenTTY(device='/dev/tty'):

Returns a text I/O object pointing a TTY (/dev/tty by default)

As of Python 3.7, /dev/tty cannot be opened in buffered and/or text mode, see https://bugs.python.org/issue20074. Work around this by using TextIOWrapper over a BufferedRWPair to return a line-buffered TTY I/O object.

Parameters
device:stringpath to the TTY/PTY device to open
def PathJoin(*args):

Safe-join a list of path components.

Requirements:

  • the first argument must be an absolute path
  • no component in the path must have backtracking (e.g. /../), since we check for normalization at the end
Parameters
*argsthe path components to be joined
Raises
ValueErrorfor invalid paths
def ReadBinaryFile(file_name, size=-1, preread=None):

Reads a binary file.

Parameters
file_nameUndocumented
size:intRead at most size bytes (if negative, entire file)
preread:callable receiving file handle as single parameterFunction called before file is read
Returns
bytesthe (possibly partial) content of the file
Raises
IOErrorif the file cannot be opened
def ReadFile(file_name, size=-1, preread=None):

Reads a text file.

Parameters
file_nameUndocumented
size:intRead at most size bytes (if negative, entire file)
preread:callable receiving file handle as single parameterFunction called before file is read
Returns
strthe (possibly partial) content of the file
Raises
IOErrorif the file cannot be opened
def ReadLockedPidFile(path):

Reads a locked PID file.

This can be used together with utils.process.StartDaemon.

Parameters
path:stringPath to PID file
Returns
PID as integer or, if file was unlocked or couldn't be opened, None
def ReadOneLineFile(file_name, strict=False):

Return the first non-empty line from a file.

Parameters
file_nameUndocumented
strict:booleanif True, abort if the file has more than one non-empty line
def ReadPidFile(pidfile):

Read a pid from a file.

Parameters
pidfile:stringpath to the file containing the pid
Returns
intThe process id, if the file exists and contains a valid PID, otherwise 0
def ReadWatcherPauseFile(filename, now=None, remove_after=3600):

Reads the watcher pause file.

Parameters
filename:stringPath to watcher pause file
now:None, float or intCurrent time as Unix timestamp
remove_after:intRemove watcher pause file after specified amount of seconds past the pause end time
def RemoveDir(dirname):

Remove an empty directory.

Remove a directory, ignoring non-existing ones. Other errors are passed. This includes the case, where the directory is not empty, so it can't be removed.

Parameters
dirname:strthe empty directory to be removed
def RemoveFile(filename):

Remove a file ignoring some errors.

Remove a file, ignoring non-existing ones or directories. Other errors are passed.

Parameters
filename:strthe file to be removed
def RenameFile(old, new, mkdir=False, mkdir_mode=488, dir_uid=None, dir_gid=None):

Renames a file.

This just creates the very least directory if it does not exist and mkdir is set to true.

Parameters
old:stringOriginal path
new:stringNew path
mkdir:boolWhether to create target directory if it doesn't exist
mkdir_mode:intMode for newly created directories
dir_uid:intThe uid for the (if fresh created) dir
dir_gid:intThe gid for the (if fresh created) dir
def SafeWriteFile(file_name, file_id, **kwargs):

Wraper over WriteFile that locks the target file.

By keeping the target file locked during WriteFile, we ensure that cooperating writers will safely serialise access to the file.

Parameters
file_name:strthe target filename
file_id:tuplea result from GetFileID
**kwargsUndocumented
def TailFile(fname, lines=20):

Return the last lines from a file.

Parameters
fnamethe file name
lines:intthe (maximum) number of lines to return
Note
this function will only read and parse the last 4KB of the file; if the lines are very long, it could be that less than the requested number of lines are returned
def TimestampForFilename():

Returns the current time formatted for filenames.

The format doesn't contain colons as some shells and applications treat them as separators. Uses the local timezone.

def VerifyFileID(fi_disk, fi_ours):

Verifies that two file IDs are matching.

Differences in the inode/device are not accepted, but and older timestamp for fi_disk is accepted.

Parameters
fi_disktuple (dev, inode, mtime) representing the actual file data
fi_ourstuple (dev, inode, mtime) representing the last written file data
Returns
booleanUndocumented
def WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None, keep_perms=KP_NEVER):

(Over)write a file atomically.

The file_name and either fn (a function taking one argument, the file descriptor, and which should write the data to it) or data (the contents of the file) must be passed. The other arguments are optional and allow setting the file mode, owner and group, and the mtime/atime of the file.

If the function doesn't raise an exception, it has succeeded and the target file has the new contents. If the function has raised an exception, an existing target file should be unmodified and the temporary file should be removed.

Parameters
file_name:strthe target filename
fn:callablecontent writing function, called with file descriptor as parameter
data:str or bytescontents of the file
mode:intfile mode
uid:intthe owner of the file
gid:intthe group of the file
atime:inta custom access time to be set on the file
mtime:inta custom modification time to be set on the file
close:booleanwhether to close file after writing it
dry_runUndocumented
backupUndocumented
prewrite:callablefunction to be called before writing content
postwrite:callablefunction to be called after writing content
keep_perms:members of KEEP_PERMS_VALUESif KP_NEVER (default), owner, group, and mode are taken from the other parameters; if KP_ALWAYS, owner, group, and mode are copied from the existing file; if KP_IF_EXISTS, owner, group, and mode are taken from the file, and if the file doesn't exist, they are taken from the other parameters. It is an error to pass KP_ALWAYS when the file doesn't exist or when uid, gid, or mode are set to non-default values.
Returns
None or intNone if the 'close' parameter evaluates to True, otherwise the file descriptor
Raises
errors.ProgrammerErrorif any of the arguments are not valid
def WritePidFile(pidfile):

Write the current process pidfile.

Parameters
pidfile:stringthe path to the file to be written
Returns
intthe file descriptor of the lock file; do not close this unless you want to unlock the pid file
Raises
errors.LockErrorif the pid file already exists and points to a live process
KEEP_PERMS_VALUES =

Undocumented

Value
[KP_NEVER, KP_ALWAYS, KP_IF_EXISTS]
KP_ALWAYS: int =

Undocumented

Value
1
KP_IF_EXISTS: int =

Undocumented

Value
2
KP_NEVER: int =

Undocumented

Value
0
URL_RE =

Undocumented

Value
re.compile(r'(https?|ftps?)://')
def _ParsePidFileContents(data):

Tries to extract a process ID from a PID file's content.

Parameters
data:stringUndocumented
Returns
intZero if nothing could be read, PID otherwise
_LOST_AND_FOUND: str =

Undocumented

Value
'lost+found'