Package ganeti :: Package utils :: Module io

[frames] | no frames]

Module io

source code

Utility functions for I/O.

Classes

[hide private]

FileStatHelper
Helper to store file handle's fstat.

TemporaryFileManager
Stores the list of files to be deleted and removes them on demand.

Functions

[hide private]

ErrnoOrStr(err)
Format an EnvironmentError exception.

source code

str

ReadFile(file_name, size=-1, preread=None)
Reads a file.

source code

None or int

WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None, keep_perms=KP_NEVER)
(Over)write a file atomically.

source code

GetFileID(path=None, fd=None)
Returns the file 'id', i.e.

source code

boolean

VerifyFileID(fi_disk, fi_ours)
Verifies that two file IDs are matching.

source code

SafeWriteFile(file_name, file_id, **kwargs)
Wraper over WriteFile that locks the target file.

source code

ReadOneLineFile(file_name, strict=False)
Return the first non-empty line from a file.

source code

RemoveFile(filename)
Remove a file ignoring some errors.

source code

RemoveDir(dirname)
Remove an empty directory.

source code

RenameFile(old, new, mkdir=False, mkdir_mode=0750, dir_uid=None, dir_gid=None)
Renames a file.

source code

EnforcePermission(path, mode, uid=None, gid=None, must_exist=True, _chmod_fn=os.chmod, _chown_fn=os.chown, _stat_fn=os.stat)
Enforces that given path has given permissions.

source code

MakeDirWithPerm(path, mode, uid, gid, _lstat_fn=os.lstat, _mkdir_fn=os.mkdir, _perm_fn=EnforcePermission)
Enforces that given path is a dir and has given mode, uid and gid set.

source code

Makedirs(path, mode=0750)
Super-mkdir; create a leaf directory and all intermediate ones.

source code

TimestampForFilename()
Returns the current time formatted for filenames.

source code

str

CreateBackup(file_name)
Creates a backup of a file.

source code

list

ListVisibleFiles(path, _is_mountpoint=os.path.ismount)
Returns a list of visible files in a directory.

source code

EnsureDirs(dirs)
Make required directories, if they don't exist.

source code

str or None

FindFile(name, search_path, test=os.path.exists)
Look for a filesystem object in a given path.

source code

IsNormAbsPath(path)
Check whether a path is absolute and also normalized

source code

IsBelowDir(root, other_path)
Check whether a path is below a root dir.

source code

IsUrl(path)
Check whether a path is a HTTP URL.

source code

PathJoin(*args)
Safe-join a list of path components.

source code

TailFile(fname, lines=20)
Return the last lines from a file.

source code

int

BytesToMebibyte(value)
Converts bytes to mebibytes.

source code

int

CalculateDirectorySize(path)
Calculates the size of a directory recursively.

source code

int

GetFilesystemStats(path)
Returns the total and free space on a filesystem.

source code

int

ReadPidFile(pidfile)
Read a pid from a file.

source code

int

_ParsePidFileContents(data)
Tries to extract a process ID from a PID file's content.

source code

ReadLockedPidFile(path)
Reads a locked PID file.

source code

tuple

_SplitSshKey(key)
Splits a line for SSH's authorized_keys file. source code

AddAuthorizedKey(file_obj, key)
Adds an SSH public key to an authorized_keys file.

source code

RemoveAuthorizedKey(file_name, key)
Removes an SSH public key from an authorized_keys file.

source code

str

DaemonPidFileName(name)
Compute a ganeti pid file absolute path

source code

int

WritePidFile(pidfile)
Write the current process pidfile.

source code

ReadWatcherPauseFile(filename, now=None, remove_after=3600)
Reads the watcher pause file.

source code

str

NewUUID()
Returns a random UUID.

source code

bool

IsUserInGroup(uid, gid)
Returns True if the user belongs to the group.

source code

bool

CanRead(username, filename)
Returns True if the user can access (read) the file.

source code

Variables

[hide private]

_LOST_AND_FOUND = "lost+found"
Directory used by fsck(8) to store recovered data, usually at a file system's root directory

KP_NEVER = 0

KP_ALWAYS = 1

KP_IF_EXISTS = 2

KEEP_PERMS_VALUES = [KP_NEVER, KP_ALWAYS, KP_IF_EXISTS,]

URL_RE = re.compile(r'(https?|ftps?)://')

Imports: os, re, logging, shutil, tempfile, errno, time, stat, grp, pwd, errors, constants, pathutils, filelock

Function Details

[hide private]

ErrnoOrStr(err)

source code

Format an EnvironmentError exception.

If the err argument has an errno attribute, it will be looked up and converted into a textual E... description. Otherwise the string representation of the error will be returned.

Parameters:

err (EnvironmentError) - the exception to format

ReadFile(file_name, size=-1, preread=None)

source code

Reads a file.

Parameters:

size (int) - Read at most size bytes (if negative, entire file)
preread (callable receiving file handle as single parameter) - Function called before file is read

Returns: str

the (possibly partial) content of the file

Raises:

IOError - if the file cannot be opened

WriteFile(file_name, fn=None, data=None, mode=None, uid=-1, gid=-1, atime=None, mtime=None, close=True, dry_run=False, backup=False, prewrite=None, postwrite=None, keep_perms=KP_NEVER)

source code

(Over)write a file atomically.

The file_name and either fn (a function taking one argument, the file descriptor, and which should write the data to it) or data (the contents of the file) must be passed. The other arguments are optional and allow setting the file mode, owner and group, and the mtime/atime of the file.

If the function doesn't raise an exception, it has succeeded and the target file has the new contents. If the function has raised an exception, an existing target file should be unmodified and the temporary file should be removed.

Parameters:

file_name (str) - the target filename
fn (callable) - content writing function, called with file descriptor as parameter
data (str) - contents of the file
mode (int) - file mode
uid (int) - the owner of the file
gid (int) - the group of the file
atime (int) - a custom access time to be set on the file
mtime (int) - a custom modification time to be set on the file
close (boolean) - whether to close file after writing it
prewrite (callable) - function to be called before writing content
postwrite (callable) - function to be called after writing content
keep_perms (members of KEEP_PERMS_VALUES) - if KP_NEVER (default), owner, group, and mode are taken from the other parameters; if KP_ALWAYS, owner, group, and mode are copied from the existing file; if KP_IF_EXISTS, owner, group, and mode are taken from the file, and if the file doesn't exist, they are taken from the other parameters. It is an error to pass KP_ALWAYS when the file doesn't exist or when uid, gid, or mode are set to non-default values.

Returns: None or int

None if the 'close' parameter evaluates to True, otherwise the file descriptor

Raises:

errors.ProgrammerError - if any of the arguments are not valid

GetFileID(path=None, fd=None)

source code

Returns the file 'id', i.e. the dev/inode and mtime information.

Either the path to the file or the fd must be given.

Parameters:

path - the file path
fd - a file descriptor

Returns:

a tuple of (device number, inode number, mtime)

VerifyFileID(fi_disk, fi_ours)

source code

Verifies that two file IDs are matching.

Differences in the inode/device are not accepted, but and older timestamp for fi_disk is accepted.

Parameters:

fi_disk - tuple (dev, inode, mtime) representing the actual file data
fi_ours - tuple (dev, inode, mtime) representing the last written file data

Returns: boolean

SafeWriteFile(file_name, file_id, **kwargs)

source code

Wraper over WriteFile that locks the target file.

By keeping the target file locked during WriteFile, we ensure that cooperating writers will safely serialise access to the file.

Parameters:

file_name (str) - the target filename
file_id (tuple) - a result from GetFileID

ReadOneLineFile(file_name, strict=False)

source code

Return the first non-empty line from a file.

Parameters:

strict (boolean) - if True, abort if the file has more than one non-empty line

RemoveFile(filename)

source code

Remove a file ignoring some errors.

Remove a file, ignoring non-existing ones or directories. Other errors are passed.

Parameters:

filename (str) - the file to be removed

RemoveDir(dirname)

source code

Remove an empty directory.

Remove a directory, ignoring non-existing ones. Other errors are passed. This includes the case, where the directory is not empty, so it can't be removed.

Parameters:

dirname (str) - the empty directory to be removed

RenameFile(old, new, mkdir=False, mkdir_mode=0750, dir_uid=None, dir_gid=None)

source code

Renames a file.

This just creates the very least directory if it does not exist and mkdir is set to true.

Parameters:

old (string) - Original path
new (string) - New path
mkdir (bool) - Whether to create target directory if it doesn't exist
mkdir_mode (int) - Mode for newly created directories
dir_uid (int) - The uid for the (if fresh created) dir
dir_gid (int) - The gid for the (if fresh created) dir

EnforcePermission(path, mode, uid=None, gid=None, must_exist=True, _chmod_fn=os.chmod, _chown_fn=os.chown, _stat_fn=os.stat)

source code

Enforces that given path has given permissions.

Parameters:

path - The path to the file
mode - The mode of the file
uid - The uid of the owner of this file
gid - The gid of the owner of this file
must_exist - Specifies if non-existance of path will be an error
_chmod_fn - chmod function to use (unittest only)
_chown_fn - chown function to use (unittest only)

MakeDirWithPerm(path, mode, uid, gid, _lstat_fn=os.lstat, _mkdir_fn=os.mkdir, _perm_fn=EnforcePermission)

source code

Enforces that given path is a dir and has given mode, uid and gid set.

Parameters:

path - The path to the file
mode - The mode of the file
uid - The uid of the owner of this file
gid - The gid of the owner of this file
_lstat_fn - Stat function to use (unittest only)
_mkdir_fn - mkdir function to use (unittest only)
_perm_fn - permission setter function to use (unittest only)

Makedirs(path, mode=0750)

source code

Super-mkdir; create a leaf directory and all intermediate ones.

This is a wrapper around os.makedirs adding error handling not implemented before Python 2.5.

TimestampForFilename()

source code

Returns the current time formatted for filenames.

The format doesn't contain colons as some shells and applications treat them as separators. Uses the local timezone.

CreateBackup(file_name)

source code

Creates a backup of a file.

Parameters:

file_name (str) - file to be backed up

Returns: str

the path to the newly created backup

Raises:

errors.ProgrammerError - for invalid file names

ListVisibleFiles(path, _is_mountpoint=os.path.ismount)

source code

Returns a list of visible files in a directory.

Parameters:

path (str) - the directory to enumerate

Returns: list

the list of all files not starting with a dot

Raises:

ProgrammerError - if path is not an absolue and normalized path

EnsureDirs(dirs)

source code

Make required directories, if they don't exist.

Parameters:

dirs (list of (string, integer)) - list of tuples (dir_name, dir_mode)

FindFile(name, search_path, test=os.path.exists)

source code

Look for a filesystem object in a given path.

This is an abstract method to search for filesystem object (files, dirs) under a given search path.

Parameters:

name (str) - the name to look for
search_path (iterable of string) - locations to start at
test (callable) - a function taking one argument that should return True if the a given object is valid; the default value is os.path.exists, causing only existing files to be returned

Returns: str or None

full path to the object if found, None otherwise

IsNormAbsPath(path)

source code

Check whether a path is absolute and also normalized

This avoids things like /dir/../../other/path to be valid.

IsBelowDir(root, other_path)

source code

Check whether a path is below a root dir.

This works around the nasty byte-byte comparison of commonprefix.

PathJoin(*args)

source code

Safe-join a list of path components.

Requirements:

the first argument must be an absolute path
no component in the path must have backtracking (e.g. /../), since we check for normalization at the end

Parameters:

args - the path components to be joined

Raises:

ValueError - for invalid paths

TailFile(fname, lines=20)

source code

Return the last lines from a file.

Parameters:

fname - the file name
lines (int) - the (maximum) number of lines to return

Note: this function will only read and parse the last 4KB of the file; if the lines are very long, it could be that less than the requested number of lines are returned

BytesToMebibyte(value)

source code

Converts bytes to mebibytes.

Parameters:

value (int) - Value in bytes

Returns: int

Value in mebibytes

CalculateDirectorySize(path)

source code

Calculates the size of a directory recursively.

Parameters:

path (string) - Path to directory

Returns: int

Size in mebibytes

GetFilesystemStats(path)

source code

Returns the total and free space on a filesystem.

Parameters:

path (string) - Path on filesystem to be examined

Returns: int

tuple of (Total space, Free space) in mebibytes

ReadPidFile(pidfile)

source code

Read a pid from a file.

Parameters:

pidfile (string) - path to the file containing the pid

Returns: int

The process id, if the file exists and contains a valid PID, otherwise 0

_ParsePidFileContents(data)

source code

Tries to extract a process ID from a PID file's content.

Parameters:

data (string)

Returns: int

Zero if nothing could be read, PID otherwise

ReadLockedPidFile(path)

source code

Reads a locked PID file.

This can be used together with utils.process.StartDaemon.

Parameters:

path (string) - Path to PID file

Returns:

PID as integer or, if file was unlocked or couldn't be opened, None

_SplitSshKey(key)

source code

Splits a line for SSH's authorized_keys file.

If the line has no options (e.g. no command="..."), only the significant parts, the key type and its hash, are used. Otherwise the whole line is used (split at whitespace).

Parameters:

key (string) - Key line

Returns: tuple

AddAuthorizedKey(file_obj, key)

source code

Adds an SSH public key to an authorized_keys file.

Parameters:

file_obj (str or file handle) - path to authorized_keys file
key (str) - string containing key

RemoveAuthorizedKey(file_name, key)

source code

Removes an SSH public key from an authorized_keys file.

Parameters:

file_name (str) - path to authorized_keys file
key (str) - string containing key

DaemonPidFileName(name)

source code

Compute a ganeti pid file absolute path

Parameters:

name (str) - the daemon name

Returns: str

the full path to the pidfile corresponding to the given daemon name

WritePidFile(pidfile)

source code

Write the current process pidfile.

Parameters:

pidfile (string) - the path to the file to be written

Returns: int

the file descriptor of the lock file; do not close this unless you want to unlock the pid file

Raises:

errors.LockError - if the pid file already exists and points to a live process

ReadWatcherPauseFile(filename, now=None, remove_after=3600)

source code

Reads the watcher pause file.

Parameters:

filename (string) - Path to watcher pause file
now (None, float or int) - Current time as Unix timestamp
remove_after (int) - Remove watcher pause file after specified amount of seconds past the pause end time

NewUUID()

source code

Returns a random UUID.

Returns: str

Note: This is a Linux-specific method as it uses the /proc filesystem.

IsUserInGroup(uid, gid)

source code

Returns True if the user belongs to the group.

Parameters:

uid (int) - the user id
gid (int) - the group id

Returns: bool

CanRead(username, filename)

source code

Returns True if the user can access (read) the file.

Parameters:

username (string) - the name of the user
filename (string) - the name of the file

Returns: bool