bexchange.net package

Submodules

bexchange.net.connections module

class bexchange.net.connections.backup_connection(backend, arguments)

Bases: publisher_connection

Backup connection, expects a list of senders in arguments. Where all senders are run.

publish(path, meta)

Publishes a file in some way using the underlying implementation :param path: The path to the file :param meta: The metadata of the file

class bexchange.net.connections.combined_connection(backend, arguments)

Bases: publisher_connection

Combined connection is a way to combine different connection types into one so that you for example can have different connections in one. For example, assume that you always wants a file to be sent to a specific target and then that you want a sequence of fail over connections after that.

publish(path, meta)

Publishes the file to all connections for this connection :param path: the file path :param meta: the metadata

class bexchange.net.connections.connection_manager

Bases: object

The connection manager is used for creating connection instances from the provided configuration

classmethod from_conf(backend, clz, arguments)

Creates an instance of clz with specified arguments :param clz: class name specified as <module>.<classname> :param arguments: a list of arguments that should be used to initialize the class

class bexchange.net.connections.distributed_connection(backend, arguments)

Bases: publisher_connection

Distributed connection, expects a list of senders in arguments. Where all senders are run. This is the same behavior as the backup connection but it’s here for readability.

publish(path, meta)

Publishes a file in some way using the underlying implementation :param path: The path to the file :param meta: The metadata of the file

class bexchange.net.connections.failover_connection(backend, arguments)

Bases: publisher_connection

Failover connection, expects a list of senders in arguments. Where they are tried in order until one works.

publish(path, meta)

Publishes the file using the senders. When first successful publishing has successfully been transmitted, the method returns. If on the other hand no senders are successfully used an exception will be thrown. :param path: The path to the file :param meta: The metadata of the file

class bexchange.net.connections.publisher_connection(backend)

Bases: object

publisher connections are used for publishing files in various ways without taking the protocol / sender into account. Each (or most) publisher_connection is associated with one or more senders.

backend()

Returns the backend

publish(path, meta)

Publishes a file in some way using the underlying implementation :param path: The path to the file :param meta: The metadata of the file

class bexchange.net.connections.simple_connection(backend, arguments)

Bases: publisher_connection

Simple connection, only parsing arguments according to. sender class + arguments.

publish(path, meta)

Publishes the file using the sender :param path: The path to the file :param meta: The metadata of the file

bexchange.net.fetchers module

class bexchange.net.fetchers.baseuri_fetcher(backend, aid, arguments)

Bases: fetcher

Base class for fetching data using basic file transmission protocols like sftp, ftp, …

hostname()

Returns the hostname extracted from the uri

password()

:return the password

path()

:return the path

port()

Returns the port number extracted from the uri.

set_port(portno)

Sets the portnumber :param portno: The portnumber to use

uri()

username()

:return the user name

class bexchange.net.fetchers.baseuri_patternmatching_fetcher(backend, aid, arguments)

Bases: baseuri_fetcher

Base class for fetching data using basic file transmission protocols like sftp, ftp, …

fnpattern()

Returns the fnpattern :return: the filename pattern, like *.h5

pattern()

:return the regular expression matching pattern

pattern_matcher()

:return the pattern matcher for the regular expression if there is one

class bexchange.net.fetchers.copy_fetcher(backend, aid, arguments)

Bases: fetcher

Fetcher file(s) using copy

fetch(**kwargs)

Fetches the file(s) using copy. :param file: path to file that should be published :param meta: the meta object for all metadata of file

pattern_matcher()

:return the pattern matcher

class bexchange.net.fetchers.fetcher(backend, aid)

Bases: object

Base fetcher. All classes implementing this should provide ability to fetch files in some way.

backend()

Returns the server backend

abstract fetch(**kwargs)

Fetches some files

id()

Returns this fetcher id

class bexchange.net.fetchers.fetcher_manager

Bases: object

Creates fetcher instances from a configuration entry

classmethod from_conf(backend, arguments)

Creates an instance of clz with specified arguments :param arguments: a list of arguments that should be used to initialize the class

class bexchange.net.fetchers.ftp_fetcher(backend, aid, arguments)

Bases: baseuri_patternmatching_fetcher

Fetcher file(s) using scp

connect()

fetch(**kwargs)

Fetches the file(s) using ftp. :param file: path to file that should be published :param meta: the meta object for all metadata of file

class bexchange.net.fetchers.scp_fetcher(backend, aid, arguments)

Bases: baseuri_patternmatching_fetcher

Fetcher file(s) using scp

fetch(**kwargs)

Fetches the file(s) using scp. :param file: path to file that should be published :param meta: the meta object for all metadata of file

class bexchange.net.fetchers.sftp_fetcher(backend, aid, arguments)

Bases: baseuri_patternmatching_fetcher

Fetcher files using sftp

fetch(**kwargs)

Fetches the file using sftp. :param file: path to file that should be published :param meta: the meta object for all metadata of file

bexchange.net.publishers module

class bexchange.net.publishers.pubQueue(qs)

Bases: object

Wrapper around queue.Queue to be able to shutdown. This will be supported in python 3.13 but for now this will be enough.

get(waittime=10)

Returns an item from the queue. The wait time is the time in seconds the thread should wait in the condition until checking for any new item in the queue. This condition will be notified whenever put or shutdown is called. :param waittime: The time to wait in seconds inside the condition :return: Will always return an item :throws: pubQueueShutdown

put(item)

Puts an item into the queue without blocking. If queue is shutdown, the item will not be added to the queue and this will be done silently :param item: the item to add to the queue

shutdown()

Shuts down the queue.

task_done()

Call this when task grabbed from queue is finished

exception bexchange.net.publishers.pubQueueShutdown

Bases: Exception

thrown to indicate that an entry already exists

class bexchange.net.publishers.publisher(backend, name, active, origin, ifilter, connections, decorators)

Bases: object

Base class used by all publishers

active()

Returns if this publication is active or not :return: if this publication is active or not

backend()

Returns the backend :return: the backend

connections()

Returns the connection(s) :return: the connection(s)

decorators()

Returns the decorator(s) :return: the decorator(s)

filter()

Returns the filter :return: the filter

initialize()

Initializes the publisher before it is started.

name()

Returns the name :return: the name

origin()

:return the list of origins that should trigger this publication

publish(file, meta)

publishes a file

Parameters:

• file – The temp file instance

• meta – The metadata

Returns:

metadata extracted from the file

start()

A publisher should most likely be managing threads and as such a start method is needed

stop()

A publisher should most likely be managing threads and as such a stop method is needed joining the threads

class bexchange.net.publishers.publisher_manager

Bases: object

The manager used for creating publishers from configuration

classmethod create_publisher(name, clz, backend, active, origin, filter, connections, decorators, extra_arguments)

Creates an instance of clz with specified arguments :param clz: class name specified as <module>.<classname> :param arguments: a list of arguments that should be used to initialize the class

classmethod from_conf(config, backend)

Creates the publisher instance from provided configuration. :param config: The json config in a dict :param backend: The backend the publisher shall have access to :return: the created publisher

class bexchange.net.publishers.standard_publisher(backend, name, active, origin, ifilter, connections, decorators, extra_arguments)

Bases: publisher

Standard publisher used for most situations. Provides two arguments. One is threads. The other is queue_size.

consumer()

The consumer called by the individual threads. Will grab one entry from the queue and pass it on to the connections.

do_publish(tmpfile, meta)

Passes a file to all connections

handle_consumer_file(tmpfile, meta)

Will handle the file that the consumer retrieved from the queue :param self: self :param tmpfile: the tmp file :param meta: the meta data

initialize()

Initializes the publisher before it is started.

publish(file, meta)

Passes on a file to the consumer queue that will be consumed and passed on to the connections. :param file: Actual file to be duplicated and sent. :param meta: The metadata to be verified by the connections

start()

Starts all consumer threads as daemon threads

stop()

Joins the threads

bexchange.net.senders module

class bexchange.net.senders.baseuri_sender(backend, aid, arguments)

Bases: sender

Base class for basic file transmission protocols like sftp, ftp, …

create_missing_directories()

Returns if missing directories should be created or not

hostname()

Returns the hostname extracted from the uri

name(meta)

Creates a unique name from the metadatanamer created from the uri :param meta: Metadata to use when creating the name

password()

port()

Returns the port number extracted from the uri.

username()

class bexchange.net.senders.copy_sender(backend, aid, arguments)

Bases: sender

Sends files by copying them

create_missing_directories()

Returns:

if missing directories should be created or not

name(meta)

Creates the name to use when copying the file from the metadata :param meta: The metadata :return: the name to use for destination

send(path, meta)

Sends the file using copy. :param file: path to file that should be sent :param meta: the meta object for all metadata of file

class bexchange.net.senders.dex_sender(backend, aid, arguments)

Bases: sender

Legacy DEX communication sending files to old nodes.

send(path, meta)

Sends the file to the dex server :param file: path to file that should be sent :param meta: the meta object for all metadata of file

class bexchange.net.senders.ftp_sender(backend, aid, arguments)

Bases: baseuri_sender

Publishes files over ftp

change_and_create_dir(ftp, path)

Changes directory and creates specified directory. :param ftp: The ftp connection :param path: The path to create

connect()

Connects to the host specified by host / port. :return: the ftp connection

does_dir_exist(ftp, path)

Tests if the specified directory exist using the provided ftp-connection :param ftp: the ftp connection :param path: the path to test :return: True or False

send(path, meta)

Sends the file using ftp. :param file: path to file that should be published :param meta: the meta object for all metadata of file

class bexchange.net.senders.rest_sender(backend, aid, arguments)

Bases: sender

Sends a file to another node that is running bexchange. The rest sender uses the internal crypto library for signing messages which currently supports DSA & RSA keys. DSA uses DSS, RSA uses pkcs1_15.

send(path, meta)

Sends the file to the bexchange server :param file: path to file that should be sent :param meta: the meta object for all metadata of file

class bexchange.net.senders.scp_sender(backend, aid, arguments)

Bases: baseuri_sender

Publishes files over scp

send(path, meta)

Sends the file using scp. :param file: path to file that should be sent :param meta: the meta object for all metadata of file

class bexchange.net.senders.sender(backend, aid)

Bases: object

Base sender. All classes implementing this should send the specified file path to a recipient over a specific protocol.

backend()

Returns the server backend

id()

Returns this uploader id

abstract send(path, meta)

Sends path with the metadata meta :param path: Path to file to be uploaded :param meta: Metadata of the uploaded file

class bexchange.net.senders.sender_manager

Bases: object

classmethod from_conf(backend, arguments)

Creates an instance of clz with specified arguments :param arguments: a list of arguments that should be used to initialize the class

class bexchange.net.senders.sftp_sender(backend, aid, arguments)

Bases: baseuri_sender

Sends files over sftp

send(path, meta)

Sends the file using sftp. :param file: path to file that should be sent :param meta: the meta object for all metadata of file

class bexchange.net.senders.storage_sender(backend, aid, arguments)

Bases: sender

Sends files using a number of storages.

send(file, meta)

Sends the file on all storages associated with self :param file: path to file that should be sent :param meta: the meta object for all metadata of file

bexchange.net.sftpclient module

class bexchange.net.sftpclient.sftpclient(host, port, username, password, timeout=30.0, banner_timeout=30)

Bases: object

Implementation of a sftpclient using paramiko

chdir(dirname)

Changes current directory on server :param dirname: The directory to change to

connect()

(Re)connects to the sftp server

disconnect()

Disconnects from the sftp server

get(filename, targetname)

Fetches the file (filename) from the server and saves it as targetname. :param filename: Source filename to download :param targetname: The destination name to save as

getfo(remotepath, fl)

Copies the file (remotepath) from the server and writes to an open file. Typically used like: with NamedTemporaryFile() as fp: c.getfo(”….”, fp)

Parameters:

• remotepath – Source filename to download

• fl – The destination file object

hostname()

:return the hostname

isdir(path)

Checks if the provided path is a directory or not

Parameters:

path – Path name to check

Returns:

if specified path is a dir or not

isfile(path)

Parameters:

path – Path name to check

Returns:

if specified path is a file or not

listdir(dirname)

Lists files and directories in the directory :param dirname: The directory to list :return a list of names

makedirs(targetdir)

Creates the directories if they don’t exist :param targetdir: The structure to create from current wd

password()

:return the password

port()

:return the port number

put(filename, targetname, confirm=True)

Uploads the file (filename) to the server named targetname. :param filename: Source filename to upload :param targetname: The destination name

username()

:return the username

Module contents