hyperstream package

Subpackages

• hyperstream.channels package
  • Submodules
  • hyperstream.channels.assets_channel module
  • hyperstream.channels.assets_channel2 module
  • hyperstream.channels.base_channel module
  • hyperstream.channels.database_channel module
  • hyperstream.channels.file_channel module
  • hyperstream.channels.memory_channel module
  • hyperstream.channels.module_channel module
  • hyperstream.channels.tool_channel module
  • Module contents
• hyperstream.itertools2 package
  • Submodules
  • hyperstream.itertools2.itertools2 module
  • Module contents
• hyperstream.models package
  • Submodules
  • hyperstream.models.factor module
  • hyperstream.models.meta_data module
  • hyperstream.models.node module
  • hyperstream.models.plate module
  • hyperstream.models.stream module
  • hyperstream.models.time_interval module
  • hyperstream.models.tool module
  • hyperstream.models.workflow module
  • Module contents
• hyperstream.stream package
  • Submodules
  • hyperstream.stream.stream module
  • hyperstream.stream.stream_collections module
  • hyperstream.stream.stream_id module
  • hyperstream.stream.stream_instance module
  • hyperstream.stream.stream_view module
  • Module contents
• hyperstream.tool package
  • Submodules
  • hyperstream.tool.aggregate_tool module
  • hyperstream.tool.base_tool module
  • hyperstream.tool.multi_output_tool module
  • hyperstream.tool.plate_creation_tool module
  • hyperstream.tool.selector_tool module
  • hyperstream.tool.tool module
  • Module contents
• hyperstream.utils package
  • Submodules
  • hyperstream.utils.decorators module
  • hyperstream.utils.errors module
  • hyperstream.utils.serialization module
  • hyperstream.utils.time_utils module
  • hyperstream.utils.utils module
  • Module contents
• hyperstream.workflow package
  • Submodules
  • hyperstream.workflow.factor module
  • hyperstream.workflow.meta_data_manager module
  • hyperstream.workflow.node module
  • hyperstream.workflow.plate module
  • hyperstream.workflow.plate_manager module
  • hyperstream.workflow.workflow module
  • hyperstream.workflow.workflow_manager module
  • Module contents

Submodules

hyperstream.channel_manager module

hyperstream.client module

The main hyperstream client connection that is used for storing runtime information. Note that this is also used by the default database channel, although other database channels (connecting to different database types) can also be used.

class hyperstream.client.Client(server_config, auto_connect=True)

Bases: hyperstream.utils.containers.Printable

The main mongo client

client = None

connect(server_config)

Connect using the configuration given

Parameters:server_config – The server configuration

db = None

get_config_value(key, default=None)

Get a specific value from the configuration

Parameters:

• key – The of the item
• default – A default value if not found

Returns:

The found value or the default

session = None

hyperstream.config module

HyperStream configuration module.

class hyperstream.config.HyperStreamConfig(filename='hyperstream_config.json')

Bases: hyperstream.utils.containers.Printable

Wrapper around the hyperstream configuration file

class hyperstream.config.OnlineEngineConfig(interval, sleep=5, iterations=100, alarm=None)

Bases: hyperstream.utils.containers.Printable

hyperstream.hyperstream module

Main HyperStream class

class hyperstream.hyperstream.HyperStream(loglevel=20, file_logger=True, console_logger=True, mqtt_logger=None, config_filename='hyperstream_config.json')

Bases: object

HyperStream class: can be instantiated simply with hyperstream = HyperStream() for default operation. Use in the following way to create a session (and store history of execution etc). >>> with Hyperstream(): >>> pass

Note that HyperStream uses the singleton pattern described here:

https://stackoverflow.com/a/33201/1038264

For py2k/py3k compatability we use the six decorator add_metaclass:

https://pythonhosted.org/six/#six.add_metaclass

add_workflow(workflow)

Add the workflow to the workflow manager

Parameters:workflow (Workflow) – The workflow Returns:None

clear_sessions(inactive_only=True, clear_history=False)

Clears all stored sessions, optionally excluding active sessions

Parameters:

• inactive_only – Whether to clear inactive sessions only
• clear_history – Whether to also clear session history

Returns:

None

create_workflow(**kwds)

Create a new workflow. Simple wrapper for creating a workflow and adding it to the workflow manager.

Parameters:

• workflow_id – The workflow id
• name – The workflow name
• owner – The owner/creator of the workflow
• description – A human readable description
• online – Whether this workflow should be executed by the online engine
• monitor – Whether the workflow computations should be monitored
• safe – If safe=True, will throw an error if the workflow already exists

Returns:

The workflow

current_session

Get the current session

Returns:the current session

new_session()

Start a new session to record computation history

Returns:the created session

populate_tools_and_factors()

Function to populate factory functions for the tools and factors for ease of access.

Returns:None

sessions

Get the list of sessions

Returns:the sessions

hyperstream.online_engine module

Online Engine module. This will be used in the online execution mode.

class hyperstream.online_engine.OnlineEngine(hyperstream)

Bases: object

OnlineEngine class.

execute(**kwargs)

Execute the engine - currently simple executes all workflows.

hyperstream.plugin_manager module

Plugin manager module for additional user added channels and tools.

class hyperstream.plugin_manager.Plugin

Bases: hyperstream.plugin_manager.PluginBase, hyperstream.utils.containers.Printable

Plugin class - simple wrapper over namedtuple

load_channels()

Loads the channels and tools given the plugin path specified

Returns:The loaded channels, including a tool channel, for the tools found.

hyperstream.time_interval module

Module for dealing with time intervals containing TimeInterval, TimeIntervals, and RelativeTimeInterval

class hyperstream.time_interval.RelativeTimeInterval(start, end)

Bases: hyperstream.time_interval.TimeInterval

Relative time interval object. Thin wrapper around a (start, end) tuple of timedelta objects that provides some validation

absolute(dt)

end

start

class hyperstream.time_interval.TimeInterval(start, end)

Bases: hyperstream.time_interval.TimeInterval

Time interval object. Thin wrapper around a (start, end) tuple of datetime objects that provides some validation

classmethod all_time()

end

humanized

classmethod now_minus(weeks=0, days=0, hours=0, minutes=0, seconds=0, milliseconds=0)

start

to_json()

to_tuple()

classmethod up_to_now()

width

class hyperstream.time_interval.TimeIntervals(intervals=None)

Bases: hyperstream.utils.containers.Printable

Container class for time intervals, that manages splitting and joining Example object: (t1,t2] U (t3,t4] U …

compress()

end

humanized

is_empty

parse(intervals)

span

split(points)

Splits the list of time intervals in the specified points

The function assumes that the time intervals do not overlap and ignores points that are not inside of any interval.

Parameters:points (list of datetime) –

start

to_json()

hyperstream.time_interval.parse_time_tuple(start, end)

Parse a time tuple. These can be:

relative in seconds, e.g. (-4, 0) relative in timedelta, e.g. (timedelta(seconds=-4), timedelta(0)) absolute in date/datetime, e.g. (datetime(2016, 4, 28, 20, 0, 0, 0, UTC), datetime(2016, 4, 28, 21, 0, 0, 0, UTC)) absolute in iso strings, e.g. (“2016-04-28T20:00:00.000Z”, “2016-04-28T20:01:00.000Z”) Mixtures of relative and absolute are not allowed

Parameters:

• start (int | timedelta | datetime | str) – Start time
• end (int | timedelta | datetime | str) – End time

Returns:

TimeInterval or RelativeTimeInterval object

hyperstream.time_interval.profile(ob)

Comment out this function to be able to use the line_profiler module. e.g. call: kernprof -l scripts/deploy_summariser.py –loglevel=10 python -m line_profiler deploy_summariser.py.lprof > deploy_summariser.py.summary :param ob: object :return: object

hyperstream.version module

Module contents