heliport.cwl_execution package

Subpackages

Submodules

heliport.cwl_execution.admin module

Registers Django admin pages.

See django.contrib.admin.ModelAdmin from Django documentation.

heliport.cwl_execution.apps module

Django app configuration.

Some HELIPORT hooks can be registered in app config django.apps.AppConfig.ready().

See also Django documentation

class heliport.cwl_execution.apps.CwlExecutionConfig(app_name, app_module)

Bases: AppConfig

App configuration for cwl execution app.

name = 'heliport.cwl_execution'
ready()

Import settings.

heliport.cwl_execution.conf module

Additional settings for this app.

See also django appconf

class heliport.cwl_execution.conf.CwlExecutionAppConf(**kwargs)

Bases: AppConf

Settings of the heliport.cwl_execution app.

All settings in this class can be overwritten in settings.py

DEFAULT_JOB_OPTIONS = '--time=10  # min or day-hour:min\n#--partition=gpu  # part of cluster\n#--nodes=8  # number compute nodes\n# ... many more options possible'
DEFAULT_JOB_PATH = '~/heliport_jobs_3_9'
DEFAULT_JOB_SETUP_COMMAND = 'ml python/3.9\n@HELIPORT load venv  # or conda activate, ...'
class Meta

Bases: object

prefix = 'HELIPORT_CWL'
SSH_DATA_TRANSFER_BATCH_SIZE = 10000
USE_SRUN = True

heliport.cwl_execution.interface module

Module with special name “interface” hooks into HELIPORT.

Some functions and heliport.core.app_interaction.Module subclasses are detected by HELIPORT and control how HELIPORT uses this app.

Note that this module must be imported in __init__.py of the django app.

class heliport.cwl_execution.interface.ArbitraryCWLModule

Bases: Module

get_url(project)
is_configured(project)
module_id = 'arbitrary_cwl'
name = 'CWL File'
class heliport.cwl_execution.interface.JobModule

Bases: Module

get_url(project)
is_configured(project)
module_id = 'job'
name = 'CWL Jobs'
class heliport.cwl_execution.interface.ToolModule

Bases: Module

get_url(project)
is_configured(project)
module_id = 'tool'
name = 'Tool'
class heliport.cwl_execution.interface.WorkflowModule

Bases: Module

get_url(project)
is_configured(project)
module_id = 'workflow'
name = 'Workflow'
heliport.cwl_execution.interface.get_search_url()
heliport.cwl_execution.interface.serialize_project(project)

heliport.cwl_execution.models module

Contains django.db.models.Model classes for Django ORM.

See Quick example from Django documentation. In HELIPORT the heliport.core.models.DigitalObject can be subclassed for models containing metadata in a project.

class heliport.cwl_execution.models.Executable(digital_object_id, persistent_id, generated_persistent_id, category, label, label_is_public, description, description_is_public, created, last_modified, last_modified_is_public, deleted, permission, special_heliport_role, is_helper, projects_is_public, owner, members_is_public, digitalobject_ptr, executable_id, cwl_json, cwl_is_public, cwl_type, type, download_url_is_public, original_id)

Bases: DigitalObject

class CWLTypes(value)

Bases: IntegerChoices

An enumeration.

JSON = 1
YAML = 2
exception DoesNotExist

Bases: DoesNotExist

class ExecutableTypes(value)

Bases: IntegerChoices

An enumeration.

ARBITRARY_CWL = 3
TOOL = 1
WORKFLOW = 2
exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

property cwl
cwl_is_public

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

cwl_json

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

cwl_type

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

digitalobject_ptr

Accessor to the related object on the forward side of a one-to-one relation.

In the example:

class Restaurant(Model):
    place = OneToOneField(Place, related_name='restaurant')

Restaurant.place is a ForwardOneToOneDescriptor instance.

digitalobject_ptr_id
download_url_is_public

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

executable_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_cwl_type_display(*, field=<django.db.models.fields.IntegerField: cwl_type>)
get_type_display(*, field=<django.db.models.fields.IntegerField: type>)
property job_count
job_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

original_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property parameters
type

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

uses_files()
class heliport.cwl_execution.models.ExecutionLog(execution_log_id, config, start_time, end_time, memory_usage, execution_id, aborted, step_times, output_code, error_message, user, original_id)

Bases: Model

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

aborted

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

config

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

config_id
property duration
property duration_str
end_time

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

error_message

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

execution_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

execution_log_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

get_next_by_start_time(*, field=<django.db.models.fields.DateTimeField: start_time>, is_next=True, **kwargs)
get_output_code_display(*, field=<django.db.models.fields.IntegerField: output_code>)
get_previous_by_start_time(*, field=<django.db.models.fields.DateTimeField: start_time>, is_next=False, **kwargs)
get_user()
memory_usage

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property memory_usage_bytes
property memory_usage_dict
objects = <django.db.models.manager.Manager object>
original_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

output_code

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

static parse_memory_units(memory_with_unit)
property profiling
property seconds_since_start
start(execution_id=None)
start_time

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

step_times

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

stop(kill=False)
user

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

user_id
class heliport.cwl_execution.models.Job(digital_object_id, persistent_id, generated_persistent_id, category, label, label_is_public, description, description_is_public, created, last_modified, last_modified_is_public, deleted, permission, special_heliport_role, is_helper, projects_is_public, owner, members_is_public, digitalobject_ptr, job_id, status, output_text, output_code, cwl_json, repeat, interval, base_executable, setup_command, cmd_options, original_id)

Bases: DigitalObject

exception DoesNotExist

Bases: DoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

class OutputCodes(value)

Bases: IntegerChoices

An enumeration.

ERROR = 2
MESSAGE = 0
UNDEFINED = -1
WARNING = 1
class StatusCodes(value)

Bases: TextChoices

An enumeration.

PAUSED = 'Paused'
RUNNING = 'Running'
STOPPED = 'Stopped'
base_executable

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

base_executable_id
cmd_options

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property cwl
cwl_json

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

digitalobject_ptr

Accessor to the related object on the forward side of a one-to-one relation.

In the example:

class Restaurant(Model):
    place = OneToOneField(Place, related_name='restaurant')

Restaurant.place is a ForwardOneToOneDescriptor instance.

digitalobject_ptr_id
get_output_code_display(*, field=<django.db.models.fields.IntegerField: output_code>)
get_status_display(*, field=<django.db.models.fields.CharField: status>)
interval

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

job_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

original_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

output_code

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

output_text

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

repeat

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

setup_command

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

status

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

userjobconfig_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

class heliport.cwl_execution.models.UserJobConfig(user_job_config_id, job, ssh_login, directory, original_id)

Bases: Model

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

directory

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

executionlog_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

job

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

job_id
objects = <django.db.models.manager.Manager object>
original_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property projects
ssh_login

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

ssh_login_id
user_job_config_id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

heliport.cwl_execution.models.json_load_dict(json_string, error_dict=None)

heliport.cwl_execution.serializers module

This module is for serialization into Datacite, RDF and JSON.

The JSON format is used for API endpoints via Django rest framework. Using this, it is typical to put the serializer classes into a “serializers.py” file.

For the serialization into RDF, attributes are described. See heliport.core.attribute_description.BaseAttribute for more detail.

class heliport.cwl_execution.serializers.ExecutableDATACITESerializer

Bases: DigitalObjectDATACITESerializer

resource_type(exe)
class heliport.cwl_execution.serializers.ExecutableRDFSerializer

Bases: DigitalObjectRDFSerializer

static_triples(exe, user)
class heliport.cwl_execution.serializers.ExecutableSerializer(*args, **kwargs)

Bases: ModelSerializer

class Meta

Bases: object

fields = ['executable_id', 'cwl_type', 'projects', 'type', 'cwl', 'cwl_json']
model

alias of Executable

class heliport.cwl_execution.serializers.ExecutionLogSerializer(*args, **kwargs)

Bases: ModelSerializer

class Meta

Bases: object

fields = ['execution_log_id', 'config', 'start_time', 'end_time', 'memory_usage', 'execution_id', 'aborted', 'step_times', 'output_code', 'error_message', 'user']
model

alias of ExecutionLog

class heliport.cwl_execution.serializers.JobDATACITESerializer

Bases: DigitalObjectDATACITESerializer

related_identifiers(job, user)
resource_type(job)
class heliport.cwl_execution.serializers.JobSerializer(*args, **kwargs)

Bases: ModelSerializer

class Meta

Bases: object

fields = ['job_id', 'projects', 'label', 'status', 'description', 'output_text', 'output_code', 'cwl', 'repeat', 'interval', 'base_executable', 'setup_command']
model

alias of Job

class heliport.cwl_execution.serializers.UserJobConfigSerializer(*args, **kwargs)

Bases: ModelSerializer

class Meta

Bases: object

fields = ['user_job_config_id', 'job', 'ssh_login', 'directory']
model

alias of UserJobConfig

heliport.cwl_execution.tasks module

heliport.cwl_execution.tests module

Test the behaviour of this app.

This follows the Writing tests guide in Django.

class heliport.cwl_execution.tests.ArbitraryCWLTests(methodName='runTest')

Bases: TestCase

setUp()

Set up project and logged-in heliport users.

test_list()
test_post_fail()
test_post_success()
test_remove()
test_update_list()
test_update_show()
class heliport.cwl_execution.tests.ToolTests(methodName='runTest')

Bases: TestCase

setUp()

Set up project and logged-in heliport users.

test_concat_base_command()
test_create_tool_and_update_tool()
test_json_description()
test_list_and_delete()
test_split_base_command()
class heliport.cwl_execution.tests.WorkflowTests(methodName='runTest')

Bases: TestCase

build_url(action, pk=None, no_project=False)
static generate_cwl()
setUp()

Set up project and logged-in heliport users.

test_create_tool_and_update_tool()
test_cwl_to_workflow()
test_generate_port_data()
test_id_generation()
test_label_index()
test_list_and_delete()
test_node_categorization()
test_node_visualization_data()
test_unique_labels()
test_workflow_to_cwl()

heliport.cwl_execution.urls module

Map django views to urls.

See this Example including explanation from the Django documentation.

heliport.cwl_execution.views module

Contains Django View classes to handle HTTP requests.

See Using class-based views from Django documentation. In HELIPORT heliport.core.mixins are used to create uniform views. Also heliport.core.views.generic.HeliportObjectListView is used to quicly create a typical HELIPORT list view.

class heliport.cwl_execution.views.ArbitraryCWLListView(**kwargs)

Bases: HeliportProjectMixin, ListView

context_object_name = 'arbitrary_cwls'
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

get_queryset()
model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/arbitrary_cwl.html'
class heliport.cwl_execution.views.ArbitraryCWLUpdateView(**kwargs)

Bases: HeliportObjectMixin, UpdateView

fields = ['cwl_json', 'cwl_type']
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes the CWL.

model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/arbitrary_cwl.html'
class heliport.cwl_execution.views.ExecutableRawJsonView(**kwargs)

Bases: DetailView

URLs of this view also used in cwl identifiers.

get(request, *args, **kwargs)
model

alias of Executable

class heliport.cwl_execution.views.ExecutableRawYamlView(**kwargs)

Bases: DetailView

static check_permission(exe, request)
get(request, *args, **kwargs)
model

alias of Executable

static redirect_part(exe, kwargs)
class heliport.cwl_execution.views.ExecutableViewSet(**kwargs)

Bases: HeliportModelViewSet

Tools, Workflows and Arbitrary CWL.

filterset_fields = ['executable_id', 'cwl_type', 'type']
get_queryset()
serializer_class

alias of ExecutableSerializer

class heliport.cwl_execution.views.ExecutionLogPermission

Bases: BasePermission

Permission class to be used for execution logs. See ExecutionLogViewSet.

has_object_permission(request, view, obj: ExecutionLog)

permission is granted if user heliport.core.permissions.has_permission_for() the job this log belongs to.

class heliport.cwl_execution.views.ExecutionLogViewSet(**kwargs)

Bases: HeliportModelViewSet

Concrete executions of a UserJobConfig.

filterset_fields = ['execution_log_id', 'config', 'execution_id', 'aborted', 'output_code', 'user']
get_queryset()
permission_classes = [<class 'heliport.cwl_execution.views.ExecutionLogPermission'>]

Make sure user is authenticated and is allowed to access the DigitalObject

serializer_class

alias of ExecutionLogSerializer

class heliport.cwl_execution.views.JobCreateView(**kwargs)

Bases: HeliportObjectMixin, DetailView

add_tags(tag_dict, cwl, job)
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes the job to edit if it exists.

get_default_tag(already_saved_job)
model

alias of Executable

pk_url_kwarg = 'executable'
post(request, *args, **kwargs)
template_name = 'cwl_execution/job_create.html'
class heliport.cwl_execution.views.JobDebugView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of UserJobConfig

class heliport.cwl_execution.views.JobDetailView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes the job output in html format.

model

alias of Job

template_name = 'cwl_execution/job_detail.html'
class heliport.cwl_execution.views.JobListView(**kwargs)

Bases: HeliportProjectMixin, ListView

context_object_name = 'userjobconfigs'
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes the default login calculated from statistics of past executions for the user.

get_queryset()
model

alias of UserJobConfig

post(request, *args, **kwargs)
template_name = 'cwl_execution/job_list.html'
class heliport.cwl_execution.views.JobOutputView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of Job

class heliport.cwl_execution.views.JobParameterValueView(**kwargs)

Bases: HeliportProjectMixin, TemplateView

get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

static serialize(tag_obj)
template_name = 'cwl_execution/job_parameter_input.html'
class heliport.cwl_execution.views.JobProfilingJsonView(**kwargs)

Bases: HeliportProjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of Project

pk_url_kwarg = 'project'
class heliport.cwl_execution.views.JobRawYamlView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of Job

class heliport.cwl_execution.views.JobStartView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of UserJobConfig

class heliport.cwl_execution.views.JobStopView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of UserJobConfig

class heliport.cwl_execution.views.JobViewSet(**kwargs)

Bases: HeliportModelViewSet

Jobs. To execute a job go to its UserJobConfig like …/user_job_config/user_job_config id/execute/.

filterset_fields = ['job_id', 'label', 'status', 'output_code', 'repeat', 'interval', 'base_executable', 'setup_command']
get_queryset()
serializer_class

alias of JobSerializer

class heliport.cwl_execution.views.PublicationWorkflowCreateView(**kwargs)

Bases: HeliportProjectMixin, TemplateView

get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

post(request, *args, **kwargs)
template_name = 'cwl_execution/publication_workflow.html'
class heliport.cwl_execution.views.RunningJobsView(**kwargs)

Bases: HeliportProjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of Project

pk_url_kwarg = 'project'
class heliport.cwl_execution.views.SearchView(**kwargs)

Bases: HeliportLoginRequiredMixin, TemplateView

get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes the search results.

template_name = 'cwl_execution/search.html'
class heliport.cwl_execution.views.ToolCreateView(**kwargs)

Bases: HeliportProjectMixin, TemplateView

post(request, *args, **kwargs)
template_name = 'cwl_execution/tool_edit.html'
class heliport.cwl_execution.views.ToolJsonDescriptionView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of Executable

class heliport.cwl_execution.views.ToolListView(**kwargs)

Bases: HeliportProjectMixin, ListView

context_object_name = 'tools'
get_queryset()
model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/tool_list.html'
class heliport.cwl_execution.views.ToolUpdateView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes to render template for editing.

model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/tool_edit.html'
class heliport.cwl_execution.views.UserJobConfigViewSet(**kwargs)

Bases: ModelViewSet

User specific config for a job.

You can Execute your config by appending /execute/ to the URL. For example: …/user_job_config/123/execute/.

execute(request, pk=None)
filterset_fields = ['user_job_config_id', 'job', 'ssh_login', 'directory']
get_queryset()
permission_classes = [<class 'rest_framework.permissions.IsAuthenticated'>, <class 'heliport.core.permissions.HeliportObjectPermission'>]
serializer_class

alias of UserJobConfigSerializer

class heliport.cwl_execution.views.VisualizationDataView(**kwargs)

Bases: HeliportObjectMixin, DetailView

get(request, *args, **kwargs)
model

alias of Executable

class heliport.cwl_execution.views.WorkflowCreateView(**kwargs)

Bases: HeliportProjectMixin, TemplateView

context_object_name = 'workflow'
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes executables that can be added to workflow.

model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/workflow_edit.html'
class heliport.cwl_execution.views.WorkflowListView(**kwargs)

Bases: HeliportProjectMixin, ListView

context_object_name = 'workflows'
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes the data sources to warn about if no exist and the Workflow uses some.

get_queryset()
model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/workflow_list.html'
class heliport.cwl_execution.views.WorkflowUpdateView(**kwargs)

Bases: HeliportObjectMixin, DetailView

context_object_name = 'workflow'
get_context_data(**kwargs)

Add extra context for rendering template (Called by django base view).

The extra context includes executables that can be added to workflow.

model

alias of Executable

post(request, *args, **kwargs)
template_name = 'cwl_execution/workflow_edit.html'
heliport.cwl_execution.views.start_job(job_config, request, project_pk=None, debug=False)

Module contents

App to run, manage and record metadata related to CWL workflows.

The interface module is imported to the top level of the package for HELIPORT app interface discovery (see heliport.core.app_interaction.get_heliport_apps()).